PRACTICE · DEEPEN

Write an assessment worth trusting

A grade is a claim: this score means the student can do this thing. Most of the effort in assessment goes into scoring and analyzing results — but a number is only worth analyzing if the task that produced it measured what you think it did. This page is about the part that comes first: building the assessment so the claim holds. It pairs with Assess by Design, which handles the Canvas machinery once the task itself is sound.

Giving this as a test that counts? Check the six test-security musts: close the window, access code, hidden results, LockDown Browser, extra time, leaving the room. See all six →

A STARTING POINT — AND IT NEEDS YOUR JUDGMENT

This page is a draft, and I’m still iterating on it. It wasn’t written by an expert in your field — it is here to start a pathway you then shape to fit you: your subject, your level, your students, your department. Your expertise runs far deeper than anything here, and none of this is finished until your judgment has been on it.

That is the whole point. Judgment is the most important part of this — yours, and the judgment you are building in your students. These pages exist to help you develop and exercise it, never to replace it. Take what is useful, change what is not, and tell me where I have gotten it wrong — that is how these get better.

YOU'LL BE ABLE TO

Name what a score is and isn't evidence of, blueprint a test to your goals before writing it, and spot the two ways an assessment quietly measures the wrong thing.

YOU'LL HAVE

A four-step blueprint, a checklist for writing and screening items, a rubric-and-consistency routine, and an honest read on which assessment claims the evidence actually supports.

THE RUNGS

Foundations, does it measure the right thing. Enhance, do the items and rubric hold up. Deepen, is the scoring consistent and fair. Innovate, does it survive the AI era honestly.

One idea holds the whole page together. Validity is not a property a test has — it is a property of the claim you make from the score. “This 40-question test is valid” is not a sentence that means anything. “This score is fair grounds for saying she can analyze a primary source” is a claim, and you can build evidence for it or fail to. Everything below is a way of building that evidence. (Messick, 1989; and the profession's shared reference, the AERA/APA/NCME Standards for Educational and Psychological Testing, 2014 — still the current edition.)

FOUNDATIONS — DOES IT MEASURE THE RIGHT THING

Before you write a single question, decide what the assessment is for and what score would count as evidence of it. An assessment goes wrong in exactly two ways, and both are invisible until you name them:

IT MEASURES SOMETHING IT SHOULDN'T

Construct-irrelevant variance. Reading speed contaminating a math score. A lab report grade that quietly rewards neat handwriting. The score moves for a reason that has nothing to do with the skill — so the same skill earns different grades depending on something you never meant to test.

IT MISSES SOMETHING IT SHOULD MEASURE

Construct underrepresentation. A writing test that only checks grammar and never asks for an argument. A unit on scientific reasoning assessed entirely by vocabulary recall. The task is too narrow for the claim you want to make from it.

Blueprint before you write. A blueprint is four columns and ten minutes, and it is the single highest-leverage habit in assessment design:

List the learning goals this assessment is meant to give evidence of — in the words you'd use to a parent, not the standard's code.
Set the cognitive level for each (recall / apply / analyze / create — Bloom, or DOK 1–4 if your department uses it). A goal that says “analyze” assessed by a recall question is underrepresentation in miniature.
Decide the weight — how many points or items each goal gets. This is where you find out that 60% of the test is on the thing you spent one day on.
Only now, write items to fill the cells. Every item traces to a goal-and-level cell. Anything that traces to no cell is measuring something you didn't mean to grade.

Foundations is done when you can hold up the assessment and say, for every part of it, which goal it gives evidence of and at what level — and when nothing on it is there by habit alone.

ENHANCE — DO THE ITEMS AND THE RUBRIC HOLD UP

Now the craft of the individual pieces. A good blueprint filled with bad items still produces a score you can't trust.

Selected-response items (multiple choice, matching). The failure mode is almost always that the item tests test-wiseness instead of the skill:

Put the actual cognitive work in the stem, phrased as a complete question, so a student who knows the material could nearly answer before reading the options.
Make every distractor a real misconception — the wrong answer a student who made a specific known error would land on. Distractors that are merely wrong test nothing; distractors that are diagnostic turn the item into information you can teach from later (this is what makes the results readable in Quiz Data Coach).
Strip the giveaways: options that differ in length or grammar, “all of the above,” absolutes like “always/never,” and negatively-worded stems that test careful reading rather than knowledge.

Constructed-response items and the rubric that scores them. Write the rubric with the task, never after collecting the work:

Each criterion should name a piece of the construct — the reasoning, the evidence use, the technique — not the packaging. If “neatness” or “followed formatting” carries real weight on a reasoning task, you have written construct-irrelevant variance into the rubric on purpose.
Write level descriptors a student could self-assess against. “Sophisticated analysis” is not a level; “explains why the evidence supports the claim, not just that it does” is.
Pair the rubric with exemplars. Decades of feedback research find that students calibrate to a standard far better from an annotated example of the standard met than from the criteria alone (Sadler, 1989). A rubric without exemplars is a rubric only you can apply.

Build once, as a level. When a whole level shares a blueprint and rubric, every teacher's scores mean the same thing and the data actually aggregates — the payoff the Assess by Design page is built to capture. Shared item banks and the Raider Quiz Builder / Item Bank Expander tools make writing the good version once, together, cheaper than each teacher writing a weaker version alone.

DEEPEN — IS THE SCORING CONSISTENT AND FAIR

A defensible task can still produce an indefensible grade if the scoring wanders — between two students, or between the same student's paper read first period and read at 10 p.m. Consistency is not fussiness; it is the difference between a score that means something and one that records your mood.

Anchor before you score. Pull three papers — strong, middle, weak — agree on where they land against the rubric, and grade the set against those anchors. On a shared assessment, a level doing this together for fifteen minutes is the whole of “inter-rater reliability” in practice: do our scores agree? Where they don't, the rubric language is ambiguous, and you've found it before it cost a student.
Grade one item across all papers, not one whole paper at a time. It holds the standard steady and blunts the halo effect — a strong first answer quietly inflating the rest of the paper.
Know the mode effect and control for it. The same essay scores differently typed versus handwritten — on the order of a third of a grade in either direction — largely because raters react to handwriting and to visible spelling and mechanical errors, not because either mode makes a student more able (Powers et al., 1994; Cambridge Assessment). If mechanics aren't the construct, grade for the construct: score blind to mode where you can, or read past the surface deliberately.
Decide what a grade represents — and be consistent about it. Points-accumulation and standards/proficiency grading answer “what does an 87 mean?” differently, and mixing them silently (averaging early-semester struggle with end-of-semester mastery, or folding compliance into an achievement grade) reintroduces exactly the construct-irrelevant variance you removed at Foundations. You don't have to adopt standards-based grading; you do have to know which claim your grade is making.

INNOVATE — DOES IT SURVIVE THE AI ERA HONESTLY

The reflex is to make tasks “authentic” and assume that settles it. The honest research picture is more uncomfortable and more useful.

Authentic design is not a cheating cure. The largest study of contract cheating to date found authentic assessment did not reliably protect against it — students in authentic-task courses still outsourced work at meaningful rates (Bretag et al., 2019). Redesign helps; it is not a silver bullet, and any PD that sells it as one is overclaiming. What actually raises the cost of outsourcing is changing the evidence you collect, not relabelling the topic.

So the Innovate move in assessment design is to build tasks whose validity survives a student having every tool available:

Run your own task against AI and grade the result on your rubric. The score tells you what the task actually measures now. If the machine earns full marks, the task was measuring the artefact, not the thinking — and you know before your students do.
Collect evidence the shortcut can't produce: a draft you watched take shape, a five-minute conversation about the reasoning, a step made in front of you. This is a validity move, not a surveillance one — you are widening the construct back out to include the thinking, which underrepresentation had let slip.
Make judging AI output the assessed skill. Students mark what a machine's answer got wrong against your rubric — ruthless about a machine's work in a way they never are about their own, and no policy required. The design and the AI-literacy lesson become the same task. The full version is Grade the AI Against Your Rubric.

WHAT THE EVIDENCE DOES — AND DOESN'T — SAY

This course has a standing rule: where the workshop version of a claim outruns the research, we say so. Assessment is full of confident claims that the evidence only partly supports. Use these; just don't oversell them.

The claim you'll hear	The honest version
“Validity is a property of the test.”	Solid and foundational — but it's the opposite. Validity is a property of the claim from the score, and you build evidence for that specific claim (Messick 1989; AERA/APA/NCME 2014).
“Make it authentic and cheating goes away.”	Not supported. A large contract-cheating study found authentic tasks were not a reliable protection (Bretag et al. 2019). Change the evidence, not just the label.
“UDL is proven to raise outcomes.”	A reasonable, widely-used design heuristic — but a 2024 review found the cited evidence base mostly doesn't test UDL's specific claims (Boysen 2024). Use it as a lens, don't cite it as an outcome guarantee.
“Extended time only helps students with disabilities.”	Extended time tends to help most students; the extra edge specifically for students with disabilities is inconsistent across studies (Sireci et al. 2005; Lovett 2010). It's a broadly helpful condition, not a targeted fix — still worth giving, just don't rest a validity argument on the differential.
“Proctoring/detection catches cheaters.”	Evidence is suggestive, not settled, and comes with a real cost: surveillance conditions can depress the scores of high-anxiety students specifically (Woldeab & Brothen 2019). AI-detection scores in particular are a weak signal, not proof — noisy, and biased against multilingual writers, enough to prompt a closer look but never to ground an accusation by themselves. Design tends to beat surveillance.

Full citations, DOIs and the honesty notes behind each row are in the verified-citations record — ask and I'll send it. The point isn't to distrust the research; it's to make claims your assessments can actually cash.

Decide what the score is meant to prove, blueprint to that, write items and a rubric that measure it, score it the same way twice, and check it still measures the thinking when a machine is on the desk. That order is the whole method.

Jason Beyer · Educational Technology · Regis Jesuit High School · A.M.D.G.

HelpDesk@regisjesuit.com · 303.269.8080 · LAST VERIFIED — AUGUST 2026