How to score a design exercise
By Sergio Gualda · 9 min read · Updated
Score a design exercise against written anchors rather than impressions. Define three to five criteria, describe what each score level looks like in observable behaviour, fix the weights before you read anything, and require a quote from the candidate's work to justify every score. Then calibrate on people whose ability you already know.
Write the rubric before you send the exercise
If you can't describe what a strong answer looks like before anyone has answered, the exercise isn't ready. Writing the rubric first is also the cheapest way to discover that your prompt doesn't draw out the thing you care about.
A useful test: if the same answer would always get the same score on two of your criteria, you have one criterion, not two.
Anchors describe behaviour, not quality
The most common failure is a rubric whose levels are “excellent / good / adequate / poor”. That's a scale with adjectives, and two reviewers will apply it differently within minutes.
An anchor should describe something you can observe. Compare “excellent problem framing” with “rejects the brief as posed, reframes it in terms of the stated outcome, and cites at least two pieces of data from the material”. Only the second can be applied consistently.
- 3: Rejects the brief as posed and supports the reframe with two or more specific data points.
- 2: Notices the brief is wrong but argues from intuition, or uses a single data point.
- 1: Executes the brief and adds an improvement of their own.
- 0: Executes the brief literally. Never questions the premise.
Fix the weights before you read anything
Weights decide what the exercise is actually measuring. Decide them after reading answers and the strongest candidate ends up defining the rubric.
Weight towards what fails hires. In most product roles, misjudging the problem causes more damage than executing imperfectly, so problem framing should usually carry more weight than craft.
Require evidence for every score
Make the reviewer paste a quote from the candidate's answer next to each score. It takes a few extra minutes and changes the review completely.
Scores have to be grounded in what was written, not in an impression formed in the first two minutes. Disagreement between reviewers becomes productive, because you're arguing about a particular sentence. And you end up with feedback you can give the candidate.
Calibrate before you use it on strangers
Take three people you know are strong and three you know are weak. They should be people you've worked with, not reputations. Have them do the exercise, have someone anonymise the answers, then score them blind.
The rubric passes if the three strong ones score above the three weak ones with no overlap, and by a clear margin. If a weak one outranks a strong one, the problem is your rubric. Fix it and repeat.
Almost nobody does this, which is why almost every internal rubric ends up measuring presentation without anyone noticing.
Three failure modes to watch for
These show up in nearly every rubric written in a hurry.
- Scoring effort. Length and polish are the easiest things to see and the least predictive. Say explicitly that they don't count.
- Halo from the first section. Read all candidates' answers to section one, then all of section two, rather than one candidate end to end.
- Criteria that measure the same thing twice, which silently doubles that criterion's weight.
Frequently asked
- How many criteria should a rubric have?
- Three to five. Beyond that, reviewers stop distinguishing between them and start spreading a general impression across all of them, which defeats the purpose.
- Should candidates see the rubric?
- Yes. There's no correct answer to leak, and knowing what's measured doesn't let anyone fake it, because they still have to do it. It does remove the sense of ambush, which is the main reason strong candidates decline exercises.