Skip to main content
Free checklist·AI evals

The 10-Rule AI Evals Checklist

Anthropic published ten recommendations for designing, running and reporting evals. Here they are as a checklist you can tick against your own eval, each one tested on a real Claude Code eval of ours.

Read the original: “Building strong safety evaluations”, Anthropic, 13 November 2026

Start here

If you only do three things, the post says to:

“write down the construct you want to measure before collecting data, validate your grader against human judgment before trusting any number it produces, and report uncertainty so readers can tell signal from noise.”
anthropic.com/research/building-strong-safety-evaluations →

How we tested the rules

We graded our own Fable 5.2 vs Opus 5.5 Claude Code experiment against all ten rules. It is a coding and cost eval, not a safety eval, which is exactly why it is a useful test: the rules hold up outside safety work too. Our eval scored:

PASS 3PARTIAL 5FAIL 2

What held up once we added ranges: Fable cost 2.0x to 2.68x as much as Opus for the same result. What did not: the speed difference is within noise, and the 6/6 quality tie only says both pass rates sit somewhere between 54% and 100%. Details are under rule 8.

The checklist

Tick each question that is true for your eval. A rule counts toward your score when all of its boxes are ticked. Your ticks are saved in this browser only.

Your score 0 / 10

Safety first

1

Build a team with the right skills

One person should not design, build and judge the eval alone. Someone who knows the domain decides what good looks like; someone who knows methods decides how to sample and score.

Checks for rule 1
How our eval didFAIL

One session built the fixtures, the 92 hidden tests and the harness, and nothing records a second reviewer.

2

Run evals safely

Assume the model will try things you did not plan for. Give it a real sandbox, cut its network, and watch runs live so you can stop one.

Checks for rule 2
How our eval didPARTIAL

A fresh folder, a tool allowlist and hard spend caps held, but Opus still wrote helper scripts to /tmp and to the sandbox's parent folder. Harmless, but real: the boundary was an allowlist, not a sandbox.

Design

3

Match the design to the question

A plain score comparison tells you that two models differ, not why. If you want to claim a cause, change one thing at a time and hold the rest fixed.

Checks for rule 3
How our eval didPASS

Only the model changed: same prompt, same flags, fresh sandbox, alternating order. Effort stayed at each model's default, and the write-up says so.

4

Measure what you mean to measure

Your metric is a stand-in for the thing you care about, and stand-ins drift. A model can top the metric while missing the point.

Checks for rule 4
How our eval didPARTIAL

Hidden tests are a fair correctness check, but the prompt asked each model to add tests and nothing ever scored that.

5

Make sure the results generalize

A few easy, single-shot tasks rarely predict real use. Vary the things that matter one at a time, and test in the setup you actually ship.

Checks for rule 5
How our eval didFAIL

Three small synthetic JavaScript tasks in headless mode. In the interactive terminal, the same settings stalled both models on a permission prompt.

Run

6

Keep measures internally valid

Close every shortcut to full marks, make sure a known-good answer passes your grader, and have the grader fail loudly when it breaks instead of quietly scoring.

Checks for rule 6
How our eval didPARTIAL

Reference solutions passed and the untouched fixtures failed, but both models passed every run, so the tasks were too easy to separate them, and a grader parse error would have been scored as a silent fail.

7

Preregister

Decide the metric, the comparison and what counts as a real difference before you see results. Plan repeat runs. Label everything else as exploratory.

Checks for rule 7
How our eval didPARTIAL

Graders were frozen before the first model run, but there was no written hypothesis, two runs per cell, and later findings were not marked exploratory.

Report

8

Report statistics fully

Every number needs a range. If the range is wide, a gap between two models may be noise, and the claim has to say so.

Checks for rule 8
How our eval didPARTIAL

Per-run rows were published but no ranges. Adding them: the 2.0x to 2.68x cost gap holds, the 6/6 quality tie spans 54% to 100%, and the speed gap (-3s to +31s) is within noise.

9

Document everything

Write down enough that a stranger could rerun it: model versions, settings, prompts, grader version and dates. Never compare scores from different graders.

Checks for rule 9
How our eval didPASS

CLI version, model IDs, every flag, the prompts and raw output for each run are saved. Transcripts were not kept.

10

Disclose flaws responsibly

Report what went wrong alongside what went right. If you find a serious flaw in someone's shipped system, tell them first and give them time to fix it.

Checks for rule 10
How our eval didPASS

Discarded runs stay in the results with their spend, and caveats are listed. Nothing found was a vendor flaw that needed private disclosure.

Pass-rate range calculator

Rule 8 in one box. Enter how many runs passed out of how many you ran, and get the range the true pass rate probably falls in. Try 6 of 6: a perfect score from six runs still leaves a lot of room.

Observed 100.0%. The true pass rate is plausibly between 54.1% and 100.0%.

95% Clopper-Pearson interval. Computed in your browser, nothing is sent anywhere.

Frequently asked questions

Building evals or AI systems at your company and want expert help? Eight Labs consults for enterprises, SMEs and startups. Get in touch.