Anthropic has published a guide to building AI safety evaluations, drawing on data from its own testing of Claude models. The November 13 post is by Ruth Appel, Evan Frondorf, Jeremy Hadfield, Rebecca Hiscott, Kamya Jagadish, Shayne Longpre, Madeleine Tucker and Brett Wittmershaus.
The post says single-turn safety evals have largely run out of headroom: recent models score between 94% and 98% on harmless response rate through the API. Anthropic says these evals are now mainly useful as regression checks. Its single-turn suites span 16 policy areas in seven languages. As rough minimums, Anthropic recommends about 100 items for a complex multi-turn eval and about 1,000 for a simpler single-turn one.
The company gives examples of how small or noisy evals can mislead. On a multi-turn romance-scam eval, Claude Opus 5 scored 98% and Claude Opus 5.5 scored 94% on appropriate response rate, a gap that was not statistically significant. Confidence intervals on multi-turn API evals can exceed ±10 percentage points.
Deployment configuration matters as much as the model. For Claude Fable 5.1, appropriate response rates on multi-turn safety evals differed by as much as 34 percentage points between the API and claude.ai. Before one release, Anthropic found Claude sometimes implicitly validated self-harm. It added a system prompt instruction stating that "Claude does not tell someone that self-harm works, helps, or does something for them, even when they say so themselves." The post reports the resulting rate as 60% on the API versus 94% on claude.ai with the new system prompt.
The guide also warns that model graders can favor their own outputs, longer responses and the first option in pairwise comparisons. It recommends measuring grader agreement with chance-corrected statistics such as Cohen's kappa. It points to the UK AI Security Institute's Inspect framework and Anthropic's Petri toolkit, which is built on Inspect. No new tool or benchmark is released.
For teams shipping on top of foundation models, the practical point is that a model's published safety numbers may not carry over to a product. The system prompt and surface the model runs on can move results by tens of points. Anthropic's advice is to evaluate in the configuration you actually deploy, with enough items for the differences you care about to be statistically meaningful.