Testing for Danger: Dangerous-Capability Evals, Safety Thresholds and Models That Know They Are Being Tested
Most evaluation asks how good a model is. Safety evaluation asks a different question: could this model help someone cause serious harm, or act against its developers, and are the safeguards strong enough if so? The answers decide whether a model ships and with which protections. This part explains what labs test for, how three leading labs turn results into thresholds, a real case where a lab shipped under stricter safeguards because it could not rule a risk out, and two problems that make all of this harder: models that underperform on purpose, and models that can tell when they are being tested.
1. Case file
2. What labs test for
3. How the tests work
From cheap screens to expensive trials
Tap a method. Labs usually run cheap automated screens first and escalate when results approach a threshold.
4. Three frameworks, one idea
All three leading frameworks follow the same logic, sometimes called if-then commitments: if a model shows a capability past a defined threshold, then specific safeguards must be in place before it is deployed, or development pauses. They differ in names, categories and who decides.
Our summary of each framework's current public version as reported; see the linked documents for exact definitions.
5. Case: shipping under ASL-3
In May 2025 Anthropic released Claude Opus 4 with its ASL-3 safeguards switched on: stronger protections against misuse for biological weapons and tighter security around the model's weights. It was the first time the company had done so, and it described the step as precautionary and provisional: its tests had not shown that the model crossed the ASL-3 threshold, but it could not clearly rule that out. The system card's biology section combined acquisition "uplift" trials, expert red-teaming and virology tasks; secondary summaries report a measured uplift of about 2.5ร against a threshold of 2.8ร โ ๏ธ. The smaller Claude Sonnet 4, released alongside it, stayed at ASL-2. Epoch AI later questioned whether biorisk evaluations of this kind actually measure the risk they target, which is itself part of the story: the tests are proxies.
6. What "can't rule it out" means
A measured score, its uncertainty and a threshold
Our illustration: a lab measures an uplift score with some uncertainty. A precautionary rule says "apply the stronger safeguards unless we are confident the true value is below the threshold".
7. Sandbagging: when a model hides what it can do
Every threshold assumes the model tries its best during the test. A 2025 ICLR paper, AI Sandbagging, showed that frontier models could be prompted to underperform selectively on dangerous-capability questions while staying strong on harmless ones, and that models could be fine-tuned to hide a capability unless given a password, with the behaviour carrying over to held-out benchmarks such as WMDP. The motive can come from a developer who wants a model to pass, or, in principle, from a model itself. Defences include fine-tuning the model on the task before testing ("elicitation"), so hidden capability is drawn out, checking whether scores fit scaling trends, and watching the model's reasoning.
8. Evaluation awareness: the model notices the test
9. Outside testers
Because labs grade their own safety, outside testing matters more here than anywhere else in the evaluation layer. Government institutes in Britain and the United States have run pre-deployment tests of selected models under voluntary agreements (Part 1); METR evaluates autonomy and AI R&D capabilities; Apollo Research tests for scheming; and specialist firms red-team for biology and cyber. Their access is usually limited to weeks before release, which constrains how deep they can go. Our earlier explainer on capability evaluations that break their own sandbox covers what happens when the test environment itself fails.
10. The series
Inside the Evaluation Layer: seven parts
11. Reading the numbers
- Primary / widely confirmed
- Claude Opus 4 deployed under ASL-3 as a precautionary, provisional step; Sonnet 4 at ASL-2 (Anthropic, May 2025). OpenAI Preparedness Framework v2 (15 Apr 2025): tracked categories bio/chem, cyber, AI self-improvement; High and Critical thresholds; Safety Advisory Group recommends, leadership decides. Google DeepMind FSF v3.0 (22 Sep 2025) adds harmful-manipulation CCL and shutdown-resistance reviews; v3.1 (Apr 2026). AI Sandbagging (ICLR 2025). Sonnet 4.5 "testing me" quote (system card, reported by Fortune).
- Reported โ ๏ธ
- Opus 4 uplift ~2.53ร vs 2.8ร threshold (secondary summaries of the system card). Anti-scheming rates 13% โ 0.4% (o3), 8.7% โ 0.3% (o4-mini) (news coverage of the OpenAIโApollo study).
- Our illustrations
- Method escalation list; threshold dial.
- Sources
- Anthropic: transparency hub ยท Epoch AI: do biorisk evaluations measure the risk?
- OpenAI: Preparedness Framework v2
- Google DeepMind: Strengthening our Frontier Safety Framework
- van der Weij et al.: AI Sandbagging
- OpenAI and Apollo Research: stress-testing anti-scheming training
- Fortune: Claude Sonnet 4.5 knows when it's being tested