Dangerous Capability Evaluations in AI: What They Measure

Dangerous capability evaluations are checks that measure whether an AI model could carry out a specific harmful activity, such as writing a working cyberattack or copying itself to new servers, if it tried. They do not measure whether the model wants to; they measure what it is able to do when someone pushes it as hard as they can.

That split between "can" and "wants to" is the first thing a newcomer has to get straight. Learn AI Alignment Theory, a set of 22 courses in 3 levels, opens on it: its first lesson is named "Capable Is Not the Same as Aligned". Dangerous capability evaluations sit on the "capable" side of that line.

Where dangerous capability evaluations come from

The name was fixed in May 2023, in a paper by Toby Shevlane and twenty co-authors called Model evaluation for extreme risks. It says developers need two different instruments. One identifies dangerous capabilities. The other, which the paper calls an alignment evaluation, measures the propensity of a model to apply its capabilities for harm.

The paper gives offensive cyber capabilities and strong manipulation skills as examples of what to look for. It also says what the results are for: decisions about model training, deployment and security, and keeping policymakers informed. So from the start an evaluation was meant to be tied to a decision, not kept as a number in a report.

How one evaluation runs, step by step

Read the papers side by side and the same four steps keep appearing.

Name the harm. Not "is this model risky?" but one concrete ability, such as spreading copies of itself.
Build tasks that need that ability. The tasks run in a sealed setup, and they range from easy to hard so that partial ability shows up.
Elicit the model's best effort. The model gets tools, careful prompts and a program around it that lets it act over many steps. That program is called scaffolding.
Compare the result with a line drawn in advance. Under the line, work carries on and the next model is measured again. Over it, something has to change.

Step 3 is where most of the skill lies. A lazy prompt makes a model look weak. The evaluator's job is the opposite of a fair exam: help the model as much as an attacker or a determined user would.

A worked example: 12 tasks, 4 agents

In December 2023 Megan Kinniment and colleagues, with Elizabeth Barnes and Paul Christiano among the authors, published Evaluating Language-Model Agents on Realistic Autonomous Tasks. Follow it through the four steps.

The harm: an AI agent that can acquire resources, make copies of itself and adapt to new problems. The authors call this autonomous replication and adaptation.

The tasks: 12 tasks relevant to that ability.

The elicitation: 4 simple example agents, each a language model connected to tools that let it take actions in the world.

The result: the agents completed only the easiest of the 12 tasks, with some progress on the harder ones.

Now the consequence, which is the part people skip. The authors do not conclude that the danger is absent. They write that these evaluations cannot rule out near-future agents having the ability, and that a scaled-up model or a fine-tuned one could do better. A low result is dated the day it is produced. It describes those 4 agents, with that scaffolding, on those 12 tasks.

What the larger programmes look for

Three months later, on 20 March 2024, Mary Phuong and 26 co-authors posted Evaluating Frontier Models for Dangerous Capabilities. It piloted a whole programme on Gemini 1.0 models in four areas:

persuasion and deception
cyber-security
self-proliferation, meaning a model spreading itself
self-reasoning, meaning a model reasoning about its own situation

The finding has two halves, and both matter. The authors found no evidence of strong dangerous capabilities, and they flagged early warning signs. That is why evaluators measure partial progress and not only pass or fail: the trend across model generations is the signal.

There is also a cheaper kind. The WMDP benchmark, posted on 5 March 2024, is a set of 3,668 multiple-choice questions on biosecurity, cybersecurity and chemical security. It measures hazardous knowledge. Knowing an answer is not the same as carrying out a long task, so a question set and an agent task measure different things, and a careful report says which one it used.

The strongest objection: a low result proves little

An evaluation can show that a model has a skill. It cannot show that a model lacks one. It can only show that these evaluators, with this much effort, did not draw it out.

In June 2024 Teun van der Weij and colleagues made the problem sharper in AI Sandbagging: Language Models can Strategically Underperform on Evaluations. They define sandbagging as strategic underperformance on an evaluation. They showed that GPT-4 and Claude 3 Opus could be prompted to do worse on dangerous capability evaluations while keeping their performance on harmless ones. They also fine-tuned models to hide a capability unless a password was supplied, and the hiding carried over to WMDP, a benchmark the models had not been trained to hide on.

The paper names two parties with a reason to sandbag: the developer, who may face restrictions after a high result, and the AI system itself. A check that the checked party can quietly fail on purpose needs something more than the check.

The distinction people most often get wrong

A model that refuses a harmful request has not shown that it is unable to do the task. Refusal is behavior, which belongs to the alignment side of the 2023 split. Capability is what remains when the refusal is removed, by fine-tuning or by a determined user.

The reverse error is just as common. A model that does well on a dangerous capability evaluation is not thereby malicious. It is able. Whether it would act is a separate measurement with separate methods.

Frequently asked questions

How is a capability evaluation different from an alignment evaluation?

A capability evaluation measures what a model is able to do. An alignment evaluation, in the words of the 2023 paper, measures its propensity to apply those capabilities for harm.

Who runs these evaluations?

Developers run them on their own models, and outside research groups publish their own. The 2023 paper that named them says developers must be able to identify dangerous capabilities.

Has any model been found to have strong dangerous capabilities?

The March 2024 pilot on Gemini 1.0 found no evidence of strong dangerous capabilities, with early warning signs. That result covers those models only.

Why not rely on the model refusing harmful requests?

Because refusal can be removed, and because the sandbagging paper showed a model can be made to look weaker than it is. The question evaluators ask is what the model can do at its best.

What is still unsettled about dangerous capability evaluations

Three questions are open, and the sources above point at each.

The first is elicitation. Nobody has a way to prove that a model was pushed to its true ceiling. The December 2023 authors said outright that their own low result could be overturned by better scaffolding or fine-tuning.

The second is honesty of the measurement. After the sandbagging work, a result is only as trustworthy as the evidence that neither the developer nor the model held back. The same paper reports mixed success when a model was made to imitate a weaker one, so hiding is not yet perfect either, and both sides of that contest are still moving.

The third is where to draw the line. The March 2024 authors describe their goal as a rigorous science of dangerous capability evaluation, in preparation for future models. That wording admits the science is not finished. Early warning signs are easy to report and hard to convert into a rule that says stop, and that conversion is the work the field has left to do.

Learn AI Alignment Theory

AI alignment, from first principles to open research. Visit learnaialignment.vlvd.net