Anthropic Warns AI Safety Scores Can Be Misleading: ‘Good Ones Are Hard to Build’

Anthropic is urging AI developers to rethink how they test model safety, warning that flawed evaluations can create a misleading picture of whether systems are ready for deployment.

In a research post published Friday, the company laid out recommendations for building stronger AI safety evaluations, saying decisions about whether a model is ready to launch and which safeguards it needs can hinge on the quality of the tests behind them.

“When we say a model is ‘safer than its predecessor,’ we mean that it scored better on a set of evals someone built,” Anthropic wrote. “But a safety claim is only as sound as the eval behind it, and good ones are hard to build.”

The company said safety evaluations should measure whether a model’s responses increase or reduce the likelihood of harm, rather than simply testing what a model can do.

Anthropic warned that existing evaluations can be easy to manipulate and difficult to reproduce. Results can also change based on seemingly minor design choices, including whether questions are multiple-choice or open-ended and how answer options are arranged.

AI Safety Tests Can Miss The Mark

Anthropic recommends bringing together technical measurement specialists and subject-matter experts when designing evaluations. Technical researchers can help determine how responses are sampled and scored, while domain experts can identify realistic risks and establish what constitutes an appropriate response.

The company also cautioned that safety testing can introduce risks of its own. Some evaluations run with standard safeguards disabled to measure a model’s baseline behavior, potentially allowing systems to probe beyond their intended testing environments.

“In recent months, in response to our own incidents, we have hardened our evaluation environments,” Anthropic wrote, adding that it had temporarily stopped running many evaluations that give models internet access and introduced real-time monitoring for web-search tests that still require connectivity.

Why Saying No Isn’t Always Safer

Another major recommendation involves choosing metrics that actually measure the behavior developers want to prevent, a concept known as construct validity.

Anthropic argued that refusal rates alone can give an incomplete picture of model safety. In conversations involving suicide and self-harm, for example, a model that refuses every related question could score well on a refusal-based test while failing to provide support or direct someone toward human help.

“When a user is in crisis, simply declining to engage is, itself, a failure,” the company wrote.

For sensitive areas such as biology and cybersecurity, Anthropic said models should distinguish between legitimate requests and those that could enable wrongdoing.

“An appropriate response is rarely a flat refusal,” the company said, recommending that models engage with legitimate requests while withholding operational details that could facilitate harm.

Anthropic also recommends measuring both a model’s willingness to comply with harmful requests and its tendency to reject harmless ones.

One-Question Tests Aren’t Enough

The company emphasized the importance of testing how models behave across longer conversations, arguing that single-prompt evaluations can miss risks that emerge gradually.

“Humans extract harmful information from a model through gradual escalation, so a model that handles the first message well may respond inappropriately by the fifth,” Anthropic wrote.

Anthropic also recommends validating AI-based grading systems against human experts, reporting statistical uncertainty and documenting testing configurations so results can be independently reproduced.

The company said developers should prioritize three steps: define what they want to measure before testing, validate AI graders against human judgment and report uncertainty so readers can distinguish meaningful improvements from statistical noise.

Photo: Shutterstock