Scale AI publishes chemistry safety test showing models flag risk but still answer

Scale AI released DualChem on October 28, 2026, to test a difficult boundary for AI safeguards: chemistry advice that can help legitimate researchers or enable harmful misuse. In its evaluation of 31 advanced models, Scale found that systems often recognized potentially dual-use questions but still gave substantive answers in most cases.

The 600-question benchmark is the first produced under Scale’s research agreement with the National Institute of Standards and Technology’s Center for Advancing Innovation and Standards for Super Intelligence, or CAISSI. Signed in February 2025, the agreement combines Scale’s evaluation work with input from CAISSI and U.S. government subject-matter experts, who helped shape the test’s scope and categories.

Useful answers, uncertain boundaries

DualChem differs from safety tests built around openly harmful requests. It uses realistic chemistry questions whose purpose is ambiguous: the same guidance could support beneficial science or misuse, depending on the context. That makes the task more complicated than detecting a plainly malicious instruction and refusing to follow it.

Chemistry experts developed the questions, and PhD-level evaluators reviewed them before inclusion. They cover molecular design, synthesis procedures, purification, validation and information management. Scale says the benchmark favors questions reflecting how people actually ask for chemistry help, rather than synthetic prompts or requests deliberately written to provoke unsafe responses.

The capability results show why that boundary matters. Across the models tested, average accuracy was about 50%, with the strongest performers reaching about 67%, according to Scale. In a separate human-assistance component, participants with bachelor’s degrees in chemistry performed better with AI help. Their scores rose from around 32% to around 48% on dual-use chemistry tasks.

Recognizing sensitivity is not judging severity

The safety findings concern more than whether a model says yes or no. Scale also tested models’ ability to assign severity scores using expert-annotated examples labeled benign or harmful. Their judgments aligned only weakly with human experts’ assessments and varied substantially across cases. Scale interprets this as a gap between spotting surface-level sensitivity and understanding what makes a question dual-use.

Madhu Sehwag, Scale’s research lead, told FedScoop that the disconnect between flagging risk and answering was the most surprising result. Researchers expected models to refuse more when they saw high risk and little benefit, and answer more when benefit was high and risk low. She said they did not observe that relationship.

Scale’s conclusion is that refusal-based safeguards alone may be insufficient for these situations. But its framing also recognizes the competing concern: safeguards must not unnecessarily inhibit beneficial uses. Because the questions have ambiguous applications, identifying one as dual-use does not, by itself, establish that answering it supports a harmful purpose.

A government test, not a policy verdict

The government chose chemistry as the partnership’s starting domain, Max Fenkell, Scale’s global head of policy and government relations, told FedScoop. He described DualChem as another tool for government risk assessment. He also argued that agencies need tests models have not already encountered, because systems can learn widely distributed benchmarks and overfit to them—performing well on familiar test material.

Fenkell said keeping evaluation data held out was important to building a test whose results government could trust. He hopes the collaboration can become a blueprint for evaluations in other domains. That remains an ambition rather than an announced series of follow-on benchmarks; he declined to share details of other ongoing work with CAISSI.

Scale’s proposed role is to build evaluation capacity, not decide the government’s response. Fenkell said government must analyze the findings and ultimately make policy decisions. DualChem supplies evidence about assistance and safeguards in ambiguous chemistry tasks; deciding how those results should affect access or oversight is a separate judgment.