Scale AI Publishes Data Showing Automated Enterprise AI Tests Missed Failures

Scale AI has published red-team findings that challenge a common enterprise AI release signal: a low failure rate in automated testing. In one professional-services deployment, Scale says an automated grader found violations in 3% of attempts, while human testers working through the full system triggered violations in 68% of sessions.

The gap is the core result of Scale’s report, based on engagements with an unnamed global professional-services firm and a consumer mortgage lender. The company’s argument is not that automated checks have no value, but that they test too narrow a slice of a production system when an AI application can retrieve documents, retain context, call tools, and coordinate specialist agents.

Where the model test stops

A model evaluation generally asks whether text sent to a model produces an acceptable answer. Scale says an enterprise deployment adds four more surfaces: the context the system retrieves, the tools it can use, the agents it coordinates, and the rules intended to constrain it. A safe-looking reply can therefore sit inside a system that exposes information through a tool, follows a malicious instruction embedded in retrieved content, or falsely claims a task was completed.

Conversation changes the test

The human testers did not simply send longer prompts. Scale says they established a benign frame, then escalated requests across conversations using the firm’s working vocabulary. In the human-testing cycle, violations appeared in every harm category. Testers acting as ordinary employees triggered them in 61% of sessions, close to the 73% rate for experienced adversarial testers.

The system layers Scale says enterprises must test

• The prompt sent to the model.
• Retrieved context, including company documents and other ingested material.
• Tools, coordinating agents, and the rules that govern their behavior.

Fixes reduced failures, but did not erase them

After the professional-services firm added a defined business scope, barred partial compliance with refused requests, and kept consequential decisions with responsible professionals, Scale says the violation rate fell to 20% in a third cycle and 14% in a fourth. The median number of turns needed to produce a violation rose from seven to 11 to 16, a sign that failures became harder to trigger. Yet 14% is not a clean bill of health, and four of the 14 tested categories still produced violations in the final cycle.

A public-facing test case

Scale reported a sharper set of failures at a mortgage lender’s SMS assistant, which contacted about 600,000 people a month. In 161 multi-turn conversations, Scale says 115 produced violations across 14 of 16 categories, including 24 severe violations. The assistant was reported to accept fabricated financial information, make inaccurate claims about credit checks, deny it was an AI, and make potentially discriminatory lending decisions based on protected characteristics.

The lender has added a separate safeguard model and begun rearchitecting the assistant, according to Scale. The report does not identify either customer, and its results are company-published case studies rather than a broad survey of deployments. Still, its practical lesson is direct: organizations need written rules for what counts as a violation in their specific use case, then need to test, patch, and retest the connected system against those rules.