Scale and IMDA find AI models skip needed questions in Singapore legal test

Scale AI and Singapore’s Infocomm Media Development Authority introduced SG-LegalClarify on November 12, testing whether AI models know enough to answer a legal question—or need to ask first. Across 19 models, Scale found that approximately 39% of scenarios with missing facts received no clarifying question when web search was disabled.

The benchmark addresses a gap between explaining a law and applying it to someone’s circumstances. Singapore’s legal system draws on English common law but includes its own statutes and principles. As Scale explains, recognizing a general legal concept is not enough if a model misses the local rule or the facts that determine how it applies.

Scale uses a speeding question to illustrate the problem. Someone asking about a penalty on a Singapore expressway may leave out the vehicle type, speed and existing demerit points. Those details are needed to answer correctly. An assistant that immediately supplies a penalty can sound authoritative while giving a legally incorrect answer.

Building missing facts into the test

SG-LegalClarify is the first research initiative from Scale’s collaboration with IMDA, announced in July 2026. Legal professionals with Singapore-law expertise wrote 1,100 original questions across 10 domains, including criminal, employment, property and family law, as well as judicial processes.

Of those questions, 760 deliberately omit necessary information. Each has a paired version supplying the missing facts, producing 1,860 scenarios in total. Models can question a simulated user, letting researchers compare performance when facts must be gathered with performance when the same facts arrive upfront.

A panel of three AI judges grades responses using 12,546 expert-written criteria. The judges were selected through comparison with human expert assessments. Their scoring separates four parts of legal reasoning rather than treating every response as a single right-or-wrong answer:

• Issue identification: recognizing the legal questions and relevant variables.
• Rule recall: identifying the law that applies.
• Application: connecting that law to the specific facts.
• Conclusion: reaching the correct legal outcome.

Search helps, but does not settle the law

The study evaluated models from developers including OpenAI, Anthropic, Google DeepMind and DeepSeek, with and without web search. Scale’s results show that access to search improved the average share of mandatory grading criteria satisfied.

The highest-performing model without search met 69.8% of mandatory criteria; the highest-performing model with search met 77.8%. Recalling the applicable law remained a weakness. Even with web access, models satisfied only 56.5% of rule-recall criteria, according to Scale.

The questioning problem ran in both directions. Without search, models asked unnecessary questions in approximately 28% of scenarios that already supplied enough information. When facts were missing, they asked about an average of just 39% of the relevant facts identified by legal experts.

Asking every necessary question did not guarantee success. Among search-enabled trials with missing facts where models asked about every relevant missing fact, only 55.6% reached the correct legal conclusion. Another 32.1% reached an incorrect conclusion, while 12.3% gave no conclusion. Gathering the information and applying it correctly remained separate hurdles.

The next step is broader scrutiny of the test itself. Scale says it plans to release a public subset of SG-LegalClarify to support further research and independent analysis. That is a planned release, rather than an announcement that the subset is already available.