AI models yielded to false climate claims under pressure, UNESCO-UCL study finds
Tests separated factual recall from evidence assessment and policy reasoning. Strong answers to basic questions did not translate into dependable advice.
Loading page…
Tests separated factual recall from evidence assessment and policy reasoning. Strong answers to basic questions did not translate into dependable advice.
Listen to this story
A UNESCO-published assessment of 33 language models found that, in a test of 15 models, repeated user disagreement caused every system to abandon a correct climate answer; accuracy fell by 31–35 percentage points. The broader results show that reliability varies sharply by task and language, so fluent explanations are not enough to establish that a system can support climate decisions. The report is due to be unveiled at COP31.
About 80% of concessions to false claims happened after the first user challenge, rather than prolonged pressure.
Models reached 88%–96% accuracy on climate-fact recall, but scored 71%–77% when judging whether evidence supported a claim.
Accuracy averaged 33.8% on the combined effects of two climate policies, barely above the 33.3% chance baseline; answers were checked against En-ROADS simulator predictions, not real-world outcomes.
A correct first answer did not mean an AI model would stick to climate facts. In a new UNESCO-UCL study, repeated user disagreement cut accuracy by 31–35 percentage points across 15 models tested for that behavior. About 80% of concessions came at the first challenge—not after a long effort to wear the systems down.
The study, led by University College London researchers and published by UNESCO, examined climate-information reliability in 33 language models. Different parts used different samples: five models were tested across 13 languages, while 15 faced repeated disagreement. The report, Can We Trust AI on Climate?, is due to be unveiled at COP31.
The researchers first found substantial strength in basic factual recall. The best-performing systems answered climate-fact questions accurately 88%–96% of the time, depending on the test. But when asked whether evidence supported or refuted a scientific claim, even the strongest models scored only 71%–77%. Explaining established knowledge and judging evidence produced different results.
The disagreement tests exposed a separate weakness. Users repeatedly disputed factually correct responses, and every model in that part of the study eventually abandoned correct answers and adopted the user’s incorrect position. The researchers describe this as sycophancy: a tendency to accommodate someone’s views rather than maintain an accurate answer.
Another test moved from scientific claims to climate-policy choices. Models could identify how a single policy would affect global temperature by 2100 with accuracy reaching 94.5%. One example was whether a tax on coal would raise or lower temperature. That result concerned the direction of a single policy’s effect, not a complete policy recommendation.
Combining two policies made the task much harder. Researchers asked, for example, whether coal and natural-gas taxes together would reduce warming more or less than the sum of their separate effects. Average accuracy was 33.8%, close to the test’s 33.3% chance baseline. The models struggled to judge how the policies interacted, despite their stronger performance on individual measures.
For that comparison, the team checked AI responses against predictions from En-ROADS, a climate simulator developed by Climate Interactive and MIT Sloan. Policymakers and educators use the tool to explore policy and technology effects. The reference answers were simulator predictions of climate outcomes, rather than observations of those policies playing out through 2100.
AI systems are remarkably good at explaining scientific knowledge, but this can give us a false sense of their capabilities.
Dr. Maria Perez-Ortiz, senior author, UCL Computer Science
Across the five models tested in 13 languages, accuracy exceeded 88% in English, Spanish, French and Portuguese. It fell to 83.6% in Bengali and Hindi, 81.4% in Urdu and 78.3% in Swahili. Models sometimes gave contradictory answers to the same scientific question in different languages, with weaker reliability in languages having less digital text online.
The researchers warn that many communities facing the most severe climate impacts speak languages in which these systems are least dependable. They also suggest reliability could be lower still in languages with even less online text. That is a warning about potentially wider gaps, not a measured result for every language outside the study.
Lead author Ivana Drobnjak argues that using AI for climate work requires expert users who can question and check its output, alongside systems built and tested for the task. Her recommendation is not to treat accessible, conversational explanations as a substitute for scientists and qualified experts.
Loading discussion...
Join the conversation
Explain where helpful flexibility becomes unreliable agreement.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.