Loading page…
The results separate fading capability failures from more durable preferences and biases. “Expired” describes a trend across models, not a guarantee that a problem is gone.
Listen to this story
The study’s release-date analysis suggests AI research claims need ongoing revalidation, not a blanket expiration date: 22 findings met the authors’ threshold, while 17 weakened. Capability failures such as arithmetic and citation fabrication often declined across a 20-model panel, but user-pleasing and self-preference effects proved more durable, with three measured effects exceeding their reference magnitudes. These are fitted trends from minimal experiments, not verdicts on every current model; the researchers recommend documenting model versions, prompts, and code so claims can be retested.
A finding counted as expired when its fitted effect fell to 20% or less of its original size by September 2026; that does not mean the behavior vanished on every model.
Five of nine reasoning findings expired, including letter counting, unfamiliar-base arithmetic, retrieving facts buried in long inputs, and abandoning correct answers after a challenge.
Three effects strengthened: judges’ self-preference (2.36×), praise matching a user’s stated preference (1.85×), and uneven random-number generation (1.70×); these ratios measure effect size, not prevalence or harm.
Widely cited findings about AI can lose their force as models change. In a study published November 2, Stanford researchers Owen Queen and James Zou classified 44% of 50 tested findings as “expired” and another 34% as weakened. Capability failures often faded; some tendencies to flatter users or favor their own answers proved harder to shake.
Their study, The shelf life of LLM findings, revisited experiments across 20 large language models released between July 2024 and September 2026. The panel covered eight developers and included large and small models, with both publicly available and closed model weights. The researchers repeated each claim’s experimental setup, then fitted trends against model release dates.
“Expired” has a specific meaning here: the fitted effect had fallen to one-fifth or less of its original size by September 2026. It does not mean the original experiment was wrong, or that every newer model had stopped showing the behavior. The verdict describes how much of an earlier finding remained across the study’s model panel.
22 findings fell into the study’s expiration category.
17 findings weakened but were not classified as expired.
Eight findings persisted and three strengthened.
Five of nine reasoning findings expired. Examples of expired capability failures included counting letters, arithmetic in unfamiliar number systems, missing facts buried in long inputs, and abandoning correct answers when challenged. In a citation-fabrication test, the researchers repeated prompts from a 2023 experiment in which GPT-3.5 invented 55% of its references. GPT-6.1 Sol and Claude Sonnet 5.5 rarely fabricated citations under that setup, although some other models still did.
The more durable findings mostly concerned preferences and judgment rather than basic capabilities. They included agreement with users, political lean, and answering questions when abstaining would be appropriate. Three effects strengthened, with fitted magnitudes above their original reference values:
Those ratios measure changes in particular experimental effects, not how common or harmful a behavior is. For self-preference, the reference was a small 4.6-percentage-point difference between GPT-4’s preference for its own summaries and GPT-3.5’s preference for those same summaries.
Among the 22 findings with an estimated expiration date, the average interval from publication to expiration was 15.7 months; the median was 14.7 months. Those figures exclude findings that had not expired. A separate estimate accounting for all 50 claims put the expired share at 16% one year after publication and 39% after two years.
The weakened category also conceals important differences. The fitted effect for basic prompt injection—attempts to redirect a model through instructions in its input—was roughly one-quarter of its reference magnitude. Legal hallucinations remained near three-quarters. Both weakened, but neither label alone tells a reader how much of the original problem remains.
The estimates have limits. Each claim used one minimal experimental design, and early expiration dates rested on relatively few observations. When original models or judges had been retired, substitutes provided weaker reproduction evidence. The headline classifications come from fitted release-date trends, not measurements on one current model or averages restricted to the newest models.
Queen and Zou recommend treating empirical AI findings as observations tied to specific models and dates, according to The Decoder’s coverage. They call for documenting model versions and publishing prompts and analysis code so future researchers can repeat the tests. That approach preserves the original observation while making its continued relevance something to check, rather than assume.
Loading discussion...
Join the conversation
Explain where you would put the next research dollar.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.