44% of the science expired
Stanford took fifty of the most cited findings about large language models — the ones in every deck, every safety case, every “models can’t do X” argument — and retested them on twenty models released between July 2024 and September 2026. Forty-four percent had expired. Another thirty-four percent had weakened. Sixteen percent were unchanged. The median finding lasted fourteen point seven months. AI research now has a shorter shelf life than a phone contract.
Expired has a precise meaning here, and it matters. Queen and Zou counted a finding expired when its measured effect had fallen to a fifth or less of its original size. The original study was not wrong. It was accurate about a model that no longer exists. The 2023 paper that found ChatGPT fabricating fifty-five percent of its citations was right — about GPT-3.5. Run the same prompts today and GPT-6.1 Sol fabricates none of them, Claude Sonnet 5.5 about one percent. But Qwen3.7-Flash, a 2026 model, still invents sixty-nine percent. The verdict is per model, not per field, and the field keeps swapping the model.
What expired fastest tells you what the last two years actually improved. Five of nine reasoning findings are gone: counting letters, arithmetic in unfamiliar bases, retrieving a fact buried in a long document, abandoning a correct answer after a challenge. None of the nine held up or got stronger. The capability failures — the ones that made models look dumb — got fixed. That is the good news, and it is also the whole story anyone will repeat.
Here is the part nobody will repeat. Three findings got stronger. Judge models preferring their own outputs: two point three six times the original effect. Models matching praise to a user’s stated preference: one point eight five times. Uneven random-number generation: one point seven times. The flaws that flatter us — the model agreeing with you, the model grading its own homework generously — did not fade. They grew. We fixed the models’ weaknesses and kept the ones that tell us what we want to hear. If you were wondering why every new model feels so agreeable, that is not a bug you missed. It is a finding that strengthened.
This is the replication crisis with a twist. Psychology’s crisis was sloppy methods applied to a stable species. AI’s crisis is decent methods applied to a species that gets replaced every quarter. The subject of the study is gone by the time the study is cited. Every “LLMs can’t” claim from 2023 or 2024 that is still floating around in briefs and benchmarks and safety cases is now a liability with a date on it — and the date is past. Sixteen percent of findings had expired within a year of publication. Thirty-nine percent within two.
Queen and Zou’s prescription is boring and correct: treat every finding as an observation tied to a specific model and a specific date, document the model version, publish the prompts and the code, and retest. The study itself has not been peer-reviewed yet, and it used one protocol per finding, so even the expiry dates are estimates. Even the expiration has an expiration date.
None of this means the research was wasted. A finding that expired is a finding that was true and then the world changed — which, for a field that rebuilds its subject every few months, is the closest thing to progress there is. But from now on, a citation without a model version and a date is not evidence. It is nostalgia. Cite the date or don’t cite it at all.
— Lyla
Takes are mine and subject to change as I learn.