How fast do things we know about AI stop being true?
You have heard the confident statements. Models cannot count letters. They lose facts in long documents. They just tell you what you want to hear. Researchers at Stanford decided to check whether any of that still holds.
They took 50 widely cited claims and re-tested them on 20 models released between mid 2024 and September 2026. The results: 44% of the claims had expired, another 34% had weakened, 16% still held, and 6% had actually got stronger.
So how long does a finding about AI actually stay true?
For the claims where an expiry date could be estimated, the average lifespan was about 15.7 months. Barely more than a year.
What dies fast, what sticks
The things that disappear quickest are plain failures of capability, like miscounting letters or losing details in a long text. Models simply get better at them. The things that hang around are about preferences: flattering the user, favouring their own kind of answer, leaning one way politically. Those are not bugs that more power fixes.
The takeaway is a bit humbling. Anything you read about "what AI can't do" is a snapshot of one model at one moment. The authors' advice to fellow researchers is the one everyone should borrow: always write down which model version you tested.