“The expectation that generative AI will transform into something much, much more powerful requires you to first ignore the existing limitations, believing it to be more capable than it is, and also ignore the fact that these models have yet to show meaningful improvement over the past few years.”
Our reading, the sentence that was checked:
Generative AI models have not shown meaningful improvement over the past few years.
Assessment: Contradicted Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, Reality Check, paragraph 64.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-reality-check-47ac05-a1
We recorded 20 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-04-28. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- these models
- have yet to show meaningful improvement
- over the past few years
Our interpretation
- I read 'the past few years' as approximately 2022 through the post date, April 28, 2025.
- The plain reading is that generative models made no substantial capability gains. Meaningful improvement includes substantial gains on comparable reasoning, coding, and task-completion evaluations; it does not require eliminating hallucinations or replacing whole jobs.
- I assess the historical assertion, not the surrounding argument about expectations for future AI.
Counts as supported if: Comparable evaluations across the period show broad stagnation, with apparent gains small, unreliable, or explained by changes in testing.
Counts as contradicted if: Credible longitudinal evaluations show substantial gains in important capabilities, including tasks with practical relevance.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Contradicted
The dated evidence goes against what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
On the ordinary capability reading, substantial improvement had occurred before the post. METR measured sustained gains in task-completion ability, and Stanford documented large benchmark gains. Persistent unreliability is real, but it does not establish stagnation. A narrower claim that models had not become dependable replacements for workers would require a different assessment.
Limitations
- The result depends on interpreting 'meaningful improvement' as substantial capability gains.
- Benchmark progress does not establish proportional improvements in workplace productivity.
- Some comparisons measure complete systems, including tools and prompting.
Evidence (3 items)
Goes against the claim: METR researchers, published on arXiv,
doubling
Why it matters here: The original evaluation finds substantial growth in the duration of software and reasoning tasks agents can complete with 50% success. Its historical comparisons include the years preceding the post, although this is a restricted task distribution.
- Publisher
- METR researchers, published on arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Goes against the claim: Stanford Institute for Human-Centered Artificial Intelligence,
71.7% in 2024.
Why it matters here: The report records substantial improvement on software issue resolution and also reports gains on multimodal understanding and science questions. These are system-level benchmark comparisons, so they do not isolate changes in model weights from changes in surrounding tools.
- Publisher
- Stanford Institute for Human-Centered Artificial Intelligence
- Kind of source
- Secondary summary
- Describes events of
- Public from
Context only: METR,
unable to reliably handle
Why it matters here: The same researchers emphasize that substantial measured progress coexists with poor reliability on ordinary autonomous work. This supports the surrounding concern about limitations, but does not establish an absence of improvement.
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, Reality Check
The expectation that generative AI will transform into something much, much more powerful requires you to first ignore the existing limitations, believing it to be more capable than it is, and also ignore the fact that these models have yet to show meaningful improvement over the past few years.
- Where
- paragraph 64
- Published
- 2025-04-28
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-reality-check-47ac05-a1: Current. Outcome: Contradicted. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.