“They're running out of training data, and the actual capabilities of these models were peaking as far back as March 2024.”
Our reading, the sentence that was checked:
LLM developers are running out of training data, and model capabilities had been peaking since March 2024.
Assessment: Mixed Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, Deep Impact, paragraph 56.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-deep-impact-dfa5aa-a1
We recorded 50 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-01-29. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- running out of training data
- actual capabilities
- peaking as far back as March 2024
Our interpretation
- The data clause ordinarily means usable human-produced training text was becoming a constraint on continued scaling, not that every possible source of training data was already exhausted.
- The capability clause ordinarily means substantive progress had broadly flattened from March 2024 through publication.
- The target period runs from the beginning of the named month through the post date.
- Substantial gains across several difficult tasks count against a broad capability plateau even if some tasks remain flat. No fixed inference-compute budget is stated.
Counts as supported if: Credible estimates show an approaching constraint on usable human-produced training text, and comparable evaluations show broadly flattened capability progress from March 2024 through January 29, 2025.
Counts as contradicted if: Evidence shows no approaching data constraint, or substantial capability improvements after March 2024. If the two clauses receive different findings, propose mixed and identify which holds.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Mixed
Part of the claim holds and part does not, and the record says which part is which.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The data-scarcity concern has support: Epoch estimated that continued scaling would use the available stock of human-generated public text between 2026 and 2032. That is an approaching constraint, not existing exhaustion. The broad capability-plateau claim is contradicted by post-March improvements on independent abstract-reasoning tests and developer mathematics, science and coding evaluations. Gains were uneven and sometimes required more inference compute; METR's autonomy results provide an important qualification.
Limitations
- Support for data scarcity depends on the ordinary human-text reading and continuation of scaling trends.
- Synthetic and other data sources prevent treating this as exhaustion of all training data.
- Some capability evidence is developer-reported.
- Improved benchmark performance does not imply improvement on every real-world task or at fixed cost.
Evidence (4 items)
Supports the claim: Epoch AI,
If trends continue
Why it matters here: The estimate supports an approaching constraint on usable human-generated public text under continuing scaling trends. It does not show that data was already exhausted, and the discussion identifies other data sources and ways progress could continue.
- Publisher
- Epoch AI
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Goes against the claim: ARC Prize,
both outperform GPT-4o
Why it matters here: Using its baseline testing framework, ARC Prize reports public-evaluation scores of 21.2% for o1-preview and 9% for GPT-4o. This is post-March capability progress, although the evaluator also reports substantially greater inference time.
- Publisher
- ARC Prize
- Kind of source
- Independent measurement
- Describes events of
- Public from
Goes against the claim: OpenAI,
significantly outperforms
Why it matters here: Appendix A reports o1-preview improvements over GPT-4o on mathematics, science and coding, including pass@1 scores of 44.6 versus 9.3 on AIME and 73.3 versus 50.6 on GPQA Diamond. These are developer measurements, but they broaden the evidence beyond a single reasoning benchmark.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: METR,
5 tasks
Why it matters here: METR found limited success on its seven research tasks and no improvement over its best existing model on the autonomy suite. This qualifies the breadth of progress, but METR also warned that its testing could underestimate the models' capabilities.
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, Deep Impact
They're running out of training data, and the actual capabilities of these models were peaking as far back as March 2024.
- Where
- paragraph 56
- Published
- 2025-01-29
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-deep-impact-dfa5aa-a1: Current. Outcome: Mixed. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.