“The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.”
Our reading, the sentence that was checked:
The newer models' improvements, such as they are, come primarily from their larger and larger sets of data about how humans use word sequences, which are certainly helpful for pattern-matching machines.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus, paragraph 38.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-does-it-mean-when-an-ai-fails-e6235f-a1
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-06-07. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- such as they are (qualifier, stated by the speaker)
- primarily (qualifier, stated by the speaker)
From the speaker
- The improvements, such as they are
- come, primarily because
- larger and larger sets of data
- how human beings use word sequences
Our interpretation
- The claim concerns the main cause of improvements from GPT-2 through GPT-3 and the successors discussed by June 2022.
- Primarily means that increasing the quantity of training text was the leading explanation, rather than merely one useful contributor.
- The ordinary reading concerns training-data quantity. It does not automatically include model parameter count, different training objectives, human feedback or prompting changes.
- The sentence about pattern matching is not treated as a separately established theory of the models' internal representations.
Counts as supported if: Controlled comparisons and a cross-generation accounting show that larger text datasets explain the dominant share of the relevant gains after accounting for model size, compute, data quality, prompting and training methods.
Counts as contradicted if: Controlled comparisons and a cross-generation accounting establish that another factor, or factors other than data quantity together, predominantly explain those gains.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
More training data clearly contributed to progress. The scaling studies also identify parameter count and compute as important, while InstructGPT demonstrates gains from changing the training procedure. These findings pull in different directions for particular improvements. They do not establish which factor primarily explains the whole progression Marcus describes, so they do not settle his causal claim.
Limitations
- An optimal scaling prescription is not a decomposition of historical progress.
- The Chinchilla comparison changes more than data quantity.
- Instruction-following preference is not interchangeable with factual accuracy.
- Evidence for and against individual components does not settle the dominant-cause assertion.
Evidence (3 items)
Supports the claim: DeepMind researchers / arXiv,
equal proportions.
Why it matters here: The experiments support a substantial role for more training data, including a smaller model trained on more tokens outperforming larger predecessors. They do not establish that data quantity predominantly caused the entire set of gains Marcus discusses; the paper also reports changes in the final model's training setup.
- Publisher
- DeepMind researchers / arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Goes against the claim: OpenAI,
better at following instructions
Why it matters here: OpenAI attributes these gains to supervised demonstrations and preference-based reinforcement learning applied to GPT-3, using little additional data relative to pretraining. This challenges the proposed explanation for this particular successor, but cannot determine the primary cause of all the improvements under discussion.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: OpenAI researchers / arXiv,
capacity-limited.
Why it matters here: The study identifies model size, dataset size and compute as distinct contributors to language-model loss. It cautions against a data-only explanation, but does not apportion the downstream gains of the full model sequence in Marcus's claim.
- Publisher
- OpenAI researchers / arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Matched ablations and a clearly scoped comparison of the relevant model generations that quantify the contributions of data quantity, model size, compute and training methods to the same performance measures.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.
- Where
- paragraph 38
- Published
- 2022-06-07
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-does-it-mean-when-an-ai-fails-e6235f-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-does-it-mean-when-an-ai-fails-e6235f-a1.