“Task-level performance, even when impressive, is not sufficient evidence of general intelligence.”
Our reading, the sentence that was checked:
Impressive task-level performance is not sufficient evidence of general intelligence.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Rumors of AGI’s arrival have been greatly exaggerated, paragraph 9.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-rumors-of-agis-arrival-have-been-9921c1-a1
We recorded 18 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-02-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Task-level performance
- even when impressive
- is not sufficient evidence of general intelligence
Our interpretation
- The ordinary reading is that high performance on individual fixed tasks alone does not establish flexible transfer across broad unfamiliar tasks.
- This does not say task performance is irrelevant, or that sufficiently broad and well-controlled behavioral evaluations cannot provide evidence of intelligence.
- Treat this as a claim about the evidentiary relationship illustrated by research available through publication, rather than a claim restricted to one current model.
Counts as supported if: Measurement research or controlled experiments show that strong performance on tested tasks can coexist with failure to transfer the relevant knowledge or skill.
Counts as contradicted if: Task scores alone reliably establish the specified broad generalization, without additional assumptions, controls, or transfer evaluations.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Measurement research distinguishes acquired task skill from the capacity to generalize. Experiments also show substantially stronger performance on one formulation than on a closely related transfer task. These findings support the limited claim that impressive task performance alone is insufficient. They do not establish that current systems lack all generalization, or that behavioral testing cannot establish general intelligence.
Limitations
- The empirical example concerns particular systems and tasks, not every model available in February 2026.
- The identified transfer failure can be mitigated.
- The conclusion applies to task performance alone, not to every possible collection of controlled evaluations.
Evidence (3 items)
Supports the claim: arXiv,
High skill is not high intelligence
Why it matters here: The paper explains how prior knowledge and training can produce task skill without establishing the ability to acquire new skills. This supports the ordinary distinction in the claim, within an explicitly stated measurement framework.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv,
compared to 33% for the latter.
Why it matters here: Accuracy drops substantially when the direction of a factual relationship is reversed. Together with the paper's controlled training experiments, this demonstrates why success on one task does not establish even closely related transfer.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv,
infer unseen reverse facts.
Why it matters here: Training that induces suitable relational representations improves transfer. This limits any inference that language models inherently cannot generalize, but does not make task performance alone sufficient evidence of general intelligence.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Rumors of AGI’s arrival have been greatly exaggerated
Task-level performance, even when impressive, is not sufficient evidence of general intelligence.
- Where
- paragraph 9
- Published
- 2026-02-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-rumors-of-agis-arrival-have-been-9921c1-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-rumors-of-agis-arrival-have-been-9921c1-a1.