“The ASU team showed that exactly the same thing was true even in the latest, greatest models. Throw in every gadget invented since 1998, and the Achilles’ Heel I identified then still remains.”
Our reading, the sentence that was checked:
The ASU study showed that even the latest, greatest models cannot reliably extend universals outside their training space of examples, so the Achilles' heel Marcus identified in 1998 still remains despite every gadget invented since.
Assessment: Contradicted Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it., paragraph 66.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-gpt-5-overdue-overhyped-and-underwhelming-e54d36-a1
We recorded 31 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-08-09. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- even in the latest, greatest models (scope, stated by the speaker)
From the speaker
- The ASU team showed
- exactly the same thing
- even in the latest, greatest models
- every gadget invented since 1998
- the Achilles’ Heel I identified then still remains
Our interpretation
- This assesses what the ASU study available before the post actually demonstrated.
- The target period covers the study's initial publication through the post, August 2–9, 2025.
- The plain reading requires empirical results on contemporary leading models showing unreliable application of learned rules beyond training examples.
- Every gadget is rhetorical emphasis on modern methods, not a literal assertion that every invention was tested.
- The claim is about this study's demonstrated scope, not whether generalization failures exist elsewhere.
Counts as supported if: The available ASU study tests contemporary leading models and demonstrates unreliable generalization beyond their training examples.
Counts as contradicted if: The study's methods establish that its experiments used a materially different model class, with application to contemporary leading models asserted rather than demonstrated.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Contradicted
The dated evidence goes against what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The paper demonstrates generalization weaknesses in its controlled experimental models. It does not demonstrate them in the latest leading models. Its authors explicitly distinguish their experiments from their proposed extension to large pretrained systems. Marcus overstates what this particular study tested.
Limitations
- This does not establish that frontier models generalize reliably.
- The authors themselves make broad claims about the implications.
- The comparison with Marcus's 1998 work is not assessed as a claim of scientific priority.
Evidence (1 item)
Goes against the claim: arXiv; Arizona State University authors,
trained from scratch
Why it matters here: The experiments use a small GPT-2-style model, with four layers and hidden dimension 32 in the default configuration. The authors extend their interpretation to large pretrained models, but do not experimentally test the contemporary frontier systems Marcus describes.
- Publisher
- arXiv; Arizona State University authors
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
The ASU team showed that exactly the same thing was true even in the latest, greatest models. Throw in every gadget invented since 1998, and the Achilles’ Heel I identified then still remains.
- Where
- paragraph 66
- Published
- 2025-08-09
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-gpt-5-overdue-overhyped-and-underwhelming-e54d36-a1: Current. Outcome: Contradicted. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-gpt-5-overdue-overhyped-and-underwhelming-e54d36-a1.