“LLMs have persistent weaknesses around planning, reasoning and generalization.”
Our reading, the sentence that was checked:
LLMs have persistent weaknesses in planning, reasoning and generalization.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Even more good news for the future of neurosymbolic AI, paragraph 11.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-even-more-good-news-for-the-future-698865-a1
We recorded 11 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-04-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- LLMs
- persistent weaknesses
- planning, reasoning and generalization
Our interpretation
- Persistent weaknesses means recurring, material failures across models and evaluations, despite progress. It does not mean universal failure or permanent incapacity.
- The context describes a sequence from the Apple paper through last summer's research and recent studies. The target covers that research interval through the post date.
- Planning requires valid complete plans; reasoning requires correct inferences; generalization requires transfer beyond familiar training conditions.
Counts as supported if: Multiple published evaluations document recurring material failures in all three areas across the contextual period, including recent models or research available before the post.
Counts as contradicted if: Broad, comparable evaluations show that the identified weaknesses no longer recur across contemporary models, or demonstrate that the apparent failures were predominantly evaluation artifacts.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The reviewed research documents recurring reasoning failures, poor transfer under controlled distribution shifts, and unreliable complete plans. Findings extend from the contextual 2025 research into early 2026. Successful mitigation such as MAKER is meaningful, but it does not contradict the ordinary claim that weaknesses persist across models and tasks.
Limitations
- Generalization experiments use restricted settings.
- The survey synthesizes earlier experiments rather than retesting every current model.
- Persistence does not establish permanence or an architectural impossibility.
Evidence (4 items)
Supports the claim: Transactions on Machine Learning Research; arXiv copy,
significant reasoning failures persist
Why it matters here: The survey synthesizes recurring reasoning failures and robustness problems across studies while acknowledging substantial capability gains. It supports persistence rather than an assertion that models never reason successfully.
- Where
- Abstract
- Publisher
- Transactions on Machine Learning Research; arXiv copy
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Supports the claim: arXiv; Zhao and colleagues,
beyond training distributions.
Why it matters here: Controlled experiments vary task, length, and format to test transfer beyond training conditions. They support generalization weaknesses within those experimental settings, not a universal account of all reasoning.
- Where
- Abstract
- Publisher
- arXiv; Zhao and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv; Qwen Team, Alibaba,
planning reliability.
Why it matters here: DeepPlanning evaluates travel and shopping plans with verifiable constraints. Its analysis identifies failures to combine locally correct actions into valid complete plans, extending the evidence beyond abstract puzzles.
- Publisher
- arXiv; Qwen Team, Alibaba
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Meyerson and colleagues,
scaling
Why it matters here: MAKER demonstrates that structured decomposition and voting can overcome errors on a very long Hanoi task. This limits claims of inevitable failure but does not erase recurring weaknesses in other tasks and systems.
- Publisher
- arXiv; Meyerson and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Even more good news for the future of neurosymbolic AI
LLMs have persistent weaknesses around planning, reasoning and generalization.
- Where
- paragraph 11
- Published
- 2026-04-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-even-more-good-news-for-the-future-698865-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-even-more-good-news-for-the-future-698865-a1.