“A new paper with a hard coding benchmark called LiveCodeBenchPro designed to resist data contamination shows still more converging evidence for the two core notions behind the Apple paper (and behind my own decades-long critique): (a) the systems are challenged at reasoning, and (b) performance declines as one strays further and further from familiar problems.”
Our reading, the sentence that was checked:
LiveCodeBenchPro gives further evidence that LLMs are challenged at reasoning and that their performance declines the further problems are from familiar ones.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Five quick updates about that Apple reasoning paper that people can’t stop talking about, paragraph 10.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-five-quick-updates-about-that-apple-4748af-a1
We recorded 9 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- A new paper
- LiveCodeBenchPro
- designed to resist data contamination
- the systems are challenged at reasoning
- performance declines
- further and further from familiar problems
Our interpretation
- Assess the findings reported in the original June 2025 paper available before the post.
- The plain reading is comparative: models have more difficulty with problems requiring original insights than with familiar algorithmic patterns.
- Treat the paper's knowledge-heavy versus observation-heavy categories as an indirect measure of familiarity. They do not measure distance from each model's training examples.
- Converging evidence means evidence consistent with the claim, not proof of a universal or causal law.
Counts as supported if: The paper reports reasoning-related failures and worse performance on problems demanding novel insights than on problems built from familiar patterns.
Counts as contradicted if: The paper reports no relevant reasoning weakness or shows equally strong or stronger performance on the less template-based problems.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The reported reasoning failures and weaker results on insight-driven problems support the ordinary comparative reading. The paper does not establish a measured, continuous relationship between training-data familiarity and performance.
Limitations
- Problem categories are an indirect measure of familiarity.
- The detailed failure analysis covers one model.
- These results do not show that models lack all reasoning ability.
Evidence (2 items)
Supports the claim: Zihan Zheng and colleagues, arXiv,
Knowledge-heavy problems are comfort zones for LLMs.
Why it matters here: The category analysis reports stronger performance on reusable algorithmic patterns and weaker performance on tasks requiring novel observations.
- Publisher
- Zihan Zheng and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Zihan Zheng and colleagues, arXiv,
LLMs struggle with case work.
Why it matters here: The authors identify missed edge cases and conceptual failures. Their detailed human comparison concerns o3-mini.
- Publisher
- Zihan Zheng and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Five quick updates about that Apple reasoning paper that people can’t stop talking about
A new paper with a hard coding benchmark called LiveCodeBenchPro designed to resist data contamination shows still more converging evidence for the two core notions behind the Apple paper (and behind my own decades-long critique): (a) the systems are challenged at reasoning, and (b) performance declines as one strays further and further from familiar problems.
- Where
- paragraph 10
- Published
- 2025-06-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-five-quick-updates-about-that-apple-4748af-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-five-quick-updates-about-that-apple-4748af-a1.