“The important problem is that they sometimes claim to they have solved problems that they have not properly solved; as with hallucinations, this reflects a deeply problematic failure to sanity check their own work.”
Our reading, the sentence that was checked:
LLMs sometimes falsely claim to have solved problems, reflecting a failure to sanity-check their own work.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Reports of LLMs mastering math have been greatly exaggerated, paragraph 15.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-reports-of-llms-mastering-math-have-2bc2e7-a1
We recorded 11 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-04-05. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- sometimes
- claim to they have solved problems
- have not properly solved
- failure to sanity check their own work
Our interpretation
- The plain reading is behavioral: models sometimes present materially flawed proofs as completed solutions.
- Failure to sanity-check means that errors remain undetected or uncorrected in the delivered answer. It does not establish that no internal checking was attempted.
- Deeply problematic is an evaluative judgment and is not separately scored.
- The surrounding discussion concerns the March 2025 USAMO evaluation. I use March 2025 as the latest completed monthly observation period before the post.
Counts as supported if: An evaluation pairs model outputs presented as solved with independent mathematical grading that identifies substantive errors or gaps.
Counts as contradicted if: The relevant alleged examples are shown to be valid solutions or explicit acknowledgments of failure, and a comprehensive review of the defined evaluation finds no false claims of completion.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The original March 2025 evaluation directly reports models claiming completion while expert graders identified substantive proof defects. That establishes the modest claim that this sometimes happens. It supports a failure to catch errors in delivered work, without establishing what internal checking the models attempted.
Limitations
- The evidence concerns difficult Olympiad proofs and the tested model versions.
- It does not establish a failure rate across ordinary questions or later models.
- The severity judgment is not independently scored.
Evidence (1 item)
Supports the claim: arXiv,
claimed to have solved the problems.
Why it matters here: The study pairs claims of completion with expert findings of flawed logic, unjustified steps and other proof defects. Its independent double-grading procedure supports the limited claim that models sometimes deliver incorrect work as solved.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Reports of LLMs mastering math have been greatly exaggerated
The important problem is that they sometimes claim to they have solved problems that they have not properly solved; as with hallucinations, this reflects a deeply problematic failure to sanity check their own work.
- Where
- paragraph 15
- Published
- 2025-04-05
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-reports-of-llms-mastering-math-have-2bc2e7-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-reports-of-llms-mastering-math-have-2bc2e7-a1.