“The paper turns out to be pretty great, doing exactly the right test, but it doesn’t prove what its title alleges.”
Our reading, the sentence that was checked:
The MathGLM paper does not prove its titular claim that GPT can solve mathematical problems without a calculator.
Assessment: Can’t be checked Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, “Math is hard” — if you are an LLM – and why that matters, paragraph 2.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-math-is-hard-if-you-are-an-llm-and-4f12b3-a2
We recorded 5 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-19. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- doesn’t prove what its title alleges
- GPT Can Solve Mathematical Problems Without a Calculator
- no partial credit for coming close
Our interpretation
- Marcus accepts that the model sometimes succeeds. His objection concerns calculator-like reliability, not the existence of successful calculations.
- The title's capability claim does not specify universal correctness. Equating it with calculator-like reliability adds a disputed standard.
- The reported accuracy is checkable, but whether it proves the title has no sufficiently definite binary criterion. I therefore do not substitute an accuracy claim for this assessment of the title.
This claim is not graded on evidence. The record checks that the words are quoted correctly and attributed to the right person, and that the claim is classified correctly.
Current assessment
Not graded on evidence
Outcome: Can’t be checked
The statement is a recommendation, a value judgment, or too open to test. It is recorded, not scored.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The September 2023 paper reports both successful calculations and errors, including 93.03% overall accuracy for the 2-billion-parameter MathGLM model [e1]. These results cannot settle whether the paper “proves” its title without a specified standard distinguishing demonstrated capability from calculator-like reliability [e1].
Limitations
- The reported accuracy does not establish guaranteed correctness, and the reported errors do not establish an absence of calculator-free capability.
- There is no agreed reliability threshold for deciding whether the experiments prove the title.
Evidence (1 item)
No outcome evidence: a claim of this kind is recorded and attributed, not scored.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, “Math is hard” — if you are an LLM – and why that matters
The paper turns out to be pretty great, doing exactly the right test, but it doesn’t prove what its title alleges.
- Where
- paragraph 2
- Published
- 2023-10-19
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-math-is-hard-if-you-are-an-llm-and-4f12b3-a2: Current. Outcome: Can’t be checked. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-math-is-hard-if-you-are-an-llm-and-4f12b3-a2.