“People had been testing these Ais abilities at simple arithmetic for years before it occurred to anyone to check whether they can count. (They can’t, reliably).”
Our reading, the sentence that was checked:
LLMs cannot count reliably.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How Not to Test GPT-3, paragraph 49.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-not-to-test-gpt-3-1a2e80-a1
We recorded 12 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-02-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- GPT and similar AI systems
- whether they can count
- They can’t, reliably
Our interpretation
- Assess the counting assertion identified by the screens, separately from the historical assertion about when researchers first tested counting.
- The plain reading is that contemporary GPT-like models frequently make mistakes when directly counting items or characters. It does not mean they never produce correct counts.
- Include ordinary and step-by-step prompting. Treat execution of externally generated programs separately from the model answering directly.
- For this ongoing research-state observation, use the latest completed research year and the elapsed publication year: 2022 through 17 February 2023. This observation window is evaluator-selected.
Counts as supported if: Repeated evaluations show appreciable counting errors across contemporary GPT-like models or ordinary counting tasks, including errors that remain with reasonable prompting.
Counts as contradicted if: Representative evaluations show consistently dependable direct counting across varied ordinary tasks and contemporary models, with reported failures confined to unusual or misleading prompts.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporary evaluations document frequent direct counting errors across several models. Reasonable prompting improved performance substantially, but errors remained. The qualification “reliably” matters: Codex reached 93.2% on one counting evaluation, and program execution raised another result to 96.7%. The evidence supports unreliable direct counting, not an inability to count or to build a useful counting system.
Limitations
- Reliability has no speaker-specified threshold; a human-comparative interpretation would give more weight to Codex's strong result.
- Accuracy depends strongly on prompts and task construction.
- The research papers do not specify exact experiment dates; their results are dated at year precision.
- The chronology of the first counting tests is not adjudicated.
Evidence (3 items)
Supports the claim: Suzgun et al., arXiv,
Few-shot prompting performance
Why it matters here: Object-counting accuracy was 44.0%, 45.2%, and 51.2% with answer-only prompts for InstructGPT, Codex, and PaLM; step-by-step prompts raised these to 77.2%, 93.2%, and 83.2%. The Codex result exceeded the average human-rater score of 86.1%, materially limiting any claim of general counting incapacity.
- Publisher
- Suzgun et al., arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Gao et al., arXiv,
reaching 96.7%
Why it matters here: The study's Codex baseline scored 37.6% with direct answers and 73.0% with step-by-step answers. The 96.7% result used generated programs and a Python interpreter, showing an effective workaround while leaving direct counting errors substantial.
- Publisher
- Gao et al., arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: p.b., LessWrong,
unable to count
Why it matters here: The main post records several incorrect character counts, alongside a correct zero count. It corroborates failures in the contemporary chat interface but is an informal, selected demonstration rather than a representative accuracy estimate.
- Publisher
- p.b., LessWrong
- Kind of source
- Demonstration
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How Not to Test GPT-3
People had been testing these Ais abilities at simple arithmetic for years before it occurred to anyone to check whether they can count. (They can’t, reliably).
- Where
- paragraph 49
- Published
- 2023-02-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-not-to-test-gpt-3-1a2e80-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.