“That’s just obviously not so, e.g., “write a biography of X individual without making stuff up” remains pretty unreliable; same for frontier models if you ask them to “play grandmaster level chess without making up illegal moves”.”
Our reading, the sentence that was checked:
It is obviously not true that frontier language models can perform competently at pretty much any information task that humans can do, that can be posed and answered in natural language, and that has quantifiable performance; for example, writing a biography of an individual without making things up remains pretty unreliable, as does playing grandmaster-level chess without making illegal moves.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, The Five Stages of AGI Grief, paragraph 13.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-the-five-stages-of-agi-grief-dfad7e-a1
We recorded 7 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-01-07. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- information tasks doable by humans, posed and answered in natural language, with quantifiable performance (scope, stated by the speaker)
From the speaker
- frontier models
- write a biography of X individual without making stuff up
- remains pretty unreliable
- play grandmaster level chess without making up illegal moves
Our interpretation
- The ordinary reading concerns repeated performance by contemporary general-purpose language models, rather than a specially trained chess engine or an isolated successful response.
- Both examples matter: factual biography generation and sustained grandmaster-level chess with legal moves.
- For the ongoing state, I use December 2024, the latest complete month before publication. Marcus supplied no reporting period.
- A nontrivial recurring failure rate would establish unreliability; a single error would not establish its prevalence. No numerical reliability threshold is imposed.
Counts as supported if: Representative evaluations of models current in the target period show recurring invented biographical facts and failure to sustain grandmaster-level legal chess.
Counts as contradicted if: Representative evaluations establish reliable factual biographies and sustained grandmaster-level chess without recurring illegal moves for contemporary frontier models.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The evidence is consistent with Marcus's concerns, but does not settle this combined claim for the contemporary frontier. OpenAI documented factual errors about people in December 2024, alongside improvement over GPT-4o. The direct chess experiment documents substantial failures in 2023. Treating either as a complete test of December 2024 biographies and grandmaster-level chess would overstate its scope.
Limitations
- The unresolved result reflects missing matched evidence, not evidence that the models were reliable.
- The selected reporting window and unspecified meaning of frontier materially affect the assessment.
Evidence (2 items)
Context only: OpenAI,
o1‑preview and o1 hallucinate less frequently
Why it matters here: Table 3 reports PersonQA hallucination rates of 0.30 for GPT-4o and 0.20 for o1. This establishes remaining errors and improvement on questions about people, but does not measure the reliability of complete biographies.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: Mathieu Acher,
32 % of illegal games.
Why it matters here: The experiment directly documents recurring illegal moves in GPT-4 games. Its earlier model generation and observation period prevent treating it as decisive evidence about the December 2024 frontier.
- Publisher
- Mathieu Acher
- Kind of source
- Independent measurement
- Describes events of
- Public from
What would settle it
Dated evaluations of the models current in December 2024, covering representative biography prompts and complete chess games with transparent prompting, tool access, factual verification, and strength calibration.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, The Five Stages of AGI Grief
That’s just obviously not so, e.g., “write a biography of X individual without making stuff up” remains pretty unreliable; same for frontier models if you ask them to “play grandmaster level chess without making up illegal moves”.
- Where
- paragraph 13
- Published
- 2025-01-07
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-the-five-stages-of-agi-grief-dfad7e-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-the-five-stages-of-agi-grief-dfad7e-a1.