“(Other models like o3 might do better on this specific task but are still vulnerable to the same overall issue.)”
Our reading, the sentence that was checked:
Other models like o3 might do better on this specific tic-tac-toe variant but are still vulnerable to the same overall issue of lacking world models.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Generative AI’s crippling and widespread failure to induce robust models of the world, paragraph 72.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-generative-ais-crippling-and-widespread-eb4d81-a3
We recorded 21 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-28. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- might do better on this specific task (qualifier, stated by the speaker)
From the speaker
- Other models like o3
- might do better on this specific task
- still vulnerable to the same overall issue
Our interpretation
- The ordinary behavioral reading is that o3 can fail to maintain an accurate game state or apply its rules consistently. Losing a game alone would not establish this.
- The comparison with Grok is a concession, not a conditional forecast.
- The observation concerns the contemporary o3 release through the post date. I use its release-to-post interval because no observation window is specified.
- Behavioral failures can establish vulnerability without establishing the complete absence of internal world representations.
Counts as supported if: Documented tests of contemporary o3 show substantive illegal moves or incorrect representations of an established game state.
Counts as contradicted if: The alleged state and rule failures are shown to be evaluation artifacts, and sufficiently broad comparable testing demonstrates reliable state tracking under the challenged conditions.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Acher's report available May 25, 2025 documents substantive illegal moves by o3, separately identifying two notation-related exceptions [e1]. It also records o3 incorrectly reconstructing a chess position and rejecting a legal opponent move [e1]. This supports the claim that o3 remained vulnerable to failures in maintaining game state and applying rules [e1]. The same report includes competent legal play, so the finding concerns vulnerability rather than failure in every game [e2]. Support is limited to this behavioral reading of the claimed world-model problem [e1, e2].
Limitations
- The chess tests do not establish that o3 lacks internal world models or identify the cause of its errors [e1, e2].
- They do not test the y/z tic-tac-toe variant or settle the hedged comparison with Grok [e1, e2].
- Individual test dates are unspecified, and inconsistent sample descriptions prevent a reliable aggregate failure-rate estimate [e1, e2].
Evidence (2 items)
Supports the claim: Mathieu Acher,
clear illegal moves.
Why it matters here: Acher describes substantive o3 rule violations and an incorrect reconstruction of a game state. He separately identifies two notation-related cases, so the substantive finding does not depend on counting those as failures.
- Publisher
- Mathieu Acher
- Kind of source
- Independent measurement
- Describes events of
- Public from
Context only: Mathieu Acher,
quite good.
Why it matters here: The same report includes competent legal play and a winning o3-high game. This limits any interpretation that o3 invariably fails or has no chess competence, but does not contradict vulnerability.
- Publisher
- Mathieu Acher
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Generative AI’s crippling and widespread failure to induce robust models of the world
(Other models like o3 might do better on this specific task but are still vulnerable to the same overall issue.)
- Where
- paragraph 72
- Published
- 2025-06-28
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-generative-ais-crippling-and-widespread-eb4d81-a3: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-generative-ais-crippling-and-widespread-eb4d81-a3.
HeldBy this site
The final-text check failed: proposition_faithful.
About assessment marcus-generative-ais-crippling-and-widespread-eb4d81-a2, which was not published.