“They don’t induce abstract models of the world that they can reliably engage, even when those models are given to them on a silver platter (as with the rules of chess).”
Our reading, the sentence that was checked:
LLMs do not induce abstract world models they can reliably use, even when given the rules explicitly.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, LLMs are not like you and me—and never will be., paragraph 7.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-llms-are-not-like-you-and-meand-never-b5e141-a1
We recorded 11 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-08-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- even when models are given to them (qualifier, stated by the speaker)
From the speaker
- don’t induce abstract models of the world
- that they can reliably engage
- even when those models are given to them
- the rules of chess
Our interpretation
- The ordinary reading concerns dependable application of learned states and rules, including when relevant rules are explicitly supplied.
- Reliability requires repeated successful application on held-out cases, rather than an isolated correct answer. The source supplies no numerical cutoff.
- Treat the supplied-rules phrase as an observational test setting, not a conditional forecast.
- Assess the accumulated capability record through 2025-08-12, with no lower bound supplied by the post.
Counts as supported if: Controlled tests show substantial failures to apply explicitly supplied rules across relevant LLMs, with evidence that these failures reflect inadequate usable representations rather than only arithmetic or interface mistakes.
Counts as contradicted if: Controlled tests demonstrate that an in-scope LLM learns and repeatedly uses an abstract state-and-rule model on held-out cases, including the supplied-rules setting, without an external system enforcing correct actions.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
RuleArena supports the concern that LLMs can fail even with rules supplied. Specialist chess results show that language models can also learn substantial rule-following competence. These findings concern different training and testing settings, and neither establishes the full claim about abstract models that can be reliably used. A mixed verdict would imply a closer match between the competing tests than the evidence provides.
Limitations
- RuleArena tests practical regulations rather than chess.
- The positive chess result concerns a small, specially fine-tuned model.
- Reliability and the required breadth of a world model remain undefined.
Evidence (2 items)
Supports the claim: arXiv; Ruiwen Zhou and colleagues,
prone to irrelevant distractions.
Why it matters here: RuleArena supplies domain rules in the prompt and finds failures in combining and applying them. This supports the unreliability component in its tested settings, but does not establish that the models lack abstract representations; the date identifies the documented experimental record in version 2.
- Publisher
- arXiv; Ruiwen Zhou and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Ben Fauber,
nearly always proposed a legal move.
Why it matters here: A specialized small language model demonstrates substantial legal-move competence, limiting any blanket conclusion that language models cannot learn rules. Fine-tuning on examples differs from supplying rules in a prompt, and legal moves alone do not prove an abstract internal world model.
- Publisher
- arXiv; Ben Fauber
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A matched evaluation of pre-post LLMs given explicit rules, measuring held-out legality, state tracking, and transfer, together with evidence connecting their behavior to learned representations.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, LLMs are not like you and me—and never will be.
They don’t induce abstract models of the world that they can reliably engage, even when those models are given to them on a silver platter (as with the rules of chess).
- Where
- paragraph 7
- Published
- 2025-08-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-llms-are-not-like-you-and-meand-never-b5e141-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-llms-are-not-like-you-and-meand-never-b5e141-a1.