Generative AI’s crippling and widespread failure to induce robust models of the world
We recorded 18 claims from this post. 11 were checked against evidence. 7 can’t be checked. 18 more are held. Read the post at its source.
Claims from this post (18)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Causal claim
“LLMs try — to their peril — to live without classical world models.”
Our reading: LLMs try to function without classical world models, and this harms them.
No horizon Evidence as of
Limitation: The cited experiment concerns planning rather than the full range of LLM failures.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The central importance of such cognitive (a.k.a. world) models to AI was something that I highlighted again in my 2020 article The Next Decade in AI, which laid out a roadmap that I still believe the field ought to be following.”
Our reading: The AI field should follow the roadmap centred on cognitive world models that Marcus set out in 2020.
No horizon
Assessment: UnresolvedSaid
Causal claim
“The fact that they can get by as far as they can without explicit world models is actually astonishing. But much of what ails them comes from that design choice.”
Our reading: Much of what goes wrong with LLMs comes from the design choice to do without explicit world models.
No horizon Evidence as of
Limitation: Much and the relevant population of failures remain unspecified.
Assessment: SupportedSaid
Observation
“A whole field known as “mechanistic interpretability” has tried to derive (or infer) world models from LLMs, but with very limited success.”
Our reading: Mechanistic interpretability has had very limited success at finding world models inside LLMs.
No horizon Evidence as of
Limitation: The qualitative judgment depends on distinguishing partial representations from robust dynamic models.
Assessment: UnresolvedSaid
Causal claim
“And that is why they hallucinate, frequently. They wouldn’t need to do so if they actually had reliable access to relevant databases.”
Our reading: LLMs hallucinate often because they rebuild facts stochastically from fragments of text instead of using structured databases.
No horizon Evidence as of
Limitation: The supporting experiment concerns structured workflow generation.
Assessment: UnresolvedSaid
Causal claim
“In short ChatGPT can approximate the game of chess, but it can’t play it reliably, precisely because (despite immense relevant evidence) it never induces proper world model of the board and the rules.”
Our reading: ChatGPT cannot play chess reliably because it never induces a proper world model of the board and the rules.
No horizon Evidence as of
Limitation: The available intervention studies concern earlier ChatGPT experiments or different language models.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Inducing the rules of chess should be table stakes for AGI.”
Our reading: Any system that counts as AGI should, at a minimum, be able to induce the rules of chess.
No horizon
Assessment: UnresolvedSaid
Causal claim
“A huge number of peculiar LLM errors can be understood in this way.”
Our reading: The lack of world models explains a huge number of peculiar LLM errors.
No horizon Evidence as of
Limitation: The claim supplies no threshold for huge number.
Assessment: SupportedSaid
Observation
“Video-generators (even the latest ones) too are insufficiently anchored in the world.”
Our reading: Even the latest video generators, such as Veo 3, are not anchored firmly enough in the world.
No horizon Evidence as of
Limitation: The result depends on interpreting anchored as observable physical and entity consistency.
Assessment: SupportedSaid
Observation
“Video comprehension (as opposed to video generation) shows similar problems, especially in unusual situations that can’t simply be handled by retrieval of stored cases.”
Our reading: Video comprehension models show world-model failures, especially in unusual situations that stored cases cannot cover.
No horizon Evidence as of
Limitation: Support applies to the behavioral reading and the models actually tested.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The idea that the military would use such obtuse tools in the fog of war is halfway between alarming and preposterous.”
Our reading: The idea that the military would use such obtuse video-understanding tools in the fog of war is halfway between alarming and preposterous.
No horizon
Scope: in the fog of war
Assessment: UnresolvedSaid
Observation
“Game play continued, and I was able to beat the “smartest AI on earth” in four moves.”
Our reading: Grok 3 lost to Marcus in four moves at a tic-tac-toe variant that used y's and z's.
No horizon Evidence as of
Limitation: A public record may exist outside the indexed sources searched.
Assessment: Can’t be checkedSaid
Causal claim
“The same lack of capacity for representing and maintaining world models over time that explains the other examples in this essay is likely involved in poker as well.”
Our reading: The same lack of capacity for representing and maintaining world models over time that explains the essay's other examples is likely involved in LLMs' poor poker play as well.
No horizon
Scope: in poker as well
Assessment: Can’t be checkedSaid
Rhetoric
“In the tricky world of human relationships, parroting words can only get you so far.”
Our reading: Parroting words has limited value for understanding human relationships.
No horizon
Assessment: SupportedSaid
Observation
“But can Claude stick to that rule? Nope. As Adam Gleave recently reported, with a simply jailbreak Claude can easily be coaxed to violate its instructions, disgorging page after page on how to manufacture sarin gas. So much for alignment through system prompts.”
Our reading: As Adam Gleave reported, a simple jailbreak can easily coax Claude into violating its system-prompt instruction against providing weapons information, including producing pages on how to make sarin gas, which Marcus takes as undercutting alignment through system prompts.
No horizon Evidence as of
Limitation: The incident is supported by the originating researcher's public account, not an independent replication.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“In some ways LLMs far exceed humans, but in other ways, they are still no match for an ant. Without robust cognitive models of the world, they should never be fully trusted.”
Our reading: As long as LLMs lack robust cognitive models of the world, they should never be fully trusted.
No horizon
Qualifier: without robust cognitive models of the world
Assessment: Can’t be checkedSaid
Forecast
“Their work is interesting, and converges to a degree with the current argument, but my intuition is that such functions will not suffice for AGI.”
Our reading: Marcus's intuition is that world models defined as environment transition functions will not be enough for AGI.
No horizon
Assessment: UnresolvedSaid
Conditional forecast
“As the number of propositions and the complexity of what is represented increases (e.g., to include logical quantifiers), I expect encoding too will turn out to be unreliable.”
Our reading: As propositions become more numerous and complex, LLMs' encoding of them will also prove unreliable.
No deadline stated Evidence as of
Only if: as number and complexity of propositions increases
Held from this post (18)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: In a test game, o3 attempted an illegal chess move.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: Despite massive training data, LLMs fail to induce proper, reliable world models of chess.
- The criteria are ambiguous. Our reading, not assessed: Leaving our lives in the hands of LLMs that lack explicit models is literally putting our lives at risk.
- The criteria are ambiguous. Our reading, not assessed: LLMs never induce proper world models of anything; everything they do is mimicry.
- The criteria are ambiguous. Our reading, not assessed: LLMs often get lost by the chess midgame, playing worse and making illegal moves.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: ChatGPT recently lost a chess game to an Atari 2600.
- The record does not meet the current rules. Our reading, not assessed: In Anthropic's Project Vend, Claude had no world model: it did not understand its own non-corporeal nature or profits.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: LLMs cannot build and maintain adequate, interpretable, dynamically updated world models, a shortcoming arguably more fundamental than their reasoning failures.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: Proper world models do not emerge on their own in LLMs.
- The two reviews reached different assessments. Our reading, not assessed: In a chatbot, hallucinations are always a risk.
- The criteria are ambiguous. Our reading, not assessed: ChatGPT cannot reliably use factual information like Wikipedia's and has no explicit, stable internal world model.
- The two reviews read the claim differently