“Still, even though Chomsky’s argument surely could have used considerably more nuance (see my article with Ernest Davis on how not to evaluate GPT), his overall point is correct: LLMs don’t reliably understand the world.”
Our reading, the sentence that was checked:
LLMs do not reliably understand the world.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Caricaturing Noam Chomsky, paragraph 20.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-caricaturing-noam-chomsky-a8598b-a1
We recorded 5 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-03-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- LLMs
- don't reliably understand the world
- present tense, March 2023
- ChatGPT is wildly stochastic and unreliable
Our interpretation
- Plain reading: the LLMs of early 2023 (ChatGPT and its peers) did not consistently get world facts and reasoning right. They made frequent factual and reasoning errors, so no one could count on them to be correct about the world.
- This reading tests the word 'reliably' through behaviour. It does not try to settle the philosophical question of whether LLMs 'understand' anything at all.
- Target period: January to March 2023, the state of LLMs when the post appeared. The claim describes an ongoing state, and this is the most recent period the evidence covers.
Counts as supported if: Credible evaluations or developer disclosures from early 2023 show that leading LLMs still hallucinated facts or made reasoning errors often enough that they could not be relied on.
Counts as contradicted if: Credible evidence from early 2023 shows that leading LLMs were consistently accurate on factual, commonsense and causal reasoning about the world, at a level that could be relied on.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The plain reading of the claim is behavioural. In early 2023, LLMs did not get the world right consistently enough to be relied on. OpenAI's GPT-4 technical report, published days after the post, says GPT-4, like earlier GPT models, 'still is not fully reliable', because it hallucinates facts and makes reasoning errors. That backs the claim that LLMs of that period were unreliable about the world. Whether they 'understand' in a deeper sense is a philosophical question no evidence can settle. This verdict rests only on reliability.
Limitations
- It rests on a behavioural reading of 'understand'. Someone who reads 'understanding' differently could dispute it.
- It uses a single developer disclosure, not a broad benchmark survey.
Evidence (1 item)
Supports the claim: OpenAI (arXiv),
Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors).
Why it matters here: The developer of the leading LLM of early 2023 says the model, like earlier GPT models, is not fully reliable: it hallucinates facts and makes reasoning errors. This matches the behavioural reading of 'don't reliably understand the world'.
- Publisher
- OpenAI (arXiv)
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Caricaturing Noam Chomsky
Still, even though Chomsky’s argument surely could have used considerably more nuance (see my article with Ernest Davis on how not to evaluate GPT), his overall point is correct: LLMs don’t reliably understand the world.
- Where
- paragraph 20
- Published
- 2023-03-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-caricaturing-noam-chomsky-a8598b-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-caricaturing-noam-chomsky-a8598b-a1.