“As the number of propositions and the complexity of what is represented increases (e.g., to include logical quantifiers), I expect encoding too will turn out to be unreliable.”
Our reading, the sentence that was checked:
As propositions become more numerous and complex, LLMs' encoding of them will also prove unreliable.
Assessment: Unresolved No deadline stated on 2026-10-29 Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Generative AI’s crippling and widespread failure to induce robust models of the world, paragraph 99.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-generative-ais-crippling-and-widespread-137104-a2
We recorded 18 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-28. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Conditional forecast
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No deadline stated
- Deadline state on 2026-10-29
- No deadline stated (the release’s reference date)
Conditions
- as number and complexity of propositions increases (antecedent, stated by the speaker)
From the speaker
- As the number of propositions and the complexity of what is represented increases
- e.g., to include logical quantifiers
- I expect encoding too will turn out to be unreliable
Our interpretation
- Encoding means the internal representation of the supplied propositions, distinguished from producing an answer or decoding that representation.
- The antecedent is exposure to larger and more complex propositional inputs. Quantifiers are an example, not a required feature of every test.
- Increasing the number of interacting Boolean clauses is an ordinary instance of increasing proposition count and represented logical complexity.
- The forecast concerns systematic encoding failures, not an isolated mistake or a requirement that every model fail on every larger input.
- No deadline is stated.
Counts as supported if: Controlled experiments with larger and more complex propositional inputs demonstrate systematic loss or corruption of their internal representation, with controls separating encoding failure from probe, reasoning, and decoding failure.
Counts as contradicted if: Well-controlled experiments across substantial increases in proposition count and logical complexity establish robust internal encoding, including on cases where generated answers fail.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Did the forecast come true?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No deadline stated
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
SATBench's September 22, 2025 version varies clause counts and reports answer errors on logical puzzles, meeting the condition of larger and harder propositional inputs [e1]. These results do not isolate internal encoding failure [e1]. The December 6, 2024 probing study reports that its similarity measure misses distinctions in a three-entity example that interventions still detect; the authors suggest their methods may be too noisy for contexts with more than two entities [e2]. Neither item establishes systematic encoding failure or robust encoding across substantial increases in logical complexity [e1, e2].
Limitations
- Answer accuracy does not distinguish encoding errors from reasoning or decoding errors [e1].
- The probing study does not establish encoding reliability at the larger logical complexity contemplated by the claim [e2].
Evidence (2 items)
Supports the claim: arXiv,
varying the number of clauses
Why it matters here: SATBench tests models on logical puzzles with varying clause counts and difficulty, satisfying the antecedent under the stated reading. Its answer-level errors do not identify whether internal encoding, reasoning, or decoding failed.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv,
our current methods may be too noisy
Why it matters here: The authors distinguish shortcomings in their probe from shortcomings in the model's representation: interventions still capture binding differences that the similarity measure misses. Consequently, poorer probing performance on larger inputs cannot alone settle Marcus's prediction.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Reproducible experiments that increase proposition count and logical complexity, measure internal representations with validated methods, and establish systematic encoding failure or robust encoding while separating probe, reasoning, and decoding errors.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Generative AI’s crippling and widespread failure to induce robust models of the world
As the number of propositions and the complexity of what is represented increases (e.g., to include logical quantifiers), I expect encoding too will turn out to be unreliable.
- Where
- paragraph 99
- Published
- 2025-06-28
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-generative-ais-crippling-and-widespread-137104-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-generative-ais-crippling-and-widespread-137104-a2.