“Without flexible underlying models that can reliably support inferences on novel questions, we can expect more of the same of what we are seeing over and over now: more hallucinations, more reasoning that is hit or miss, and nothing like the stability we should insist on for future AI.”
Our reading, the sentence that was checked:
Without flexible underlying models, AI systems will keep producing hallucinations and hit-or-miss reasoning, and will lack the needed stability.
Assessment: Unresolved No deadline stated on 2026-10-29 Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Muddles about Models, paragraph 23.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-muddles-about-models-390731-a2
We recorded 9 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-05. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Conditional forecast
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No deadline stated
- Deadline state on 2026-10-29
- No deadline stated (the release’s reference date)
Conditions
- without flexible underlying models that reliably support inference on novel questions (antecedent, stated by the speaker)
From the speaker
- Without flexible underlying models
- reliably support inferences on novel questions
- more hallucinations
- more reasoning that is hit or miss
- nothing like the stability we should insist on for future AI
Our interpretation
- The plain reading predicts continuing factual errors and inconsistent reasoning in subsequent systems that lack the specified flexible models.
- More means continued occurrence, not necessarily an increasing error rate.
- The antecedent must be established independently; errors alone cannot prove that a system lacks flexible underlying models.
- The stability requirement is qualitative and does not supply a numerical threshold or deadline.
Counts as supported if: For systems independently shown to satisfy the antecedent, repeated evaluations find continuing hallucinations and inconsistent reasoning on novel questions.
Counts as contradicted if: A system independently shown to satisfy the antecedent nevertheless demonstrates stable factual and reasoning performance across representative novel questions under sustained testing.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Did the forecast come true?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No deadline stated
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The GSM-Symbolic study released on October 7, 2024 reports variation in mathematical performance when names and values change while required reasoning steps remain the same [e1]. OpenAI’s October 30, 2024 SimpleQA disclosure documents continuing factual errors in language models [e2]. These findings are consistent with the predicted behavior, but neither independently establishes that the tested systems lack the specified flexible underlying models [e1, e2]. They therefore leave the conditional forecast unresolved [e1, e2].
Limitations
- No independent evidence establishing the forecast’s antecedent for the evaluated systems.
- Task-specific evaluations, without sustained testing across representative novel questions.
- No defined stability threshold or forecast deadline; finite observations of errors cannot close the unlimited horizon.
Evidence (2 items)
Context only: arXiv; Iman Mirzadeh and colleagues,
non-negligible variance
Why it matters here: The study finds inconsistent performance across variants of mathematical questions. Its proposed explanation involving pattern matching is a hypothesis, not independent proof that the tested systems lack the models specified in the antecedent.
- Publisher
- arXiv; Iman Mirzadeh and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: OpenAI,
false outputs
Why it matters here: The disclosure and benchmark document continuing factuality problems after Marcus’s post. They do not establish whether the tested systems satisfy his condition about underlying models.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Independent technical evidence establishing whether named systems lack the specified flexible models, paired with reproducible factuality and novel-reasoning evaluations. A system established to lack those models that demonstrates sustained stable performance across representative novel questions could conclusively contradict the forecast under its early-resolution rule.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Muddles about Models
Without flexible underlying models that can reliably support inferences on novel questions, we can expect more of the same of what we are seeing over and over now: more hallucinations, more reasoning that is hit or miss, and nothing like the stability we should insist on for future AI.
- Where
- paragraph 23
- Published
- 2023-10-05
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-muddles-about-models-390731-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.