“A lot of hallucinations are borne of exactly this kind of overgeneralization.”
Our reading, the sentence that was checked:
Many LLM hallucinations result from overgeneralizing word correlations such as semantic leakage.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, “New Ways to Corrupt LLMs”, paragraph 3.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-new-ways-to-corrupt-llms-4af24a-a2
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-12-14. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- A lot of hallucinations
- borne of exactly this kind of overgeneralization
Our interpretation
- The plain reading is that transferring irrelevant learned associations into answers causes a substantial, recurring portion of LLM hallucinations.
- The claim requires more than demonstrations that semantic leakage exists. It supplies no numerical threshold or population of hallucinations.
Counts as supported if: Controlled experiments establish the proposed causal mechanism, and representative evaluations show that it accounts for a substantial portion of hallucinations.
Counts as contradicted if: Representative causal evaluations show that this mechanism is absent or accounts for only rare hallucinations.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The 2024 study measures whether outputs become more semantically related to an irrelevant detail added to a prompt, compared with control outputs [e1]. Its Leak-Rate does not measure the share of factual hallucinations caused by that effect [e1]. This evidence therefore does not establish whether the mechanism causes many hallucinations or only a few [e1].
Limitations
- The cited measurement does not provide representative causal attribution across factual hallucinations, tasks, and models [e1].
- Semantic similarity in generated completions does not by itself establish factual error [e1].
Evidence (1 item)
Context only: arXiv; Hila Gonen and colleagues,
Leak-Rate higher than 50%
Why it matters here: The experiment compares semantic similarity of outputs with and without an irrelevant prompt detail. Its leakage rate measures an association effect, not the share of factual hallucinations caused by that effect.
- Publisher
- arXiv; Hila Gonen and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Controlled studies across representative models and factual tasks that manipulate irrelevant associations and estimate the share of hallucinations they cause, with an explicit threshold for a substantial share.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, “New Ways to Corrupt LLMs”
A lot of hallucinations are borne of exactly this kind of overgeneralization.
- Where
- paragraph 3
- Published
- 2025-12-14
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-new-ways-to-corrupt-llms-4af24a-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.