“Bullshit. Hallucinations are still here, and aren’t going away, anytime soon.”
Our reading, the sentence that was checked:
LLM hallucinations persist and will not go away anytime soon.
Assessment: Supported Deadline passed on 2026-10-29 Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AI literacy, hallucinations, and the law: A case study, paragraph 4.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-ai-literacy-hallucinations-and-the-a3f7cf-a1
We recorded 9 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-05-24. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Forecast
- Deadline, as the speaker wrote it
- “anytime soon”
- Deadline state when assessed
- Deadline passed
- Deadline state on 2026-10-29
- Deadline passed (the release’s reference date)
From the speaker
- Hallucinations are still here
- aren't going away
- anytime soon
Our interpretation
- The forecast is that LLM hallucinations will not be eliminated or become negligible in the near term.
- 'Anytime soon' has no stated date. Read plainly as at least about a year, so the window is set from the post date (2025-05-24) to 2026-05-24.
- 'Going away' means frontier LLMs no longer hallucinate at material rates on standard evaluations and in real use. Lower rates alone do not count as going away.
Counts as supported if: Evidence from within the window shows frontier LLMs still hallucinating at material rates, or researchers and labs describing hallucinations as persisting.
Counts as contradicted if: Evidence from within the window shows frontier LLMs with negligible hallucination rates across standard evaluations and real use, or a credible demonstration that the problem has been eliminated.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Did the forecast come true?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- Deadline passed
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Read plainly, 'aren't going away anytime soon' means hallucinations would still be a material problem for at least about a year after May 2025. They were. In September 2025, OpenAI researchers wrote that hallucinations 'persist even in state-of-the-art systems.' In May 2026, a replication study found that frontier models released through March 2026 still invented nonexistent software packages 4.6–6.1% of the time. Labs did report real progress: OpenAI says GPT-5 made 'significant advances in reducing hallucinations.' But progress is not the same as the problem going away. Court cases involving hallucinated citations also kept accumulating through 2026.
Limitations
- The one-year window is the evaluator's reading of 'anytime soon'.
- Reductions in some benchmarks might lead some readers to call the problem largely solved for certain tasks.
Evidence (4 items)
Supports the claim: arXiv (Kalai, Nachum, Vempala, Zhang; OpenAI),
Such "hallucinations" persist even in state-of-the-art systems and undermine trust.
Why it matters here: Researchers at a leading lab said in September 2025 that hallucinations persist even in state-of-the-art systems.
- Where
- Abstract
- Publisher
- arXiv (Kalai, Nachum, Vempala, Zhang; OpenAI)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv (Aleksandr Churilov),
we measure overall hallucination rates between 4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini) -- an order-of-magnitude compression of the inter-model spread observed by Spracklen, but not a retirement of the threat.
Why it matters here: A May 2026 measurement found that frontier models released up to March 2026 still invented nonexistent software packages about 5–6% of the time. Hallucinations had not gone away by the end of the window.
- Where
- Abstract
- Publisher
- arXiv (Aleksandr Churilov)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: OpenAI (arXiv),
We've made significant advances in reducing hallucinations
Why it matters here: OpenAI reports large reductions in hallucinations with GPT-5. This is progress, not elimination, but it shows the problem shrank during the window.
- Where
- Abstract
- Publisher
- OpenAI (arXiv)
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: Damien Charlotin,
While seeking to be exhaustive (2231 cases identified so far)
Why it matters here: Court decisions involving hallucinated content kept piling up through October 2026. This is context, because the count is dated after the one-year window.
- Where
- Introductory section
- Publisher
- Damien Charlotin
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AI literacy, hallucinations, and the law: A case study
Bullshit. Hallucinations are still here, and aren’t going away, anytime soon.
- Where
- paragraph 4
- Published
- 2025-05-24
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-ai-literacy-hallucinations-and-the-a3f7cf-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-ai-literacy-hallucinations-and-the-a3f7cf-a1.