“The hallucination problem is one that is nowhere closer to being solved — and, at least with the current technology — may never go away, and it makes it a non-starter for a great many business tasks, where you need a high level of reliability.”
Our reading, the sentence that was checked:
The hallucination problem is nowhere closer to being solved, may never go away at least with current technology, and makes generative AI a non-starter for a great many business tasks that need high reliability.
Assessment: Mixed Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, Godot Isn't Making it, paragraph 6.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-godot-isnt-making-it-df862b-a1
We recorded 33 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-12-03. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- at least with the current technology (qualifier, stated by the speaker)
- may never go away (qualifier, stated by the speaker)
- a great many business tasks where you need a high level of reliability (scope, stated by the speaker)
From the speaker
- nowhere closer to being solved
- at least with the current technology
- may never go away
- a non-starter for a great many business tasks
- a high level of reliability
Our interpretation
- The firm, testable component is the assertion that there had been no meaningful progress in reducing hallucinations.
- The surrounding retrospective discusses warnings beginning in March 2024. I use March through the post date as the comparison period; the sentence itself supplies no baseline date.
- For the business-use component, I read high reliability as requiring factual output that can be relied upon without independent verification. This added interpretation materially affects the assessment.
- May never go away expresses a possibility, not a forecast that hallucinations will certainly persist forever.
Counts as supported if: Comparable evaluations show no meaningful reduction in hallucinations, and realistic evaluations show consequential factual errors that prevent reliance on outputs for substantial high-reliability business work.
Counts as contradicted if: Comparable evaluations show meaningful reductions in hallucinations, or realistic business evaluations establish the required reliability for the tasks said to be excluded.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Mixed
Part of the claim holds and part does not, and the record says which part is which.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The no-progress component is contradicted by contemporary comparisons showing lower hallucination rates. The reliability concern is supported for unverified legal research: a specialized product still hallucinated in roughly one-third of tested responses. That supports a substantial practical barrier, not a count of all unsuitable business tasks. The suggestion that hallucinations may never disappear remains an unscored possibility.
Limitations
- The supporting finding concerns legal research and cannot establish the full breadth of great many.
- Reading non-starter as excluding unverified reliance is an evaluator interpretation.
- The improvement evidence is vendor-reported and includes contrary anecdotal feedback.
- The system-card availability date uses the September 17 version identified in its filename.
Evidence (2 items)
Supports the claim: arXiv,
one-third of its responses contained a hallucination.
Why it matters here: Testing on May 23–27 found frequent hallucinations in a specialized legal research product. This supports the barrier to unverified, high-reliability legal work under the stated interpretation, but does not establish the number of business tasks excluded.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Goes against the claim: OpenAI,
o1-preview hallucinates less frequently than GPT-4o
Why it matters here: Table 2 reports SimpleQA hallucination rates of 0.61 for GPT-4o and 0.44 for o1-preview, with improvements on the other listed tests. The same paragraph reports contrary anecdotal feedback and cautions against generalizing beyond these evaluations.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Representative business-task evaluations with explicit error tolerances and verification costs would resolve the remaining breadth of the reliability claim.
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, Godot Isn't Making it
The hallucination problem is one that is nowhere closer to being solved — and, at least with the current technology — may never go away, and it makes it a non-starter for a great many business tasks, where you need a high level of reliability.
- Where
- paragraph 6
- Published
- 2024-12-03
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-godot-isnt-making-it-df862b-a1: Current. Outcome: Mixed. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.