“And there are so many stories about how unreliable this technology is.”
Our reading, the sentence that was checked:
Generative AI is unreliable, as shown by many stories.
Assessment: Supported Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, Subprime Intelligence, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-sam-altman-fried-d011df-a1
We recorded 26 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-02-19. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- so many stories
- how unreliable this technology is
Our interpretation
- The context identifies generative AI and lists concrete incidents. The plain reading is that multiple public reports and evaluations documented incorrect or inappropriate outputs.
- This is not read as a claim that every output fails, or that incident reports establish a population-wide failure rate.
- The observation period is 2023 through the post date because the surrounding examples expressly include events from the previous year and recent incidents.
Counts as supported if: Multiple independently documented incidents or evaluations before the post establish incorrect, fabricated, or inappropriate generative-AI outputs in distinct settings.
Counts as contradicted if: The cited class of incidents is shown to be fabricated, misattributed, or unrelated to generative AI, and credible contemporaneous records fail to substantiate repeated examples.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Before publication, a court had documented fabricated ChatGPT citations, DPD had confirmed disabling its AI component after a reported inappropriate exchange, and an original financial-question evaluation had documented hallucinations and other failures. These establish repeated public examples in distinct settings. They support the claim about documented unreliability without establishing that all generative AI use is unsuccessful.
Limitations
- Selected incidents cannot establish a representative failure rate.
- The DPD incident involved deliberate prompting and an error following a system update.
- FinanceBench was a preprint, and its findings depend on the tested configurations.
Evidence (3 items)
Supports the claim: U.S. District Court for the Southern District of New York, reproduced by FindLaw,
ChatGPT, which fabricated the cited cases.
Why it matters here: The court found that ChatGPT fabricated cases used in the March 1 filing. This is a primary judicial finding of incorrect generated material, rather than an allegation repeated by the speaker.
- Publisher
- U.S. District Court for the Southern District of New York, reproduced by FindLaw
- Kind of source
- Regulatory filing
- Describes events of
- Public from
Supports the claim: TIME,
The AI element was immediately disabled
Why it matters here: TIME describes the chatbot producing profanity and criticism of DPD after user prompting, and quotes DPD confirming a post-update error and disabling the AI component. No separately accessible primary incident record was obtained.
- Publisher
- TIME
- Kind of source
- News report
- Describes events of
- Public from
Supports the claim: Pranab Islam and colleagues, arXiv,
such as hallucinations
Why it matters here: FinanceBench tested multiple contemporary model configurations on financial questions and manually reviewed their answers. It supplies systematic evidence of failures beyond isolated anecdotes; its combined incorrect-answer and refusal statistic should not be described solely as hallucinations.
- Publisher
- Pranab Islam and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, Subprime Intelligence
And there are so many stories about how unreliable this technology is.
- Where
- paragraph 19
- Published
- 2024-02-19
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-sam-altman-fried-d011df-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.