“You can add as many levels of intrigue and “reasoning” as you want, but Large Language Models cannot be trusted to do something correctly, or even consistently, every time.”
Our reading, the sentence that was checked:
Regardless of added reasoning, LLMs cannot be trusted to do things correctly or consistently every time.
Assessment: Supported Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, The Case Against Generative AI, paragraph 98.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-the-case-against-generative-ai-a9057b-a1
We recorded 39 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-09-29. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- regardless of added reasoning (qualifier, stated by the speaker)
From the speaker
- as many levels of intrigue and “reasoning” as you want
- cannot be trusted
- correctly, or even consistently, every time
Our interpretation
- The plain reading is that existing LLMs, including reasoning-enhanced systems, do not guarantee correct, repeatable performance across user tasks.
- This does not mean every task fails, every repeated answer differs, or no bounded application can be reliable.
- For this ongoing technical observation, use the latest quarter-to-date before publication: July 1 through September 29, 2025. Model evaluations have no uniform financial-style reporting cycle.
- The observation concerns available reasoning methods, not a proof about every possible future architecture.
Counts as supported if: Evaluations in the target period show nonzero task failures in reasoning-capable systems and variation across repeated runs despite reasoning prompts or other enhancements.
Counts as contradicted if: Evidence establishes general correct and repeatable performance across the relevant tasks with added reasoning; a perfect score on one finite, narrow benchmark is insufficient.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The ordinary current-system reading holds. A leading reasoning-capable model still failed benchmark tasks, and independent repeated-run research found that reasoning prompts did not provide uniform consistency. Narrow successes, including a reported zero-error editing benchmark, do not establish correct performance across tasks every time. The evidence does not support extending this verdict to all future systems.
Limitations
- One source is a vendor evaluation and the independent study is a preprint.
- The independent study tests reasoning prompts rather than every dedicated reasoning-model architecture.
- The model announcement and the post share a publication date; their intraday order is not established.
Evidence (2 items)
Supports the claim: Anthropic,
61.4%
Why it matters here: The company's reported success rate for its new reasoning-capable model is below perfect performance. The same page includes a customer's zero-error result on a narrow internal editing benchmark, which does not establish general reliability.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: Shyr et al., medRxiv,
repeatability varied
Why it matters here: The August evaluation reports differing repeatability across models and reasoning prompts. It supports the absence of a general consistency guarantee; it does not say every output is unstable.
- Publisher
- Shyr et al., medRxiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, The Case Against Generative AI
You can add as many levels of intrigue and “reasoning” as you want, but Large Language Models cannot be trusted to do something correctly, or even consistently, every time.
- Where
- paragraph 98
- Published
- 2025-09-29
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-the-case-against-generative-ai-a9057b-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment wyea-the-case-against-generative-ai-a9057b-a1.