“LLMs are sold as some sort of magic tool that can fix “anything” without ever specifying what that thing might be, mostly because they cannot be trusted, even in things that they mostly get right, to do things right every time.”
Our reading, the sentence that was checked:
LLMs cannot be trusted to do things right every time, even on tasks they mostly get right.
Assessment: Supported Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, The AI Hater's Manifesto, paragraph 13.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-the-ai-haters-manifesto-ab77a8-a2
We recorded 50 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-08-25. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- cannot be trusted
- even in things that they mostly get right
- to do things right every time
Our interpretation
- Assess the normalized reliability claim, not the surrounding claims about marketing or its motives.
- The plain reading is that usually successful LLM workflows still produce occasional errors. It does not mean that every conceivable narrow task must fail.
- For this ongoing observation, use the latest complete calendar quarter before publication: April through June 2026. Zitron supplied no reporting period.
Counts as supported if: Repeated evaluations show that LLMs sometimes fail tasks they can ordinarily complete successfully.
Counts as contradicted if: Representative repeated evaluations establish dependable, error-free performance across the workflows covered by the claim. An isolated perfect benchmark score is insufficient.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The paper version released June 2, 2026 reports five runs per task across 15 models and two benchmarks [e1](https://arxiv.org/html/2602.16666v3). It finds that agents able to solve a task often fail to solve it consistently, supporting the claim that successful performance does not ensure success on every attempt [e1](https://arxiv.org/html/2602.16666v3). Microsoft Research's May 15, 2026 discussion limits broader conclusions: its separate artifact-integrity stress test did not measure task completion or user satisfaction and was not representative of optimized production systems [e2](https://www.microsoft.com/en-us/research/blog/further-notes-on-our-recent-research-on-ai-delegation-and-long-horizon-reliability).
Limitations
- The June paper version was released within April–June 2026, but exact experiment dates are not separately reported.
- The evaluations do not cover every model or workflow or establish that every narrow task must fail.
- The findings do not establish a lack of practical usefulness or show that verification, oversight and specialized workflows cannot improve reliability.
Evidence (2 items)
Supports the claim: Rabanser and colleagues, Princeton University; ICML 2026 paper hosted on arXiv,
consistently.
Why it matters here: The June evaluation reports five runs per task across 15 models and two benchmarks. Its finding that solvable tasks still fail on repeat directly supports the distinction between capability and dependable completion.
- Publisher
- Rabanser and colleagues, Princeton University; ICML 2026 paper hosted on arXiv
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: Microsoft Research,
a stress test for long-horizon delegated execution.
Why it matters here: The authors warn that their benchmark does not measure overall task completion or user outcomes. They also explain that verification and specialized workflows can mitigate errors, limiting any broader inference that LLMs cannot be useful.
- Publisher
- Microsoft Research
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, The AI Hater's Manifesto
LLMs are sold as some sort of magic tool that can fix “anything” without ever specifying what that thing might be, mostly because they cannot be trusted, even in things that they mostly get right, to do things right every time.
- Where
- paragraph 13
- Published
- 2026-08-25
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-the-ai-haters-manifesto-ab77a8-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
HeldBy this site
The final-text check failed: citations_correct.
About assessment wyea-the-ai-haters-manifesto-ab77a8-a1, which was not published.