“You cannot rely on a large language model to do what you want.”
Our reading, the sentence that was checked:
Large language models cannot be relied on to do what users want.
Assessment: Supported Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, The Case Against Generative AI, paragraph 94.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-the-case-against-generative-ai-a42fdf-a1
We recorded 46 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-09-29. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- cannot rely on a large language model
- to do what you want
- can’t actually be relied upon to do exactly what you want
- correctly, or even consistently, every time
Our interpretation
- The ordinary contextual reading is that contemporary LLMs cannot generally be trusted to complete requested work correctly without checking, particularly complex or multistep work.
- This does not mean that LLMs never succeed, that every task is unreliable, or that useful systems must be mathematically infallible.
- Use September 2025 through publication as the latest contemporaneous observation window; there is no standardized reporting quarter for model reliability.
- Assess models and configurations available then. Later improvements do not settle this historical observation.
Counts as supported if: Contemporaneous evaluations show substantial failures to complete clear, realistic user tasks, including complex work emphasized in the context, sufficient to undermine general reliance without verification.
Counts as contradicted if: Broad, independently replicated evaluations establish consistently correct completion across representative routine and multistep user tasks without substantial human correction. Isolated successful demonstrations are insufficient.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The contextual warning against general reliance without checking is supported. September testing found very low completion rates for leading models on complex commercial coding tasks, directly relevant to the post's examples. GDPval also showed useful, competitive professional work, so the evidence does not support the stronger claim that every task or use is unreliable. This assessment depends on the stated ordinary reading.
Limitations
- The strongest outcome evidence concerns difficult software-engineering tasks.
- Performance depends on the model, task, tools, and execution setup.
- Successful assisted use is compatible with the supported warning.
- The cited tests do not establish universal or repeated-run unreliability.
Evidence (2 items)
Supports the claim: Scale AI Research,
17.8% resolution
Why it matters here: The September benchmark report found that leading models completed only a small minority of complex private-codebase tasks under the tested setup. This supports the warning against assuming correct autonomous completion of such work.
- Publisher
- Scale AI Research
- Kind of source
- Benchmark result
- Describes events of
- Public from
Context only: OpenAI,
approaching the quality
Why it matters here: GDPval supplies a material qualification: models produced competitive work across professional tasks. Its expert-preference comparisons and one-shot design do not establish consistent correctness over repeated runs.
- Publisher
- OpenAI
- Kind of source
- Benchmark result
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, The Case Against Generative AI
You cannot rely on a large language model to do what you want.
- Where
- paragraph 94
- Published
- 2025-09-29
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-the-case-against-generative-ai-a42fdf-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment wyea-the-case-against-generative-ai-a42fdf-a1.