“They can’t reliably sanity check themselves for hallucinations, and they can’t reliably sanity check for whether what they are doing is harmful.”
Our reading, the sentence that was checked:
LLMs cannot reliably check themselves for hallucinations or for whether their actions are harmful.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Humanity’s “Oh shit!” AI moment?, paragraph 6.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-humanitys-oh-shit-ai-moment-56953e-a1
We recorded 10 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-12-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- can’t reliably sanity check themselves for hallucinations
- can’t reliably sanity check for whether what they are doing is harmful
Our interpretation
- The plain reading concerns dependable detection of their own factual fabrications and harmful outputs or actions across ordinary uses.
- Reliable does not mean infallible. Neither occasional failures nor isolated successes settle broad dependability.
- Self-checking includes a fresh review by the same model, but excludes external factual answers or a separate specialist verifier.
- The two components require separate evidence. Generating harmful content, refusing a request, detecting harm, and correcting an answer are different measurements.
- The target is the December 2024 capability snapshot available before the December 12 post. Earlier model evaluations provide context unless they establish that current state.
Counts as supported if: Direct self-review evaluations of the relevant models show material, repeated detection failures across both hallucinations and harmful outputs or actions, sufficient to undermine broad dependability.
Counts as contradicted if: Broad, independently checked evaluations establish dependable self-detection for the relevant models and uses. Narrow successes alone do not settle the general claim.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The literature gives reasons to doubt broad self-checking reliability, but it also contains successful harm-screening experiments. Neither general correction failures nor harmful behavior by itself proves failure to detect hallucinations and harm. The located evidence does not settle both parts of this claim for the state described.
Limitations
- This is an unresolved combined claim, not a finding that self-checking is dependable.
- The positive harm-screening result is older and limited in scope.
- A numerical reliability threshold would be an evaluator addition.
Evidence (2 items)
Context only: Transactions of the Association for Computational Linguistics; author manuscript on arXiv,
These conflicting observations
Why it matters here: The December synthesis reports conflicting findings and weaknesses in evaluation design. It primarily reviews older research and does not directly measure both claimed capabilities in the December model population.
- Publisher
- Transactions of the Association for Computational Linguistics; author manuscript on arXiv
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: Phute and colleagues via arXiv,
reduces attack success rate to virtually 0
Why it matters here: GPT-3.5 and Llama 2 successfully screened generated harmful text in a limited experiment using a separate checking context. This is material evidence against a blanket inability, but concerns older models and a narrower task than general harmful-action assessment.
- Publisher
- Phute and colleagues via arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Direct evaluations of the models available at the post date, measuring their own hallucination and harmful-action detection across representative tasks, with independently checked labels and reported error rates.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Humanity’s “Oh shit!” AI moment?
They can’t reliably sanity check themselves for hallucinations, and they can’t reliably sanity check for whether what they are doing is harmful.
- Where
- paragraph 6
- Published
- 2024-12-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-humanitys-oh-shit-ai-moment-56953e-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-humanitys-oh-shit-ai-moment-56953e-a1.