“we should not take the apparent “confidence” of large language models — itself an illusion of how they are trained — to mean”
Our reading, the sentence that was checked:
The apparent confidence of LLMs is an illusion that comes from how they are trained.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Please don’t trust your chatbot for medical advice, paragraph 15.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-please-dont-trust-your-chatbot-for-f290cb-a1
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-04-21. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- apparent “confidence” of large language models
- itself an illusion
- of how they are trained
Our interpretation
- Read apparent confidence as expressed certainty, including verbalized confidence scores, rather than a claim about consciousness.
- In the surrounding warning, read 'illusion' as confidence that can misleadingly suggest correctness. Do not interpret it as asserting that every confidence signal contains zero information.
- Read training as a contributing cause of this mismatch, not necessarily its sole cause.
- Assess the causal parenthesis separately from the normative recommendation about trusting medical advice.
Counts as supported if: Controlled comparisons show that training increases expressed confidence beyond what correctness warrants, with interventions on training or rewards supporting a causal contribution.
Counts as contradicted if: Comparable experiments show that the claimed training-induced mismatch is absent or reversed across the relevant tested scope. Merely finding some informative confidence estimates does not refute the ordinary reading, but limits a universal interpretation.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Supported on the ordinary reading that apparent certainty can misleadingly suggest correctness and that training contributes. Controlled comparisons found greater verbalized overconfidence after preference training, reward models favored higher confidence for otherwise identical answers, and changes to reward training reduced the mismatch. Further experiments separated harmful instruction-tuning effects from chat-format effects. The wording is broader than the evidence if 'illusion' means that every confidence signal is meaningless: other experiments elicited informative, better-calibrated confidence estimates.
Limitations
- The result depends on reading 'illusion' as misleading certainty rather than complete absence of information.
- Most direct experiments concern prompted confidence scores and benchmark answers, not the tone of medical conversations.
- Training methods and models differ; the finding is not universal.
- The June 2026 paper was available by the evaluation date but after the post; the earlier training experiments independently support the assessment.
Evidence (3 items)
Supports the claim: ICLR 2025; author manuscript on arXiv,
greater verbalized overconfidence
Why it matters here: Before-and-after comparisons found increased expressed overconfidence after preference training. Reward experiments favored higher confidence for otherwise identical responses, while modified reward training reduced calibration error, supporting training as a contributing cause.
- Publisher
- ICLR 2025; author manuscript on arXiv
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Supports the claim: arXiv; manuscript marked accepted to ACL 2026 Findings,
this miscalibration
Why it matters here: Experiments across six open-weight models found that instruction tuning harmed calibration and chat formatting further increased confidence in models' own answers. This supports training as one cause while identifying an additional cause in how the interaction is presented.
- Where
- Abstract
- Publisher
- arXiv; manuscript marked accepted to ACL 2026 Findings
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: EMNLP 2023; author manuscript on arXiv,
better-calibrated than conditional probabilities
Why it matters here: Elicited confidence can track correctness better than token probabilities, with results varying by model and prompt. This limits the stronger interpretation that confidence is always meaningless; it does not establish that ordinary confident wording guarantees accuracy.
- Publisher
- EMNLP 2023; author manuscript on arXiv
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Please don’t trust your chatbot for medical advice
we should not take the apparent “confidence” of large language models — itself an illusion of how they are trained — to mean
- Where
- paragraph 15
- Published
- 2026-04-21
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-please-dont-trust-your-chatbot-for-f290cb-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-please-dont-trust-your-chatbot-for-f290cb-a1.