Please don’t trust your chatbot for medical advice
We recorded 6 claims from this post. 4 were checked against evidence. 2 can’t be checked. 3 more are held. Read the post at its source.
Claims from this post (6)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“That’s still true – and it very much applies in medicine.”
Our reading: Large language models are still frequently wrong yet confident, producing authoritative bullshit that shouldn't be trusted, and this applies strongly in medicine.
No horizon Evidence as of
Limitation: Support depends on the contextual evidence-period reading.
Assessment: SupportedSaid
Observation
“Because a large fraction of the population has begun to turn to chatbots for medical advice.”
Our reading: A large fraction of the population has begun using chatbots for medical advice.
No horizon Evidence as of
Limitation: The U.S.-adult population is an interpretation, not an explicit commitment by Marcus.
Assessment: SupportedSaid
Observation
“All of this – the hallucinations, mistakes, and overconfidence – is entirely typical of LLMs, and entirely problematic in medicine.”
Our reading: Hallucinations, mistakes and overconfidence are entirely typical of LLMs and entirely problematic in medicine.
No horizon Evidence as of
Limitation: The decisive audit uses misinformation-oriented prompts and older model versions.
Assessment: Can’t be checkedSaid
Causal claim
“(An expert doctor might use the LLM to better effect, by asking the right questions.)”
Our reading: An expert doctor might use an LLM to better effect than lay users by asking the right questions.
No horizon
Qualifier: by asking the right questions
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“There will always be better models, but for now, and until proven otherwise, we should not take the apparent “confidence” of large language models — itself an illusion of how they are trained — to mean that we should trust large language with our lives.”
Our reading: For now and until proven otherwise, people should not take LLMs' apparent confidence as reason to trust them with their lives.
No horizon
Qualifier: for now
Assessment: SupportedSaid
Causal claim
“we should not take the apparent “confidence” of large language models — itself an illusion of how they are trained — to mean”
Our reading: The apparent confidence of LLMs is an illusion that comes from how they are trained.
No horizon Evidence as of
Limitation: The result depends on reading 'illusion' as misleading certainty rather than complete absence of information.
Held from this post (3)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: Converging evidence from four studies shows chatbots, especially used by amateurs, cannot be trusted for medical advice.
- The criteria are ambiguous. Our reading, not assessed: LLMs don't know what they don't know: they work decently well with the information they've got but don't know how to conduct clinical interviews, and in the hands of the lay public they can easily give bad advice because the proper questions never get asked, either by the patient or the LLM.
- Attribution is uncertain