“Because essentially all their software has a tendency to fabricate fairly regularly and even defame people and to sometimes give bad medical information, and so on.”
Our reading, the sentence that was checked:
Essentially all LLM providers' software fabricates fairly regularly, sometimes defaming people and giving bad medical information.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Maybe Section 230 doesn’t shield AI companies from liability, after all, paragraph 15.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-maybe-section-230-doesnt-shield-ai-6da00c-a1
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-06-11. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- essentially all their software
- fabricate fairly regularly
- even defame people
- sometimes give bad medical information
Our interpretation
- Read the scope as broad coverage of major LLM providers, rather than every individual model or software product.
- Read recurring fabrication as a reproducible tendency in realistic uses, without inventing a numerical frequency threshold.
- Read “defame” in its ordinary sense of false, reputation-damaging statements; a completed legal finding of defamation is not required.
- For this ongoing technical claim, use the current-year evaluation period before publication: January 1 through June 11, 2026. No standard quarterly reporting cycle exists.
- Older incidents provide context but cannot alone establish the state of current systems.
Counts as supported if: Contemporary, sufficiently broad evaluations establish recurring fabrication across major providers, with documented reputationally harmful falsehoods and erroneous medical outputs.
Counts as contradicted if: Representative contemporary evidence establishes that recurring fabrication is not broadly present, or directly refutes the claimed categories of harmful output.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The evidence supports a substantial part of the claim: multiple model families repeatedly produce unsupported information on demanding tasks, and medical evaluations identify serious errors. It does not settle the full breadth or ordinary-use frequency implied by “essentially all” and “fairly regularly.” The inspected reputational fabrication is older than the target period. These gaps prevent a supported verdict on the whole proposition; they do not demonstrate that it is false.
Limitations
- Difficult and adversarial benchmarks cannot be treated as random samples of everyday use.
- Models and safeguards change, so older incidents cannot establish current behavior by themselves.
- A false accusation does not automatically satisfy the legal elements of defamation.
Evidence (4 items)
Supports the claim: arXiv; Dongyang Fan and colleagues,
high hallucination rates.
Why it matters here: The February benchmark release reports substantial errors across proprietary and open-weight model families on difficult multi-turn legal, research, medical, and coding tasks. It supports recurring fabrication in those settings, but does not estimate ordinary-use frequency or establish every listed harm.
- Publisher
- arXiv; Dongyang Fan and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Proceedings of the AAAI Conference on Artificial Intelligence,
All models struggled
Why it matters here: The March publication reports clinically relevant errors across evaluated models, including a 29.1% rate for its best performer. Its tests deliberately inject hallucination traps, and the accessible abstract does not date the underlying experiments; it is therefore supporting context rather than decisive evidence of current ordinary-use frequency.
- Publisher
- Proceedings of the AAAI Conference on Artificial Intelligence
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: Quinn Emanuel Urquhart & Sullivan,
The accusation was invented.
Why it matters here: The summary describes ChatGPT inventing an embezzlement accusation against Mark Walters, but the incident predates the target period. It also explains that OpenAI won summary judgment, so fabricated reputational content must not be equated with established legal liability.
- Publisher
- Quinn Emanuel Urquhart & Sullivan
- Kind of source
- Secondary summary
- Describes events of
- Public from
Context only: arXiv; Dongyang Fan and colleagues,
hallucinate less.
Why it matters here: The paper finds meaningful variation across models and tasks and discusses much lower rates on easier evaluations. This limits extrapolation from its difficult benchmark to a general claim about frequency.
- Where
- Section 5.2
- Publisher
- arXiv; Dongyang Fan and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Dated, reproducible evaluations of contemporary models across the major providers, using representative tasks, together with authenticated reputational and medical falsehoods from the target period.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Maybe Section 230 doesn’t shield AI companies from liability, after all
Because essentially all their software has a tendency to fabricate fairly regularly and even defame people and to sometimes give bad medical information, and so on.
- Where
- paragraph 15
- Published
- 2026-06-11
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-maybe-section-230-doesnt-shield-ai-6da00c-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-maybe-section-230-doesnt-shield-ai-6da00c-a1.