“Which is to say it makes stuff up, regularly; a fair bit of what it is says simply isn’t true.”
Our reading, the sentence that was checked:
ChatGPT regularly makes things up and a fair amount of its output is false.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What to Expect When You’re Expecting … GPT-4, paragraph 4.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-to-expect-when-youre-expecting-8696cb-a1
We recorded 26 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-12-25. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- makes stuff up, regularly
- a fair bit
- simply isn’t true
Our interpretation
- The ordinary reading is recurring fabrication and a nontrivial amount of false factual content, rather than merely the possibility of an occasional error.
- No exact percentage or majority threshold is supplied or imposed.
- The target period is the initial public release through the post date, reflecting the post's current-state discussion.
Counts as supported if: Contemporary evaluations or well-documented observations establish recurring fabrication and materially frequent false factual content across ordinary uses.
Counts as contradicted if: Representative contemporary evaluations show that fabrication is exceptional and false factual content is negligible.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Stack Overflow's original policy provides substantial contemporary evidence of recurring incorrect answers in programming submissions. It supports part of Marcus's observation. It does not distinguish fabrication from other mistakes or establish the prevalence across ChatGPT's output generally. The verified records therefore do not settle the full frequency claim.
Limitations
- This does not mean fabrication was absent or rare.
- Moderators' observations were qualitative and concerned a selected use case.
- A narrower claim about frequent incorrect programming submissions would have stronger support.
Evidence (1 item)
Supports the claim: Stack Overflow moderators / Meta Stack Overflow,
temporarily banned.
Why it matters here: The accompanying explanation reports frequent incorrect answers among thousands of submissions and an overwhelmed moderation process. These firsthand qualitative observations support recurring false answers in programming submissions, but provide neither a general output denominator nor a separate fabrication rate.
- Publisher
- Stack Overflow moderators / Meta Stack Overflow
- Kind of source
- Independent measurement
- Describes events of
- Public from
What would settle it
Contemporary output samples across ordinary factual uses, with checked assertions, identified fabrications, and enough sampling information to assess whether false content was materially frequent.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What to Expect When You’re Expecting … GPT-4
Which is to say it makes stuff up, regularly; a fair bit of what it is says simply isn’t true.
- Where
- paragraph 4
- Published
- 2022-12-25
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-to-expect-when-youre-expecting-8696cb-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-to-expect-when-youre-expecting-8696cb-a1.