“There’s some information in there, but whatever’s there is often both patchy and problematic.”
Our reading, the sentence that was checked:
Chatbots contain some information, but whatever is there is often both patchy and problematic.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What was 60 Minutes thinking, in that interview with Geoff Hinton?, paragraph 52.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-was-60-minutes-thinking-in-that-77bf3b-a1
We recorded 20 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-10. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- often (qualifier, stated by the speaker)
From the speaker
- some information in there
- often
- patchy and problematic
Our interpretation
- The plain reading is that contemporary chatbots can supply correct information but repeatedly produce factual gaps, unsupported assertions, or inconsistent access to related facts.
- 'Often' means a recurring, substantial problem in ordinary factual tasks, not that most answers to every kind of prompt are wrong.
- Outputs provide an operational test of accessible information; they do not reveal everything encoded internally.
- I use 2023 through publication as the latest partial calendar year for the current generation of chatbots, which has no regular reporting period.
Counts as supported if: Independent evaluations of contemporary chatbots show useful factual knowledge alongside substantial recurring factual errors or inconsistencies across more than isolated anecdotes.
Counts as contradicted if: Broad evaluations show reliable factual and relational answers across relevant tasks, with the alleged problems limited to rare or unrepresentative cases.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Two contemporary studies support the ordinary reading. Biography generation contained substantial unsupported factual content, and GPT-4 answered related family-relationship questions very unevenly. Both studies also show correct factual answers, consistent with Marcus's acknowledgment that the systems contain information. Their results establish recurring problems in tested tasks, not a universal error rate.
Limitations
- Biography and family-relationship tasks are not representative of every chatbot use.
- FActScore measures support against a reference source; unsupported does not always mean demonstrably false.
- Behavioral tests do not fully characterize internal knowledge representations.
- Both cited versions were preprints.
Evidence (2 items)
Supports the claim: arXiv,
Factual precisions of InstructGPT and ChatGPT are 42.5% and 58.3%, respectively.
Why it matters here: Human evaluation of generated biographies found substantial mixtures of supported and unsupported atomic facts. This supports both useful factual content and recurring reliability problems within that task.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv,
GPT-4 correctly answers questions like the former 79% of the time, compared to 33% for the latter.
Why it matters here: The former questions ask for a celebrity's parent; the latter reverse the relationship. The large accuracy difference demonstrates uneven access to related factual information while also documenting considerable successful recall.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What was 60 Minutes thinking, in that interview with Geoff Hinton?
There’s some information in there, but whatever’s there is often both patchy and problematic.
- Where
- paragraph 52
- Published
- 2023-10-10
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-was-60-minutes-thinking-in-that-77bf3b-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-was-60-minutes-thinking-in-that-77bf3b-a1.