“It makes current AI hard to debug (nobody knows for example how to ground them in facts), and it makes them difficult predict, which means, unlike calculators or spreadsheets, we don’t really know what’s going to happen when we ask them a question.”
Our reading, the sentence that was checked:
Because we don't really understand how trained neural networks do what they do, current AI is hard to debug (for example, nobody knows how to ground them in facts) and difficult to predict, so unlike calculators or spreadsheets we don't really know what will happen when we ask them a question.
Assessment: Mixed Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What was 60 Minutes thinking, in that interview with Geoff Hinton?, paragraph 59.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-was-60-minutes-thinking-in-that-23d4af-a2
We recorded 20 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-10. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- because we don't really understand how the trained networks do those things (qualifier, stated by the speaker)
From the speaker
- current AI hard to debug
- nobody knows for example how to ground them in facts
- difficult predict
- unlike calculators or spreadsheets
Our interpretation
- Current AI means the contemporary language-model chatbots discussed in the surrounding passage, not every kind of AI.
- The causal claim is that limited understanding of internal mechanisms obstructs diagnosing failures and predicting responses.
- Grounding ordinarily means connecting generated answers to checked factual sources. Marcus does not explicitly require perfect factual accuracy in this sentence.
- The assessment covers the 2023 technical reporting window through publication. These systems have no standard quarterly reporting period.
- Difficulty predicting responses does not mean that no behavior or aggregate performance can be predicted.
Counts as supported if: Contemporary research connects limited mechanistic understanding to difficulties diagnosing and controlling model behavior, documents unreliable response behavior, and establishes that factual grounding was unavailable.
Counts as contradicted if: Demonstrated factual-grounding methods contradict the categorical parenthetical. Evidence of dependable general prediction or easy diagnosis despite opacity would contradict the broader engineering claim.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Mixed
Part of the claim holds and part does not, and the record says which part is which.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The concern about limited understanding has support: Anthropic's October 5, 2023 report described uninterpretable neurons as a roadblock to mechanistic understanding, while showing progress in interpreting and steering a small model [e1]. A study published July 18 found substantial behavioral changes between March and June 2023 versions of GPT-3.5 and GPT-4, supporting practical uncertainty across updates [e3]. The categorical claim that nobody knew how to ground these systems in facts does not hold under source-based grounding: the May 23, 2023 WikiChat paper described retrieving information and individually verifying proposed facts against Wikipedia [e2]. This evidence supports interpretability obstacles and behavioral uncertainty, but does not establish how much limited understanding caused debugging or prediction difficulties [e1, e3].
Limitations
- The supplied Anthropic passage describes interpretability obstacles, rather than directly measuring difficulty diagnosing or fixing failures; its demonstrated progress concerns a small model [e1].
- Changes between model versions do not establish unpredictability within a fixed model or isolate the causal contribution of opacity [e3].
- WikiChat demonstrates source-based grounding, not guaranteed factual accuracy [e2].
Evidence (3 items)
Supports the claim: Anthropic,
a serious roadblock
Why it matters here: The opening paragraph explicitly connects poor understanding with difficulty diagnosing failures, fixing them, and certifying safety. The research also demonstrates limited progress in interpreting and steering a small model, so it supports difficulty rather than total ignorance.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: arXiv; Lingjiao Chen, Matei Zaharia and James Zou,
vary greatly
Why it matters here: Comparisons of March and June versions support practical uncertainty about service behavior. They do not isolate mechanistic opacity as the cause or show that a fixed model's outputs are inherently unpredictable.
- Publisher
- arXiv; Lingjiao Chen, Matei Zaharia and James Zou
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Goes against the claim: arXiv; Sina J. Semnani and colleagues,
grounded with live information
Why it matters here: The original paper describes retrieval and individual verification of proposed facts against Wikipedia, with improved factual accuracy. This contradicts the absence of grounding methods, but does not establish error-free generation.
- Publisher
- arXiv; Sina J. Semnani and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What was 60 Minutes thinking, in that interview with Geoff Hinton?
It makes current AI hard to debug (nobody knows for example how to ground them in facts), and it makes them difficult predict, which means, unlike calculators or spreadsheets, we don’t really know what’s going to happen when we ask them a question.
- Where
- paragraph 59
- Published
- 2023-10-10
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-was-60-minutes-thinking-in-that-23d4af-a2: Current. Outcome: Mixed. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-was-60-minutes-thinking-in-that-23d4af-a2.