“But they remain superficial, in that their responses invariably have the right kind of language for any given problem, but (unlike the hypothetical Star Trek computer) rarely trustworthy.”
Our reading, the sentence that was checked:
LLMs remain superficial: their responses invariably have the right kind of language for any given problem, but they are rarely trustworthy.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AGI versus “broad, shallow intelligence”, paragraph 21.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-agi-versus-broad-shallow-intelligence-9e6c99-a1
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-01-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- they remain superficial
- their responses invariably have the right kind of language
- for any given problem
- rarely trustworthy
Our interpretation
- The ordinary reading is that contemporary LLM answers consistently sound appropriate to the task, while dependable answers are uncommon across ordinary uses.
- Trustworthy means sufficiently correct and reliable for the task, rather than merely fluent. It does not require perfect performance on every conceivable question.
- Superficial is assessed through the stated contrast between appropriate language and dependable answers, not as a separate claim about consciousness or internal understanding.
- Because this is an ongoing state without a reporting interval, the target is the latest reportable snapshot immediately before publication, January 13, 2025. Earlier evaluations are context unless they establish that snapshot.
- No numerical cutoff for 'rarely' is imposed; the evidence must establish that dependable answers are clearly uncommon or clearly common across a representative task mix.
Counts as supported if: Representative evaluations of contemporary systems show consistently appropriate language but dependable answers only as an uncommon exception across ordinary tasks.
Counts as contradicted if: Representative evaluations show that dependable answers are common across ordinary tasks, or verified in-scope responses clearly fail the asserted invariant of task-appropriate language.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The claim has an ordinary empirical reading, but the recovered evidence does not settle its breadth. Reliability research documents plausible errors and inconsistent behavior. A company coding evaluation also reports substantial success. Neither establishes how often ordinary contemporary answers were dependable, and neither tests invariant linguistic appropriateness across problems. Scoring the claim from selected failures or selected benchmark successes would impose a task distribution Marcus did not supply.
Limitations
- The cited evaluations predate the target snapshot and are context only.
- Correct answers on selected tasks do not establish dependable behavior across uses.
- The coding result is company-reported and involves tool-assisted work.
- A numerical interpretation of rarity could change the assessment.
Evidence (2 items)
Context only: Nature; Zhou and colleagues,
However, larger and more instructable large language models may have become less reliable.
Why it matters here: The abstract reports plausible wrong answers, difficulties supervising errors and remaining sensitivity to phrasing. Those findings inform the reliability concern, but do not establish that dependable answers were rare across ordinary January 2025 uses.
- Publisher
- Nature; Zhou and colleagues
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: Anthropic,
In an internal agentic coding evaluation, Claude 3.5 Sonnet solved 64% of problems, outperforming Claude 3 Opus which solved 38%.
Why it matters here: The company reports successful solutions on a majority of its selected coding problems. This weighs against treating successful answers as universally rare, but an internal, tool-assisted coding evaluation does not measure general trustworthiness or the January snapshot.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Representative evaluations of systems available at publication, with separate ratings of task-appropriate language and dependable answers, repeated trials, and explicit coverage of ordinary uses.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AGI versus “broad, shallow intelligence”
But they remain superficial, in that their responses invariably have the right kind of language for any given problem, but (unlike the hypothetical Star Trek computer) rarely trustworthy.
- Where
- paragraph 21
- Published
- 2025-01-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-agi-versus-broad-shallow-intelligence-9e6c99-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-agi-versus-broad-shallow-intelligence-9e6c99-a1.