“Bottom line: From the outset Large Language Models like GPT-3 have great at generating surrealist prose, and they can beat a lot of benchmarks, but they are not (and may never be) great tech for reliably inferring user intent from what users say.”
Our reading, the sentence that was checked:
LLMs like GPT-3 are not great technology for reliably inferring user intent from what users say.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How come smart assistants have virtually no ability to converse, despite all the spectacular progress with large language models?, paragraph 23.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-come-smart-assistants-have-virtually-b38685-a1
We recorded 10 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-11-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Large Language Models like GPT-3
- they are not
- great tech for reliably inferring user intent from what users say
- may never be
Our interpretation
- The checkable assertion is about contemporary capability. May never be is a hedged possibility, not a separate forecast of permanent failure.
- The ordinary reading concerns dependable understanding of intended meaning across ordinary language, including implications and context. It does not deny useful performance on bounded classification tasks.
- Substantial recurring errors on simple tests of intended meaning count against dependable general understanding; occasional errors alone do not.
- The present-tense assertion is assessed over the current research year through November 22, 2022. From the outset principally introduces the contrasting statement about prose and benchmarks.
Counts as supported if: Contemporary evaluations of GPT-3-class models show substantial, systematic difficulty recovering intended meaning, including with reasonable prompting or instruction tuning.
Counts as contradicted if: Contemporary evaluations demonstrate consistently dependable recovery of intended meaning across representative explicit, indirect, and context-dependent requests.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Under the ordinary reading of dependable contextual understanding, contemporary evidence supports the claim. A pre-post study found substantial difficulty recovering implied meaning, including a remaining human-performance gap for adapted models. Instruction tuning already produced real gains, so the evidence does not support treating LLMs as useless or permanently incapable. The assessment is limited to the 2022 capability claim.
Limitations
- The principal evidence is a research benchmark, not a representative household deployment study.
- The result depends on reading user intent broadly, rather than as a narrow classification task.
- May never be is not established by these results.
Evidence (2 items)
Supports the claim: Ruis et al., arXiv,
a significant gap
Why it matters here: The study tests implied meaning rather than fluent text production and finds substantial shortcomings even in adapted models. It supports the ordinary reading that dependable contextual understanding remained a weakness, while acknowledging meaningful improvements.
- Where
- Abstract
- Publisher
- Ruis et al., arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Ouyang et al., arXiv,
more reliably follow explicit constraints
Why it matters here: Instruction tuning had already improved performance on actual user prompts. This qualifies any reading that LLMs were useless for intent-related tasks, but relative improvement does not establish dependable general understanding.
- Publisher
- Ouyang et al., arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How come smart assistants have virtually no ability to converse, despite all the spectacular progress with large language models?
Bottom line: From the outset Large Language Models like GPT-3 have great at generating surrealist prose, and they can beat a lot of benchmarks, but they are not (and may never be) great tech for reliably inferring user intent from what users say.
- Where
- paragraph 23
- Published
- 2022-11-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-come-smart-assistants-have-virtually-b38685-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-how-come-smart-assistants-have-virtually-b38685-a1.