“Unfortunately, nobody has ever fully implemented the comprehension challenge test that I proposed.”
Our reading, the sentence that was checked:
Nobody has fully implemented Marcus's proposed comprehension challenge test.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AI has (sort of) passed the Turing Test; here’s why that hardly matters, paragraph 32.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-ai-has-sort-of-passed-the-turing-f44fd7-a1
We recorded 9 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-04-02. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- nobody has ever
- fully implemented
- the comprehension challenge test that I proposed
- continuously update the questions
- the same footing as a first time human viewer
Our interpretation
- The historical interval runs from the June 9, 2014 proposal through the April 2, 2025 post.
- A full implementation operationalizes comprehension of arbitrary audiovisual narratives, including explanations of events and characters' motives, with a scoring procedure.
- The surrounding post makes protection against prior exposure and contamination part of the intended implementation.
- Implementing a test does not require an AI to pass it.
- Four seasons of Succession illustrates difficulty; it is not treated as a mandatory dataset.
- Related video-question-answering benchmarks are candidates to examine, not automatically full implementations.
Counts as supported if: A sufficiently comprehensive independent review establishes that no implementation meeting the stated scope and safeguards existed by the post date.
Counts as contradicted if: A dated implementation report, benchmark release, or evaluation protocol demonstrates that the complete challenge had been implemented by April 2, 2025.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Related benchmarks existed well before the post, including television, movie-story, and long-video comprehension tests. They prevent treating the area as unimplemented altogether. However, the reviewed records do not establish a complete implementation with the safeguards Marcus describes. Nor can an unsuccessful search establish his universal claim that nobody had implemented it.
Limitations
- The boundary between a related benchmark and full implementation is not formally defined.
- Public records cannot exclude undocumented implementations.
- An AI's failure to pass an existing benchmark would not support the claim that the benchmark was never implemented.
Evidence (3 items)
Context only: arXiv; Jie Lei and colleagues,
21,793 clips
Why it matters here: TVQA establishes that substantial television question-answering evaluations existed. Its clip-based scope does not by itself establish the complete challenge or its contamination safeguards.
- Where
- Abstract
- Publisher
- arXiv; Jie Lei and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Makarand Tapaswi and colleagues,
story comprehension from both video and text.
Why it matters here: MovieQA is a substantive related effort, including questions about why events occur. This first version does not document implementation of Marcus's complete protocol.
- Publisher
- arXiv; Makarand Tapaswi and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Weihan Wang and colleagues,
103 videos and 1,549 question-answer pairs
Why it matters here: LVBench implements long-video evaluation using public videos, multiple-choice questions, and human comparisons. These features make it a serious candidate, but the reviewed protocol does not establish the continuously refreshed, first-viewing safeguards described in the post.
- Publisher
- arXiv; Weihan Wang and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A dated complete implementation meeting the stated scope would contradict the claim. Support would require a comprehensive independent review against an agreed specification.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AI has (sort of) passed the Turing Test; here’s why that hardly matters
Unfortunately, nobody has ever fully implemented the comprehension challenge test that I proposed.
- Where
- paragraph 32
- Published
- 2025-04-02
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-ai-has-sort-of-passed-the-turing-f44fd7-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-ai-has-sort-of-passed-the-turing-f44fd7-a1.