“What I proposed then, as a replacement to the Turing Test, was a Comprehension Challenge; even now, no software would be able to meet that challenge.”
Our reading, the sentence that was checked:
As of December 2023, no AI software could meet Marcus's Comprehension Challenge.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Nope, the Turing Test has not been solved, paragraph 10.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-nope-the-turing-test-has-not-been-81898b-a1
We recorded 4 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-12-09. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- Comprehension Challenge proposed by Marcus in 2014 (qualifier, stated by the speaker)
From the speaker
- a Comprehension Challenge
- even now
- no software would be able to meet that challenge
Our interpretation
- This describes capability at the post date, rather than a forecast.
- The referenced 2014 proposal asks software to watch arbitrary television programs or YouTube videos and answer questions about their content, including explanations of motives and humor.
- The ordinary reading requires reliable comprehension across varied unfamiliar material, comparable to a competent human viewer; it does not require literal perfection on every imaginable question.
- For this ongoing capability claim, use December 2023 through December 9 as the latest reporting window. Earlier evaluations are context unless they establish capability in that window.
- Marcus's 2014 article defines the task but cannot establish that his 2023 capability assessment was correct.
Counts as supported if: A sufficiently broad evaluation of leading software existing by December 9, 2023 shows that none can reliably answer open-ended questions across the challenge's range of video content and reasoning requirements.
Counts as contradicted if: At least one system existing by December 9, 2023 demonstrably performs the full challenge reliably on varied unfamiliar videos, including explanatory questions, under a credible evaluation.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The sources show relevant abilities and limitations, but do not settle the full challenge. MovieChat evaluated open-ended long-video questions, and Google demonstrated reasoning over image sequences. Neither establishes reliable comprehension across arbitrary programs. Conversely, missing demonstrations and individual failures do not prove that every contemporary system lacked the capability.
Limitations
- The original challenge lacks a scoring protocol.
- The evidence includes developer evaluations and selected demonstrations.
- The search cannot establish a universal negative from missing public tests.
Evidence (2 items)
Context only: arXiv; Enxin Song and colleagues,
MovieChat’s performance is hindered by the pretrained short video understanding model.
Why it matters here: MovieChat reports gains on open-ended long-video questions, providing relevant evidence of progress. Its acknowledged perceptual and temporal limitations, and comparison with other models rather than a full challenge passing standard, leave Marcus's universal claim unsettled.
- Publisher
- arXiv; Enxin Song and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Google Developers Blog,
Of course, it won’t always get this challenge right.
Why it matters here: Google documents successful reasoning over image sequences while acknowledging failures on the cup task. These selected, prompted examples neither demonstrate the full Comprehension Challenge nor establish that Gemini could not perform it.
- Publisher
- Google Developers Blog
- Kind of source
- Demonstration
- Describes events of
- Public from
What would settle it
A reproducible evaluation of preserved December 2023 systems on varied unfamiliar videos, using open-ended explanatory questions and an explicit human comparison, or a documented contemporary system passing such an evaluation.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Nope, the Turing Test has not been solved
What I proposed then, as a replacement to the Turing Test, was a Comprehension Challenge; even now, no software would be able to meet that challenge.
- Where
- paragraph 10
- Published
- 2023-12-09
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-nope-the-turing-test-has-not-been-81898b-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-nope-the-turing-test-has-not-been-81898b-a1.