“Over and over, the field of AI has discovered otherwise. Guy van den Broeck’s study is a perfect example of this.”
Our reading, the sentence that was checked:
The AI field has repeatedly found that doing well on a benchmark designed for X does not mean a system has actually solved X.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, The New Sport of Misrepresenting AI Criticism, paragraph 40.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-the-new-sport-of-misrepresenting-633bac-a2
We recorded 4 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-09-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- if a system does well on a benchmark that is designed to solve X, the system has actually solved X
- Over and over, the field of AI has discovered otherwise
- Guy van den Broeck’s study is a perfect example
Our interpretation
- The plain reading is that multiple studies have found benchmark success without reliable mastery of the intended task.
- I treat 'solved X' as reliable performance on valid instances of the intended task beyond the benchmark's incidental cues.
- The named study is assessed alongside other documented cases. 'Perfect example' is rhetorical emphasis rather than a separate measurable threshold.
- The claim describes the research record through the post date, with no stated beginning.
Counts as supported if: Multiple studies document benchmark success alongside failures on valid tests of the intended task, and the van den Broeck study demonstrates this mismatch.
Counts as contradicted if: The named study does not demonstrate the alleged mismatch, and the purported repeated examples do not survive examination of their tasks, controls, or results.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The May 2022 study coauthored by van den Broeck found that BERT achieved near-perfect test accuracy but failed when sampling changed within the same logical problem space [e4]. An independent April 2018 study found that inference labels could often be predicted without reading the premise, and that high-performing models did much worse on examples where this shortcut failed [e5]. Together, these studies establish repeated cases of benchmark success without reliable mastery of the intended task, including the named example [e4, e5].
Limitations
- The studies cover selected logical-reasoning and language-inference tasks, not all AI benchmarks or models [e4, e5].
- They do not establish that the tested models learned nothing useful [e4, e5].
- The assessed version of the van den Broeck study was a preprint [e4].
Evidence (2 items)
Supports the claim: arXiv,
correct reasoning function.
Why it matters here: Zhang and colleagues, including van den Broeck, found near-perfect performance within a training distribution but substantial failures under different sampling of the same logical problem space. Their analysis attributes this to statistical features rather than acquisition of the intended reasoning procedure.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv; NAACL 2018 author manuscript,
remains an open problem.
Why it matters here: Gururangan and colleagues found that labels could often be predicted without the premise needed for genuine inference. High-performing models also performed much worse on the subset where that shortcut failed, providing an independent earlier example.
- Publisher
- arXiv; NAACL 2018 author manuscript
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, The New Sport of Misrepresenting AI Criticism
Over and over, the field of AI has discovered otherwise. Guy van den Broeck’s study is a perfect example of this.
- Where
- paragraph 40
- Published
- 2022-09-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-the-new-sport-of-misrepresenting-633bac-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-the-new-sport-of-misrepresenting-633bac-a2.