“Cherry-picked studies, benchmark-gaming, and now even vibe-coded graphs, with zero transparency about how systems work or how they have been trained; public science is in the rear view mirror.”
Our reading, the sentence that was checked:
AI has been marked by cherry-picked studies, benchmark-gaming and now even vibe-coded graphs, with zero transparency about how systems work or how they have been trained, and public science is in the rear view mirror.
Assessment: Mixed Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it., paragraph 77.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-gpt-5-overdue-overhyped-and-underwhelming-02efa1-a1
We recorded 23 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-08-09. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Cherry-picked studies
- benchmark-gaming
- now even vibe-coded graphs
- zero transparency about how systems work or how they have been trained
- public science is in the rear view mirror
Our interpretation
- The surrounding reference to the last several years covers recent industry practices through publication. I operationalize that imprecise period as January 2023 through August 9, 2025.
- Selective reporting and benchmark gaming mean practices that inflate apparent performance through selection or benchmark-specific optimization.
- Vibe-coded graphs means graphs produced using AI-assisted coding; incorrect graphs alone do not establish their provenance.
- Zero transparency is read as an absence of substantive public information about system design or training, not merely an inability to reproduce a complete commercial model.
- Public science being in the rear view mirror is rhetorical emphasis, without a separate measurable threshold.
- The components are assessed separately; evidence for one does not establish all of them.
Counts as supported if: Documented selective reporting or benchmark manipulation, verified AI-assisted graph production, and a demonstrated absence of substantive design or training disclosures support the respective components.
Counts as contradicted if: Substantive public architecture or training disclosures contradict the zero-transparency component. Verified non-AI graph production or evidence disproving a specified manipulation allegation would contradict those components.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Mixed
Part of the claim holds and part does not, and the record says which part is which.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Research supports the narrower allegation of selective benchmark reporting, although the benchmark operator disputes its magnitude and interpretation. The zero-transparency assertion is contradicted by substantive public architecture disclosures. Incorrect launch charts are documented, but their AI-assisted provenance remains unverified. This supports neither the entire accusation nor a dismissal of its benchmark concerns.
Limitations
- The judgment depends on reading zero transparency as no substantive disclosure, rather than insufficient disclosure for reproduction.
- Selective benchmark reporting does not establish that every cited study was cherry-picked or that all developers manipulated results.
- The chart evidence is reporting, not a production log.
Evidence (4 items)
Supports the claim: arXiv; Singh and colleagues,
We establish that the ability of these providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results.
Why it matters here: The study supports the selective benchmark-reporting component using observations of private testing and analysis of selection effects. Its private-variant observations cover January–March 2025; the operator disputes the effects' magnitude and some interpretations.
- Where
- Abstract
- Publisher
- arXiv; Singh and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Goes against the claim: Meta,
We use alternating dense and mixture-of-experts (MoE) layers for inference efficiency. MoE layers use 128 routed experts and a shared expert.
Why it matters here: These are concrete public details about a major developer's model architecture. They contradict an industry-wide absence of substantive transparency, while leaving open whether disclosure was sufficient for full reproduction.
- Where
- Pre-training
- Publisher
- Meta
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: The Verge,
In this same chart, o3 and GPT-4o’s scores are different but shown with equally-sized bars.
Why it matters here: The report documents inaccurate launch graphics and reproduces company explanations. It does not verify that AI produced the charts, so it cannot settle the vibe-coding allegation.
- Publisher
- The Verge
- Kind of source
- News report
- Describes events of
- Public from
Context only: LMArena,
Because Arena is constantly collecting fresh data from new users, selection bias quickly goes to zero.
Why it matters here: The benchmark operator disputes large or persistent score inflation and describes policy changes. This materially limits treating the paper as uncontested proof of widespread manipulation.
- Publisher
- LMArena
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Graph-production records, identified studies with complete reporting histories, and independent analysis of the disputed benchmark effects would settle the remaining components.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Cherry-picked studies, benchmark-gaming, and now even vibe-coded graphs, with zero transparency about how systems work or how they have been trained; public science is in the rear view mirror.
- Where
- paragraph 77
- Published
- 2025-08-09
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-gpt-5-overdue-overhyped-and-underwhelming-02efa1-a1: Current. Outcome: Mixed. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-gpt-5-overdue-overhyped-and-underwhelming-02efa1-a1.