โSame was true of the openAI graph: the MIT work (halfway in between o1 and o3) and many others results werenโt shown, making the breakthrough relative to the field seem far bigger than it really was.โ
Our reading, the sentence that was checked:
OpenAI's graph left out the MIT work (halfway between o1 and o3) and many other results, which made o3's breakthrough relative to the field seem far bigger than it really was.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-c39-73750f-a1
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-12-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the releaseโs reference date)
From the speaker
- the MIT work
- halfway in between o1 and o3
- many others results weren't shown
- making the breakthrough relative to the field seem far bigger
Our interpretation
- The causal assertion concerns the apparent numerical performance gap created by OpenAI's December 20, 2024 graph.
- The plain reading is that adding relevant omitted prior results would substantially reduce the apparent advance over the field.
- Halfway is read as approximately midway, not exact equality.
- The mechanism can be assessed by reconstructing the comparison; actual changes in audience beliefs would require separate evidence.
Counts as supported if: The original graph omits the identified MIT result and other comparable results, MIT falls approximately midway between the displayed o1 and o3 results, and adding the omitted results substantially reduces the apparent gap from the prior field frontier.
Counts as contradicted if: The graph includes those comparators, MIT is not approximately intermediate under matching conditions, or a corrected comparison does not materially reduce the apparent advance.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The MIT paper confirms a prior 61.9% ensemble result on public validation. It does not establish the claimed omission or causal exaggeration in OpenAI's graph. The graph's exact entries and evaluation settings were not recovered reliably, so neither the halfway description nor the corrected performance gap can be verified. An intermediate score on a different test set would not settle the claim.
Limitations
- The identified MIT figure is an ensemble result, not simply the standalone test-time-training model.
- A numerical reconstruction would assess the presentation mechanism, not prove how viewers actually changed their beliefs.
Evidence (1 item)
Context only: Ekin Akyรผrek and colleagues, arXiv,
By ensembling our method with recent program generation approaches, we get SoTA public validation accuracy of 61.9%, matching the average human score.
Why it matters here: The MIT paper establishes a substantial prior result, specifically an ensemble score on public validation. It does not establish that this result was omitted from OpenAI's graph or that it was halfway between that graph's matched o1 and o3 scores.
- Where
- Abstract, version 1
- Publisher
- Ekin Akyรผrek and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
The original OpenAI graph with readable labels and presentation context, followed by a comparison using the matching MIT result and other contemporaneous systems.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, ๐ผ๐ฏ โ๐๐ฅ๐ ๐๐๐โ ๐ฝ๐ผ๐๐๐บ๐ผ๐ฟ๐๐ฒ๐บ ๐บ๐ฒ๐ด๐ฎ๐๐ต๐ฟ๐ฒ๐ฎ๐ฑ: ๐๐ต๐ ๐๐ต๐ถ๐ป๐ด๐ ๐ด๐ผ๐ ๐ต๐ฒ๐ฎ๐๐ฒ๐ฑ, ๐๐ต๐ฎ๐ ๐๐ฒ๐ป๐ ๐๐ฟ๐ผ๐ป๐ด, ๐ฎ๐ป๐ฑ ๐๐ต๐ฎ๐ ๐ถ๐ ๐ฎ๐น๐น ๐บ๐ฒ๐ฎ๐ป๐
Same was true of the openAI graph: the MIT work (halfway in between o1 and o3) and many others results werenโt shown, making the breakthrough relative to the field seem far bigger than it really was.
- Where
- paragraph 8
- Published
- 2024-12-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-c39-73750f-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.