“And the lower 56% is not much different from what Jacob Andreas’s lab at MIT got in November. It’s arguably worse; if I followed correctly, and if the measures are the same, Andreas lab’s best score was actually higher, at 61%.”
Our reading, the sentence that was checked:
The 56% is not much different from what Jacob Andreas's lab at MIT got in November, and arguably worse, since, if Marcus followed correctly and the measures are the same, the Andreas lab's best score was higher at 61%.
Assessment: Can’t be checked Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, OpenAI’s dirty December o3 demo doesn’t readily replicate, paragraph 5.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-openais-dirty-december-o3-demo-doesnt-eaeb66-a1
We recorded 8 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-04-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- if I followed correctly (qualifier, stated by the speaker)
- if the measures are the same (qualifier, stated by the speaker)
- arguably (qualifier, stated by the speaker)
From the speaker
- Jacob Andreas’s lab at MIT
- in November
- if I followed correctly
- if the measures are the same
- best score was actually higher, at 61%
Our interpretation
- November means November 2024 in this April 2025 post.
- The comparison explicitly depends on compatible evaluation measures, including the task split.
- The approximate numerical report can be checked separately, but the qualified ranking should not be converted into an unconditional assertion.
Counts as supported if: The November work reports a best score near 61%, and compatible evaluations establish that this exceeds the cited released-o3 result.
Counts as contradicted if: Comparable records establish that the November score was materially different or did not exceed the released-o3 score.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Can’t be checked
The statement is a recommendation, a value judgment, or too open to test. It is recorded, not scored.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The paper supports the approximate 61% recollection. The ranking is explicitly conditional, and the public and semi-private results do not satisfy that condition. It would be unfair to score the conditional ranking as an unconditional error.
Limitations
- This disposition concerns the qualified comparison, not the existence of the November result.
- Numerically nearby scores on different task sets do not establish similar capability.
Evidence (2 items)
Supports the claim: Akyürek et al., arXiv,
public validation accuracy of 61.9%
Why it matters here: The November paper supports the approximate numerical recollection. Its result is an ensemble's score on public validation tasks.
- Publisher
- Akyürek et al., arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: ARC Prize Foundation,
o3-medium is currently the strongest publicly available model we've tested.
Why it matters here: The accompanying table uses semi-private tasks, unlike the MIT paper's public validation tasks. The comparison condition is unmet.
- Publisher
- ARC Prize Foundation
- Kind of source
- Independent measurement
- Describes events of
- Public from
What would settle it
Matched evaluations of the November ensemble and released o3 on the same task set and scoring protocol.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, OpenAI’s dirty December o3 demo doesn’t readily replicate
And the lower 56% is not much different from what Jacob Andreas’s lab at MIT got in November. It’s arguably worse; if I followed correctly, and if the measures are the same, Andreas lab’s best score was actually higher, at 61%.
- Where
- paragraph 5
- Published
- 2025-04-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-openais-dirty-december-o3-demo-doesnt-eaeb66-a1: Current. Outcome: Can’t be checked. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-openais-dirty-december-o3-demo-doesnt-eaeb66-a1.