Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

OpenAI’s dirty December o3 demo doesn’t readily replicate

We recorded 8 claims from this post. 3 were checked against evidence. 5 can’t be checked. 7 more are held. Read the post at its source.

Claims from this post (8)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: UnresolvedSaid

    “There were problems with experimental design, misleading graphs that left out competing work, and more.”

    What we tested: OpenAI's December 2024 o3 demo had experimental design problems and misleading graphs that omitted competing work.

  2. Assessment: SupportedSaid

    “OpenAI had privileged access to data their competitors didn’t have, but didn’t acknowledge this.”

    What we tested: OpenAI had privileged access to FrontierMath data that competitors lacked and did not acknowledge it.

    What happened: Epoch retrospectively states that OpenAI owned the commissioned problems and could access problems and solutions subject to a holdout exception, while sharing with other parties required OpenAI's written permission. It distinguishes the partnership disclosure around the o3 announcement from the data-access arrangement, which it acknowledges was not clarified then. (Epoch AI, )

  3. Assessment: SupportedSaid

    “the crown jewel that they reported on the demo — the 75% on Francois Chollet’s ARC test (once called ARC-AGI) doesn’t readily replicate.”

    What we tested: OpenAI's reported 75% o3 score on ARC-AGI does not readily replicate.

    What happened: As reported April 22, o3-medium scored 53% on ARC-AGI-1's semi-private set, while o3-high returned only 37 of 100 tasks; the evaluator rejected its 82% returned-task accuracy as unrepresentative. The same report affirms the December result but explains that production o3 differed in model, training and available compute, supporting practical non-reproduction rather than invalidity of the original experiment. (ARC Prize, )

  4. Assessment: Can’t be checkedSaid

    “And the lower 56% is not much different from what Jacob Andreas’s lab at MIT got in November. It’s arguably worse; if I followed correctly, and if the measures are the same, Andreas lab’s best score was actually higher, at 61%.”

    What we tested: The 56% is not much different from what Jacob Andreas's lab at MIT got in November, and arguably worse, since, if Marcus followed correctly and the measures are the same, the Andreas lab's best score was higher at 61%.

  5. Assessment: Can’t be checkedSaid

    “Forgive if me I am getting Theranos vibes.”

    What we tested: Marcus says he is getting Theranos vibes from OpenAI's handling of the o3 results.

  6. Assessment: Can’t be checkedSaid

    “some possible explanations are given, but this is again a very bad look.”

    What we tested: The gap between OpenAI's reported 32% and Epoch's 11% for o3-mini on FrontierMath is again a very bad look for OpenAI, even though some possible explanations have been given.

  7. Assessment: Can’t be checkedSaid

    “The truth is that we don’t really know how good o3 is or isn’t, and nobody should ever take OpenAI’s video presentations particularly seriously again, until they have been fully vetted by the community.”

    What we tested: We don't really know how good o3 is or isn't, and nobody should ever again take OpenAI's video presentations particularly seriously until the community has fully vetted them.

  8. Assessment: Can’t be checkedSaid

    “And given that Meta also appears to have just juiced some benchmarks, the whole thing is starting to look like a bunch of over-promisers scrambling to make things look better than they really are.”

    What we tested: Given that Meta also appears to have just juiced some benchmarks, the whole thing is starting to look like a bunch of over-promisers scrambling to make things look better than they really are.

Held from this post (7)

These records did not pass every check, so nothing about them is published. Held is not a finding.