โWhat was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.โ
Our reading, the sentence that was checked:
What was done for o3 on ARC, pretraining on what Marcus believes was hundreds of public examples, is not comparable to what humans require.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-c39-a0bac4-a1
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-12-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the releaseโs reference date)
Conditions
- what I believe was hundreds of public examples (qualifier, stated by the speaker)
From the speaker
- pretraining
- what I believe was hundreds of public examples
- NOT comparable to what humans require
Our interpretation
- The claim concerns the preparation of the o3 system presented on December 20, 2024.
- The plain reading is that o3 received substantially more ARC-specific preparation than humans need to attempt ARC successfully.
- I read 'pretraining' broadly as training before evaluation. The wording does not establish a particular training stage or a separate fine-tuned checkpoint.
- 'What I believe' qualifies the example count; it is not a conditional forecast.
- Public training tasks and the demonstration pairs within each task are different units.
Counts as supported if: The tested o3 received training on hundreds of public ARC tasks, while published human experiments demonstrate successful ARC solving after substantially less ARC-specific preparation.
Counts as contradicted if: Reliable records show that the tested o3 did not receive the described ARC training, or that humans needed comparable preparation on hundreds of ARC tasks.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The preparation comparison holds on the stated plain reading. ARC Prize reports training on 75% of the public training set, equivalent to 300 tasks. The human study demonstrates successful solving after a brief tutorial rather than hundreds of preparatory tasks. This supports a difference in ARC-specific preparation. It does not establish how much preparation humans need to match o3's score, or whether their total learning experience is comparable.
Limitations
- The conclusion depends on reading comparable as referring to ARC-specific preparation.
- The human study predates the o3 announcement and used different evaluation conditions.
- The ARC Prize page has subsequent updates and does not separately date its training clarification.
Evidence (2 items)
Supports the claim: ARC Prize,
OpenAI shared they trained the o3 we tested on 75% of the Public Training set.
Why it matters here: The evaluator confirms substantial ARC-specific training for the system presented that day. Applying the disclosed fraction to the 400-task training set described in e2 gives 300 tasks.
- Publisher
- ARC Prize
- Kind of source
- Benchmark result
- Arithmetic
- 0.75 ร 400 = 300 training tasks (inputs: 75% of the public training set, disclosed in e1; 400 public training tasks, described in e2; None.)
- Describes events of
- Public from
Context only: Solim LeGris and colleagues, arXiv,
After the tutorial, participants were asked to answer several comprehension questions to make sure they understood the instructions.
Why it matters here: The earlier human study describes one tutorial task followed by a short experimental session, rather than hundreds of preparatory tasks, and reports 64.2% average public-evaluation accuracy. It also identifies 400 training and 400 evaluation tasks; this is historical comparator evidence, not an experiment on the December o3 system.
- Publisher
- Solim LeGris and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, ๐ผ๐ฏ โ๐๐ฅ๐ ๐๐๐โ ๐ฝ๐ผ๐๐๐บ๐ผ๐ฟ๐๐ฒ๐บ ๐บ๐ฒ๐ด๐ฎ๐๐ต๐ฟ๐ฒ๐ฎ๐ฑ: ๐๐ต๐ ๐๐ต๐ถ๐ป๐ด๐ ๐ด๐ผ๐ ๐ต๐ฒ๐ฎ๐๐ฒ๐ฑ, ๐๐ต๐ฎ๐ ๐๐ฒ๐ป๐ ๐๐ฟ๐ผ๐ป๐ด, ๐ฎ๐ป๐ฑ ๐๐ต๐ฎ๐ ๐ถ๐ ๐ฎ๐น๐น ๐บ๐ฒ๐ฎ๐ป๐
What was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.
- Where
- paragraph 6
- Published
- 2024-12-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-c39-a0bac4-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.