“Models like o3 are far better than non-code-interpreter systems on some tasks but not all (for now o3 is actually worse than o1 on hallucinations) and effective for some tasks, but not others.”
Our reading, the sentence that was checked:
Models like o3 are far better than non-code-interpreter systems on some tasks but not all, and for now o3 is worse than o1 on hallucinations.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI, paragraph 108.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-o3-and-grok-4-accidentally-vindicated-5c5278-a1
We recorded 47 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-07-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- for now (qualifier, stated by the speaker)
- on some tasks but not all (scope, stated by the speaker)
From the speaker
- Models like o3
- far better than non-code-interpreter systems on some tasks but not all
- for now o3 is actually worse than o1 on hallucinations
- effective for some tasks, but not others
Our interpretation
- Assess the ordinary o3 release available when the post was written, rather than later models or o3-pro.
- Use the latest publicly reported launch evaluations of that release: April 16, 2025, through the July 13 post. Benchmark reports have no quarterly reporting cadence.
- Read 'far better' as a substantial demonstrated advantage on some tasks, not universal superiority.
- Read 'worse on hallucinations' as higher measured hallucination rates on comparable fact-seeking evaluations, not lower accuracy on every factual question.
- This is a performance comparison; it does not claim that code-interpreter access alone explains every difference.
Counts as supported if: Contemporaneous evaluations show substantial o3 advantages on some tasks and higher o3 hallucination rates than o1 on comparable tests.
Counts as contradicted if: Comparable evaluations show no substantial task advantages, or show o3 hallucinating no more than o1 on the relevant fact-seeking tests. Conflicting results across these components may warrant mixed.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The launch evaluations report substantial task improvements, while the system card reports higher o3 hallucination rates than o1 on both PersonQA and SimpleQA. This supports the claim's uneven-performance comparison. The same card shows higher o3 correct-answer rates, so 'worse on hallucinations' should not be expanded into worse factual performance overall.
Limitations
- These are company-published evaluations, including expert results summarized by the company.
- Hallucination and correct-answer rates measure different aspects of performance.
- The evidence supports the model comparison without proving that code-interpreter access caused the advantages.
Evidence (2 items)
Supports the claim: OpenAI,
Specifically, o3 tends to make more claims overall, leading to more accurate claims as well as more inaccurate/hallucinated claims.
Why it matters here: Table 4 reports hallucination rates of 0.33 for o3 versus 0.16 for o1 on PersonQA, and 0.51 versus 0.44 on SimpleQA. It also reports higher o3 accuracy on both tests, qualifying any broader inference that o3 is simply worse at factual answering.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: OpenAI,
It sets a new SOTA on benchmarks including Codeforces, SWE-bench (without building a custom model-specific scaffold), and MMMU.
Why it matters here: The launch report describes broad task advances and 20% fewer major errors than o1 in external expert evaluations. It introduces full tool access for these reasoning models and reports strong tool-enabled mathematics results, supporting substantial advantages on some tasks.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Models like o3 are far better than non-code-interpreter systems on some tasks but not all (for now o3 is actually worse than o1 on hallucinations) and effective for some tasks, but not others.
- Where
- paragraph 108
- Published
- 2025-07-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-o3-and-grok-4-accidentally-vindicated-5c5278-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-how-o3-and-grok-4-accidentally-vindicated-5c5278-a1.