“Lots of quirky errors are already being reported, same as ever, like this reasoning error from Jane Rosenzweig:”
Our reading, the sentence that was checked:
Lots of quirky errors in GPT-4o are already being reported, same as ever, such as a reasoning error reported by Jane Rosenzweig.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Hot take on OpenAI’s new GPT-4o, paragraph 7.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-hot-take-on-openais-new-gpt-4o-53a025-a1
We recorded 5 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-05-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- same as ever (qualifier, stated by the speaker)
From the speaker
- Lots of quirky errors
- already being reported
- same as ever
- this reasoning error from Jane Rosenzweig
Our interpretation
- The relevant period is GPT-4o's launch day, May 13, 2024, up to the post's publication.
- The plain reading requires numerous identifiable error reports, including the named Rosenzweig example; one report alone is insufficient.
- 'Same as ever' refers to recurring kinds of reasoning mistakes and hallucinations, not an assertion that measured error rates were unchanged.
- The figure and error descriptions are observations, not conditional forecasts.
Counts as supported if: Contemporaneous, independently identifiable reports establish numerous GPT-4o errors of familiar types by publication, including an attributable Rosenzweig reasoning example.
Counts as contradicted if: Positive evidence establishes that the named example was not a GPT-4o reasoning error and that the broader collection relied upon was materially misattributed or postdated publication. Failure to find reports is not contradiction.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Riley's original post confirms that a GPT-4o reasoning-error report circulated on launch day. That supports part of Marcus's description. It does not establish 'lots' of reports, verify Rosenzweig's named example, or show that the reports preceded publication. The missing evidence is not evidence that Marcus was wrong.
Limitations
- Social-media text extraction omitted the underlying example.
- Anecdotal reports cannot establish an unchanged error rate.
- No contradictory outcome evidence was verified, so mixed is inappropriate.
Evidence (1 item)
Supports the claim: Benjamin Riley on X,
ChatGPT-4o is here and omg...it still can't handle a simple reasoning task that most adult humans can figure out.
Why it matters here: This establishes one explicit launch-day report of a GPT-4o reasoning failure. The extracted text does not expose the underlying interaction, establish numerous reports, or verify Rosenzweig's example.
- Publisher
- Benjamin Riley on X
- Kind of source
- Demonstration
- Describes events of
- Public from
What would settle it
Archived launch-day reports with timestamps, identifiable GPT-4o interactions, and Rosenzweig's original example, sufficient to assess the claimed abundance and recurring error types.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Hot take on OpenAI’s new GPT-4o
Lots of quirky errors are already being reported, same as ever, like this reasoning error from Jane Rosenzweig:
- Where
- paragraph 7
- Published
- 2024-05-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-hot-take-on-openais-new-gpt-4o-53a025-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-hot-take-on-openais-new-gpt-4o-53a025-a1.