OpenAI’s o3 and Tyler Cowen’s Misguided AGI Fantasy
We recorded 4 claims from this post. 3 were checked against evidence. 1 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (4)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
“without even acknowledging, for example, the massive problem LLMs have with fabricating citations”
What we tested: LLMs have a massive problem with fabricating citations.
What happened: The researchers documented invented source links across multiple systems, supporting a substantial citation-reliability problem. Their reported majority rate combines fabricated and broken URLs, so it is not a measured fabrication-only rate. (Tow Center for Digital Journalism / Columbia Journalism Review, )
Assessment: Can’t be checkedSaid
“His claims about o3 yesterday didn’t survive even the day.”
What we tested: Tyler Cowen's claim that OpenAI's o3 is AGI was refuted within a day of its release.
Assessment: UnresolvedSaid
“Mollick’s notion of “jagged” AI is exactly what I used to mean by the term “pointillisitic” and what I predicted about o3 in November. AI that works some of the time but not always, markedly better on some problems than others.”
What we tested: o3 works only some of the time and is markedly better on some problems than others, as Marcus predicted in November 2024.
Assessment: UnresolvedSaid
“my perspective that scaling was no longer working as effectively as it once did”
What we tested: Scaling of AI models is no longer working as effectively as it once did.
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: In Colin Fraser's more systematic study, o3 scored about 87% on adding largish two-digit integers, which is not miserable but far inferior to many systems that would score a perfect 100%, and is business as usual for LLMs.
- The two reviews reached different assessments. Our reading, not assessed: LLMs' jagged performance arises because they regurgitate training data with minor changes, doing well near prior data and often failing on subtly different cases.
- The two reviews read the claim differently