Gary Marcus vs Sam Altman on GPT-o1 aka Strawberry
We recorded 5 claims from this post. 3 were checked against evidence. 2 can’t be checked. 2 more are held. Read the post at its source.
Claims from this post (5)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Forecast
“Altman had, much to my surprise, just echoed my longstanding position that current techniques alone would not be enough to get to AGI”
Our reading: Current AI techniques alone will not be enough to reach AGI.
No deadline stated Evidence as of
Limitation: There is no generally accepted empirical test for AGI [e1].
Assessment: SupportedSaid
Observation
“I rushed out a hot take on X, my usual cautions about how LLMs aren’t really AGI:”
Our reading: LLMs, including OpenAI's new o1, are not really AGI.
No horizon Evidence as of
Limitation: The system card tests several model checkpoints and configurations; it is not an exhaustive evaluation of all contemporary LLMs [e3].
Assessment: Can’t be checkedSaid
Observation
“(An additional point I should have made is that the new system appears to depend heavily on synthetic data, and that such data may be easier to produce in some domains (such as those in which o1 is most successful, like some aspects of math) than others.)”
Our reading: o1 appears to depend heavily on synthetic data, and such data may be easier to produce in some domains (such as some aspects of math, where o1 is most successful) than in others.
No horizon
Qualifier: appears to depend heavily on synthetic data
Assessment: SupportedSaid
Observation
“Almost exactly what I predicted we would see with GPT-4, back on Christmas Day 2022.”
Our reading: Altman's description of o1 as 'still flawed, still limited, seem more impressive on first use' is almost exactly what Marcus predicted we would see with GPT-4 on Christmas Day 2022.
No horizon Evidence as of
Limitation: The agreement is thematic, not verbatim.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Gary Marcus continue to a dream of day in which AI research doesn’t center almost entirely around LLMs..”
Our reading: AI research should not center almost entirely on LLMs, as it currently does.
No horizon
Held from this post (2)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: OpenAI's o1 appears to depend heavily on synthetic data.