What “game over” for the latest paradigm in AI might look like
We recorded 13 claims from this post. 6 were checked against evidence. 7 can’t be checked. 1 more is held. Read the post at its source.
Claims from this post (13)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
“His hypothesis, now generally called scaling maximalism, remains extremely popular, in no small part because bigger and bigger models have indeed continued to do ever more impressive things.”
What we tested: Scaling maximalism remains extremely popular, in no small part because bigger and bigger AI models have continued to do ever more impressive things.
Assessment: SupportedSaid
“Scaling is an empirical observation, not a guaranteed, continued law of nature.”
What we tested: AI scaling is an empirical observation, not a law of nature that is guaranteed to keep holding.
What happened: The foundational paper presents empirical relationships and explicitly qualifies their theoretical status. Appendix C and section 6.3 discuss limits to trusting or extending those relationships, supporting the absence of an indefinite guarantee. (Jared Kaplan and colleagues, arXiv, )
Assessment: Can’t be checkedSaid
“The proof makes too many assumptions to be taken as gospel, but if it is even close to correct, there may soon be real trouble in Scaling City.”
What we tested: If the Merrill, Warstadt and Linzen proof is even close to correct, scaling maximalism may soon be in real trouble.
Assessment: Can’t be checkedSaid
“There might not be enough data in the world to make scaling maximalism work.”
What we tested: There may not be enough data in the world for scaling maximalism to work.
Assessment: Can’t be checkedSaid
“There might not be enough available compute in the world to make scaling maximalism feasible.”
What we tested: There may not be enough available compute in the world to make scaling maximalism feasible.
Assessment: SupportedSaid
“Scaling here was largely AWOL; even the best model was only at 80.6%, and for most models, scaling had at best a neglible effect.”
What we tested: On the Ruis et al. pragmatics task, the best model reached only 80.6%, and for most models scaling had at best a negligible effect.
What happened: The best overall mean is 80.6% with 30 examples; Section 4.1 reports near-zero or decreasing slopes for many families, with only Cohere, T0 and InstructGPT-3 performing best at their largest tested size. The same source qualifies the headline: structured prompts average 81.7%, and Table 58 includes an individual-template score of 82.8%. (arXiv; Laura Ruis and colleagues, )
Assessment: Can’t be checkedSaid
“If you get syntax and semantics but fail on pragmatics or commonsense reasoning, you don’t have AGI you can trust.”
What we tested: A system that gets syntax and semantics right but fails on pragmatics or commonsense reasoning is not AGI you can trust.
Assessment: Can’t be checkedSaid
“What hit me hard, as I was reading the paper, is that asymptotic 80% performance on even a single important task like this might spell game over for scaling.”
What we tested: If performance plateaus at about 80% on even one important task like pragmatics, that may mean game over for scaling.
Assessment: UnresolvedSaid
“As I have long argued, capturing this without cognitive models and common sense is really hard.”
What we tested: Capturing the pragmatic implications of language without cognitive models and common sense is really hard.
Assessment: Can’t be checkedSaid
“Some important tasks might simply not scale.”
What we tested: Some important tasks may simply not improve with scale.
Assessment: UnresolvedSaid
“Scaling maximalism is an interesting hypothesis, but I stand by my prediction that it won’t get us to AGI.”
What we tested: Scaling maximalism will not get us to AGI.
Assessment: UnresolvedSaid
“Any one of them would mean we need a real paradigm shift, if we are to get to AGI.”
What we tested: If any one of the three failure modes (not enough data, not enough compute, tasks that don't scale) holds, reaching AGI will need a real paradigm shift.
Assessment: Can’t be checkedSaid
“All of this points to the same conclusion: if we want to reach AGI we shouldn’t keep putting so many eggs in the scaling-über-alles basket.”
What we tested: If the goal is AGI, the AI field should stop putting so much of its effort into scaling alone.
Held from this post (1)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: Gains on some measures over months or years do not show that scale is all AI needs.