Five ways in which the last 3 months — and especially the DeepSeek era — have vindicated “Deep learning is hitting a wall"
We recorded 8 claims from this post. 5 were checked against evidence. 3 can’t be checked. 7 more are held. Read the post at its source.
Claims from this post (8)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Forecast
“A key claim of the paper was that pure scaling of LLMs – just adding more data and compute to extant LLM architectures – would not bring us to AGI, and that so-called scaling laws were merely empirical generalizations rather than physical laws.”
Our reading: Pure scaling of LLMs by adding data and compute to existing architectures will not reach AGI; scaling laws are empirical generalizations, not physical laws.
No deadline stated Evidence as of
Limitation: Public model documentation may not disclose enough to establish scaling-only provenance.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Another of the key suggestions I made in 2022 was that we should use neurosymbolic techniques, combining neural networks with classical symbolic techniques such as rules.”
Our reading: AI should combine neural networks with classical symbolic techniques such as rules.
No horizon
Assessment: SupportedSaid
Observation
“To some extent newer models are doing that.”
Our reading: Newer models such as DeepSeek R1, with a rule-based reward system, partly use neurosymbolic techniques.
No horizon Evidence as of
Limitation: The outcome depends on the broad, training-inclusive meaning supplied by the speaker's example.
Assessment: Can’t be checkedSaid
Observation
“People have been adding data and compute left and right since August 2022 when OpenAI demo’d GPT-4 to Bill Gates, but despite literally hundreds of billions of investment, pure LLM scaling has not produced the fruit some people imagined.”
Our reading: Despite hundreds of billions invested since August 2022, pure LLM scaling has not delivered what some expected.
No horizon Evidence as of
Limitation: This is not a finding that scaling met expectations or that the investment amount was false.
Assessment: UnresolvedSaid
Observation
“(Note that test-time compute systems are not across the board improvements like GPT-4 was relative to GPT-3 or GPT-3 was relative to GPT-2, but rather improvements in certain domains such as coding and math.)”
Our reading: Test-time compute systems improve only certain domains like coding and math, not across the board as GPT-4 did over GPT-3.
No horizon Evidence as of
Limitation: The decisive model evaluations are developer-reported.
Assessment: Can’t be checkedSaid
Observation
“We got one more huge advance since the 2022 paper (I was agnostic then as to how many more leaps there would be), but we still don’t have a system that would really merit the name of GPT-5.”
Our reading: Since 2022 there has been one more huge advance but no system that merits the name GPT-5.
No horizon
Assessment: SupportedSaid
Observation
“Even the latest systems like Deep Research are still struggling in a few ways – and those ways pretty much correspond exactly to the places that I warned would be LLM’s Achilles’ Heels: hallucinations and reasoning errors.”
Our reading: The latest systems such as OpenAI's Deep Research still suffer from hallucinations and reasoning errors.
No horizon Evidence as of
Limitation: Supports the existence of both error classes, not any particular frequency or severity.
Assessment: UnresolvedSaid
Observation
“LLMs, once novel, are largely a commodity.”
Our reading: LLMs have largely become a commodity.
No horizon Evidence as of
Limitation: The survey’s 2024 findings cannot be isolated to the target window [e1].
Held from this post (7)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: DeepSeek, roughly matching o1, accelerated price wars and forced OpenAI to drop prices.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: Marcus thinks it is fair to say that the 2022 essay did not anticipate how well a system like Deep Research might work, but that in most other respects, including the slowing of pure LLMs, the need for neurosymbolic AI, and the continued troubles with reasoning and hallucination, it was right.
- The record does not meet the current rules. Our reading, not assessed: If the scaling of pure LLMs ran out, one would expect multiple teams to compete and reach a point of diminishing returns, with essentially no moat and a lot of competition over price.
- The two reviews reached different assessments. Our reading, not assessed: The era of crowding at the top, with multiple teams competing at a point of diminishing returns, essentially no moat, and a lot of competition over price, has clearly been reached.
- The record does not meet the current rules. Our reading, not assessed: New techniques beyond pure LLMs, such as test-time compute, were needed, and test-time compute is working to some degree, for a while anyway, which bears out the 2022 essay's claim that new techniques besides pure LLMs would be needed.
- The criteria are ambiguous. Our reading, not assessed: The Deep Research tariff paper apparently made up many of its numbers.