Ezra Klein’s new take on AGI – and why I think it’s probably wrong
We recorded 29 claims from this post. 16 were checked against evidence. 13 can’t be checked. 12 more are held. Read the post at its source.
Claims from this post (29)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“I completely agree with Klein that AGI will matter a lot when it comes.”
What we tested: When AGI arrives, it will matter a lot.
Assessment: UnresolvedSaid
“But it is not just that I think Klein is wrong, or that he did a disservice to the other side of the argument, but that words like his are motivating a lot of dodgy bets being placed on the imminent AGI assumption — in ways that may not make sense.”
What we tested: Claims like Klein's that AGI is imminent are motivating many dodgy bets that may not make sense.
Assessment: SupportedSaid
“But it is absolutely absurd to pretend “AGI imminent” hypothesis isn’t getting play in high places.”
What we tested: The hypothesis that AGI is imminent is getting play in high places, and it is absurd to pretend otherwise.
What happened: OpenAI's leader publicly expresses confidence in a path to AGI, alongside discussion of near-term agents and further advances. This establishes high-level attention, without proving any particular arrival date. (Sam Altman, )
Assessment: Can’t be checkedSaid
“And, importantly, the reverse of what Mollick said is also true: we have to consider the scenario that the proposition that AGI is imminent is false.”
What we tested: We have to consider the scenario in which the proposition that AGI is imminent is false.
Assessment: UnresolvedSaid
“(My view, as I explained in Taming Silicon Valley, is that industry has a vested interest in making people believe that AGI is imminent — how else to justify the lavish valuations - and that they have fooled a lot of people in government who aren’t technical experts into believing that AGI is closer than it really is).”
What we tested: The AI industry has a vested interest in claiming AGI is imminent to justify lavish valuations, and has misled many non-technical government officials into thinking AGI is closer than it is.
Assessment: SupportedSaid
“Every red cell in this graph (e.g, multiplying 20 digit numbers by 19 digit numbers2) is failure to generalize, even after immense amounts of training data:”
What we tested: LLMs including o1-mini fail to generalize on large multi-digit multiplication despite immense training data.
What happened: The original experiment reports a clear limit to reliable multiplication as input length grows. The thread identifies o1-mini and reports similar accuracy for o1-preview, supporting the claim about the tested models. (Yuntian Deng, reproduced by Thread Reader, )
Assessment: SupportedSaid
“• AGI should have reliable ability to reason over abstractions. (Ongoing unpredictable errors in LLMs reflect this.)”
What we tested: LLMs still lack a reliable ability to reason over abstractions, as their ongoing unpredictable errors show.
What happened: MIR-Bench tests fifteen models on deriving outputs from examples of underlying functions. Its controlled evaluations find substantial remaining failures, including in o1-family models, supporting the limited reliability claim. (Kai Yan and colleagues, arXiv, )
Assessment: SupportedSaid
“Hallucinations simply haven’t gone away, even in the latest models.”
What we tested: Hallucinations have not gone away, even in the latest models.
What happened: Table 4 reports PersonQA hallucination rates of 0.19 for GPT-4.5, 0.30 for GPT-4o, and 0.20 for o1. The release evaluation directly establishes persistence, while also showing improvement over the comparison models. (OpenAI, )
Assessment: UnresolvedSaid
“• AGI should have a reliable ability to understand complex syntactic and semantic structure. (Failings here are also often reflected in quirky errors.”
What we tested: Current AI systems lack a reliable ability to understand complex syntactic and semantic structure, and these failings are often reflected in quirky errors.
Assessment: UnresolvedSaid
“What that means is that a quarter century has passed without a principled solution to even one of those problems.”
What we tested: In the quarter century since 2001, none of the listed AGI prerequisites (generalization, abstract reasoning, tracking individuals, understanding structure) has received a principled solution.
Assessment: UnresolvedSaid
“Especially when we just saw GPT-5 (ultimately relabeled as GPT 4.5) fail despite more than two years of hard and expensive work.”
What we tested: OpenAI's planned GPT-5, relabeled GPT-4.5, failed despite more than two years of hard and expensive work.
Assessment: Can’t be checkedSaid
“A set of problems that have resisted solution for a quarter century, even backed by a half trillion dollars, probably aren’t all going to get solved so quickly.”
What we tested: The long-standing core AGI problems probably will not all be solved quickly, even with half a trillion dollars of funding.
Assessment: Can’t be checkedSaid
“Having observed the pace of science and engineering firsthand over the last half century, the middle scenario seems to me like – by far – the most plausible.”
What we tested: The most plausible scenario by far is that getting to AGI will be a slog: some problems get solved in the next 2-3 years and others take longer.
Assessment: UnresolvedSaid
“But there’s also no reason to think that any of them are logically insolvable in principle. So we will get there someday.”
What we tested: Because there is no reason to think any of the remaining problems are logically unsolvable in principle, AGI will be achieved someday.
Assessment: UnresolvedSaid
“The fact that GPT 4.5 received very mixed reviews — despite years of labor and billions investment — and could in no reasonable way be considered a major (rather than minor, incremental) step forwards towards AGI.”
What we tested: GPT-4.5 received very mixed reviews and is at most a minor, incremental step toward AGI, despite years of work and billions invested.
Assessment: Can’t be checkedSaid
“The fact that GPT 4.5 is wildly expensive relative to its competitors, and the fact that future models following the same trajectory of scale might be prohibitively expensive, especially in a regime in which nobody has a moat and price wars have been intensifying since the release of DeepSeek.”
What we tested: Future models that keep scaling the same way might be prohibitively expensive, especially since nobody has a moat and price wars are intensifying.
Assessment: UnresolvedSaid
“The fact that Grok 3 was 15x the size of Grok 2, and also generally viewed as disappointing.”
What we tested: Grok 3 was 15 times the size of Grok 2 and was generally viewed as disappointing.
Assessment: Can’t be checkedSaid
“Automatically generated bullshit still seems to be the norm.”
What we tested: Automatically generated bullshit still seems to be the norm, including in systems like Deep Research and Grok 3 Deep Search.
Assessment: UnresolvedSaid
“And these complex systems, built on the unreliable foundations of LLMs, are going to prove to be massively difficult to debug.”
What we tested: Complex systems built on LLMs, such as deep research agents, will prove massively difficult to debug.
Assessment: UnresolvedSaid
“AI might replace average people in some tasks soon; replacing the best humans in a few years? Not going to happen.”
What we tested: AI will not replace the best humans, such as top scientists, within a few years, though it might replace average people in some tasks soon.
Assessment: SupportedSaid
“Hallucinations haven’t solved, despite constant assurances.”
What we tested: Hallucinations have not been solved despite constant assurances that they would be.
What happened: Table 4 reports a PersonQA hallucination rate of 0.19 for GPT-4.5, compared with 0.30 for GPT-4o and 0.20 for o1. These results show improvement alongside substantial remaining failures; they do not measure all everyday usage. (OpenAI, )
Assessment: SupportedSaid
“Most parts of the US still don’t have functioning driverless cars systems, for example, even after over a decade of lavish promises.”
What we tested: Most parts of the US still lack functioning driverless car systems after more than a decade of lavish promises.
What happened: The article identifies existing Waymo service in Phoenix, San Francisco and Los Angeles, with Austin launching and other cities still planned. This supports geographically limited availability; national breadth is an inference from the deployment account. (Associated Press, syndicated by Tech Xplore, )
Assessment: Can’t be checkedSaid
“If LLMs are the wrong path to AGI, as I and a great many academics think, we may have a lot of thinking to do. We may be massively wasting resources, and getting locked into a lot foolish battles around the wrong technology.”
What we tested: If LLMs are the wrong path to AGI, as Marcus and a great many academics think, we may have a lot of thinking to do and may be massively wasting resources and getting locked into foolish battles around the wrong technology.
Assessment: Can’t be checkedSaid
“In a time when Trump is dismantling American research infrastructure, it should have been front and center.”
What we tested: Research should have been front and center in the Klein–Buchanan episode, at a time when Trump is dismantling American research infrastructure.
Assessment: Can’t be checkedSaid
“If getting to AGI in the next three years is not quite as certain as the sun rising every day from the east, we might want to hedge our bets, and try to focus on what the field of AI need to develop besides scaling.”
What we tested: If getting to AGI in the next three years is not quite certain, we might want to hedge our bets and try to focus on what the field of AI needs to develop besides scaling.
Assessment: Can’t be checkedSaid
“I couldn’t help but think that the US’s single-minded commitment to Large Language Models, predicated on implausible assumptions about AGI’s imminence, particularly combined with the untimely gutting of its scientific institutions, could ultimately lead to its undoing.”
What we tested: The US's single-minded commitment to LLMs, based on implausible assumptions about AGI's imminence and particularly when combined with the untimely gutting of its scientific institutions, could ultimately lead to the US's undoing.
Assessment: Can’t be checkedSaid
“If it is this unreliable on arithmetic, it will be this unreliable on everything else, and we shouldn’t accord it the society-changing power that people intend to accord it.”
What we tested: An AI system this unreliable on arithmetic will be similarly unreliable on everything else and should not be given society-changing power.
Assessment: Can’t be checkedSaid
“I am not convinced that anything anyone could realistically do with LLMs won’t be almost immediately copied, as has been the recent trend.”
What we tested: Marcus is not convinced that anything anyone could realistically do with LLMs won't be copied almost immediately, as has been the recent trend.
Assessment: Can’t be checkedSaid
“It actually seems to me that US may be all but surrendering the AGI race over to China.”
What we tested: With its academic research institutions being dismantled, the US may be all but surrendering the AGI race to China.
Held from this post (12)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- Attribution is uncertain
- The criteria are ambiguous. Our reading, not assessed: There is almost zero chance that AGI, meaning a system that can do almost any cognitive task a human can do, will arrive within the next two to three years.
- The record does not meet the current rules. Our reading, not assessed: The hypothesis that AGI is imminent is almost certainly wrong.
- The criteria are ambiguous. Our reading, not assessed: OpenAI's Orion appears to have been downgraded and released as GPT-4.5 rather than GPT-5.
- The two reviews reached different assessments. Our reading, not assessed: AGI timelines are the focus of some of the largest investments in history and central to many delicate policy questions.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: The policies Klein and Buchanan discuss throughout the episode, such as chip export controls and massive infrastructure investments, rest almost entirely on the assumption that AGI is nigh and that LLMs will get us there.
- The two reviews reached different assessments. Our reading, not assessed: Apple's recent study on formal reasoning very much confirms that current AI systems still lack a reliable ability to freely generalize abstractions to new instances, more than two decades after Marcus first pointed out this weakness.
- Attribution is uncertain
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: The scenario in which AGI is not imminent is underrepresented in current policy discussions.