Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
We recorded 31 claims from this post. 19 were checked against evidence. 12 can’t be checked. 16 more are held. Read the post at its source.
Claims from this post (31)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“But I think a fair case can be made that it is not what it has often been cracked up to be, and probably never will be.”
What we tested: Marcus thinks a fair case can be made that ChatGPT is not what it has often been cracked up to be, and that it probably never will be.
Assessment: UnresolvedSaid
“I just don’t happen to think large language models like ChatGPT will get us there.”
What we tested: Large language models like ChatGPT will not get us to AGI.
Assessment: UnresolvedSaid
“I think that the recurring core technical problems that we have seen (as discussed below) with LLMs aren’t going way; instead they inherent to the technology.”
What we tested: The recurring core technical problems of LLMs are inherent to the technology and will not go away.
Assessment: UnresolvedSaid
“And I certainly don’t think that ChatGPT has lived up to expectations.”
What we tested: ChatGPT has not lived up to expectations.
Assessment: MixedSaid
“productivity studies sometimes show a 30% gain, but none have come close to 10x (1000% gain).”
What we tested: No productivity studies of LLMs have shown anything close to a 10x gain; some show around 30%.
What held: The published review supports the around-30% portion. An average cannot establish that no individual study found a much larger gain. (Tom Coupé and Allen Wu, University of Canterbury, )
What did not: This published pilot compared document generation for six fictional orthopedic cases and assessed quality using a blinded expert panel. It contradicts the unrestricted assertion about studies, but does not demonstrate tenfold productivity across physicians' entire jobs. (Acta Orthopaedica, )
Assessment: Can’t be checkedSaid
“Return on investment for corporate end users is also lower than many people must have imagined.”
What we tested: Return on generative AI investment for corporate end users is lower than many people must have imagined.
Assessment: UnresolvedSaid
“But time has shown that its results have held strong.”
What we tested: The MIT finding that only 5% of companies get a return on generative AI investment has held up.
Assessment: Can’t be checkedSaid
“And worse, the economy itself has become so wrapped up in generative AI and its promises, that the economy itself is, by many accounts in serious jeopardy.”
What we tested: The economy has become so wrapped up in generative AI and its promises that, by many accounts, it is in serious jeopardy.
Assessment: UnresolvedSaid
“If the economy goes down, ChatGPT will be at the center of the mess.”
What we tested: If the economy goes down, ChatGPT will be at the center of the mess.
Assessment: SupportedSaid
“The results are disappointing because the underlying tech is unreliable, And that’s been obvious from the start.”
What we tested: Generative AI results are disappointing because the underlying technology is unreliable, which has been obvious from the start.
What happened: OpenAI acknowledged unreliable answers at launch. This supports the claim that the problem was publicly knowable from the beginning. (OpenAI, )
Assessment: UnresolvedSaid
“Three years and nearly a trillion dollars later almost of all it remains true. And it’s not just that GPT-4 was pretty much as I anticipated but that the same problems have continued to plague every single model since”
What we tested: Almost all of his December 2022 predictions about GPT-4 remain true, and the same problems plague every model since, including GPT-5, Claude, Gemini, Grok, Llama and DeepSeek.
Assessment: SupportedSaid
“And LLMs themselves have already resorted to using lots of external tools like calculators and python interpreters; it is already clear that they are more of a partial solution than the full solution people imagine.”
What we tested: LLMs have already resorted to using lots of external tools such as calculators and Python interpreters, and it is already clear that they are more of a partial solution than the full solution people imagine.
What happened: ChatGPT's interpreter performed computation in an external execution environment. This directly supports tool integration and distinguishes executing Python from merely generating Python text. (OpenAI, )
Assessment: UnresolvedSaid
“And because of the discrepancy between reliability and the immense costs that stem from the inherent inefficiency of a system that it dependent on absorbing internet-scale data, it has thus far failed to make profits for companies like OpenAI.”
What we tested: Because of unreliability and immense costs from inefficiency, ChatGPT has failed so far to make profits for companies like OpenAI.
Assessment: Can’t be checkedSaid
“If things do go belly-up, and the whole economy falls into recession, the single biggest culprit in my mind, will be ChatGPT’s human avatar, its bullshit-spewing CEO Sam Altman, who hyped GPT-5 endlessly for years, pretending he knew that AGI was coming when in hindsight he was bluffing.”
What we tested: If things go belly-up and the whole economy falls into recession, the single biggest culprit in Marcus's mind will be Sam Altman, whom he describes as having hyped GPT-5 for years while bluffing about knowing that AGI was coming.
Assessment: UnresolvedSaid
“Worse, it doesn’t prove that its premise—that exponentially increasing compute—will actually bring us to artificial general intelligence. He’s conflating costs going up exponentially with benefits going up exponentially.”
What we tested: Jensen Huang's claim of exponential compute demand does not show that exponentially increasing compute will bring AGI; he conflates exponential costs with exponential benefits.
Assessment: UnresolvedSaid
“And if Sustekver and I (and at this point many others) are right that scaling is largely played out, a decline in demand is sure to follow.”
What we tested: If scaling is largely played out, a decline in demand for AI compute is sure to follow.
Assessment: Can’t be checkedSaid
“GenAI is like a rerun of that situation, an order or two of magnitude more hype, coupled with equal naïveté about what a cognitively adequate solution would actually look like, only this time with an extra decimal place, potentially a trillion dollars down the drain — and perhaps with far more collateral damage to the economy.”
What we tested: Generative AI is like a rerun of the driverless-car hype, with an order or two of magnitude more hype and equal naïveté about what a cognitively adequate solution would look like, potentially with a trillion dollars down the drain and perhaps far more collateral damage to the economy.
Assessment: Can’t be checkedSaid
“Anyone assuming that everything is simultaneously rising exponentially, and that it will continue to do so indefinitely, is a fool.”
What we tested: Anyone assuming that everything is simultaneously rising exponentially and will continue to do so indefinitely is a fool.
Assessment: UnresolvedSaid
“Too much industry hype is repeated with too little skepticism, driving up market valuations on analyses that often feel to me superficial.”
What we tested: Too much industry hype is repeated with too little skepticism, driving up market valuations on analyses that often feel to Marcus superficial.
Assessment: Can’t be checkedSaid
“The market is maybe finally catching on.”
What we tested: The market may finally be recognizing AI's limits, as shown by falls in Nvidia, CoreWeave and Oracle stock.
Assessment: Can’t be checkedSaid
“OpenAI itself isn’t traded on the open market, but if it had, it’s likely that it, too, would have fallen over the last month, for all the reasons I described, and one more: they don’t have moat.”
What we tested: OpenAI has no moat, and its value would likely have fallen over the last month if it were publicly traded.
Assessment: UnresolvedSaid
“The reality is OpenAI has ridden to a $500 billion valuation on hopes, not profits. And those hopes are slipping away, as usage declines and others catch up.”
What we tested: OpenAI's $500 billion valuation rests on hopes rather than profits, and those hopes are slipping as usage declines and competitors catch up.
Assessment: UnresolvedSaid
“In the end LLMs will continue to exist, no matter what; they will get cheaper. And they will find some utility (indeed they are already have).”
What we tested: LLMs will continue to exist, get cheaper, and have some utility.
Assessment: UnresolvedSaid
“But they will never be AGI, and they never be much more than commodity.”
What we tested: LLMs will never be AGI and will never be much more than a commodity.
Assessment: Can’t be checkedSaid
“In many domains. answers might be like 80% correct and 20% absurd, but without a whole lot of task-specific engineering required.”
What we tested: In many domains ChatGPT's answers may be roughly 80% correct and 20% absurd, with little task-specific engineering.
Assessment: UnresolvedSaid
“Ultimately though, what LLMs are great at seeming is plausible – while being much less good at getting their facts straight.”
What we tested: LLMs are good at seeming plausible but much less good at getting facts right.
Assessment: UnresolvedSaid
“In some ways, GPT-5 is more capable than most 7 or 8 year olds.”
What we tested: In some ways, such as trivia and writing python code, GPT-5 is more capable than most 7- or 8-year-olds.
Assessment: UnresolvedSaid
“More than that, any normal seven-year old develops a rich model of the world, including the objects they see, the people they interact with and so on, and even how their bodies work. GPT is still just faking it.”
What we tested: Unlike a seven-year-old, GPT lacks a rich model of the world and is just faking it.
Assessment: Can’t be checkedSaid
“People are now very actively trying to put variations of large language models into humanoid robots, but I can’t imagine any could do what my daughter was able to do then, let alone what she can do nowadays as a curious, precocious eleven-year-old.”
What we tested: People are very actively trying to put LLM variants into humanoid robots, but Marcus cannot imagine any of them could do what his daughter did at three (inventing a new way to climb out of her chair), let alone what she can do now at eleven.
Assessment: Can’t be checkedSaid
“For the love of Darwin, please let’s spend the next eight years considering new approaches, not tanking the economy on more of the same.”
What we tested: The AI field should spend the next eight years on new approaches rather than risking the economy on more LLM scaling.
Assessment: Can’t be checkedSaid
“ChatGPT has had a good (though not great) run; it’s time for something new.”
What we tested: ChatGPT has had a good but not great run, and the field should move to something new.
Held from this post (16)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: GPT-5 has not solved the core LLM problems, as many people imagined it would; had it done so, it would be of enormous economic value.
- The two reviews reached different assessments. Our reading, not assessed: Census Bureau data already shows signs of generative AI demand tapering off, which is not consistent with a genuine exponential.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: Google, with Gemini 3, appears to have overtaken OpenAI in the last month, beating GPT-5 on many benchmarks.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: LLMs have become a commodity, profits outside Nvidia are scarce, and OpenAI is losing billions of dollars a month.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: So far as Marcus knows, even the latest language models are still powerful but hard to control, still can't reason reliably, still don't work reliably with external tools, continue to hallucinate, still can't match domain-specific models, and continue to struggle with alignment.
- The record does not meet the current rules. Our reading, not assessed: Across hundreds of LLMs, the same failure modes always appear, a pattern Marcus calls astonishingly clear.
- The two reviews reached different assessments. Our reading, not assessed: ChatGPT obviously lacks the reliability of calculators, and its hallucinations and occasional bizarre errors should be shocking given that, unlike seven-year-olds, LLMs have swallowed the entire internet.
- The record does not meet the current rules. Our reading, not assessed: GPT-5 is more impressive than the original ChatGPT (GPT-3.5), the latest models have more utility, and LLMs have certainly come a long way.
- The criteria are ambiguous. Our reading, not assessed: Amateurs can build prototypes with AI coding, but the generated code tends to be shaky and few can make it robust.
- The two reviews reached different assessments. Our reading, not assessed: Because you never know what ChatGPT will produce, you can never really trust it.
- The two reviews reached different assessments. Our reading, not assessed: Scaling has not come close to solving the reliability, world-understanding and hallucination problems of ChatGPT.