Claims checked against evidence by Gary Marcus, page 4 of 15
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1471)
Each of these claims was checked against dated evidence.
Page 4 of 15: claims 301 to 400 of 1471, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Observation
“Meanwhile, both policies face strong bipartisan resistance.”
Our reading: Both federal preemption of state AI laws and H200 chip sales to China face strong bipartisan resistance.
No horizon Evidence as of
Limitation: Strong is qualitative; the evidence does not establish majority opposition within each party.
Assessment: UnresolvedSaid
Conditional forecast
“If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.”
Our reading: If we are to build AGI, we are going to need to learn something from humans: how they reason and understand the physical world, and how they represent and acquire language and complex concepts.
No deadline stated Evidence as of
Only if: if we are to build AGI
Source: “Scale Is All You Need” is dead
Assessment: SupportedSaid
Observation
“The machine learning community, which has unwelcomed me since I started vocally critiquing the scaling approach, has finally started to come around.”
Our reading: The machine learning community has started to accept critiques of the scaling approach.
No horizon Evidence as of
Limitation: Rests on one industry write-up plus individual statements. No measured trend.
Source: “Scale Is All You Need” is dead
Assessment: SupportedSaid
Observation
“And now, at last, the field is starting to realize it.”
Our reading: The AI field is now starting to realize that scaling alone will not reach AGI.
No horizon Evidence as of
Limitation: No earlier survey is available for comparison, so the growth of this view is inferred rather than measured.
Source: “Scale Is All You Need” is dead
Assessment: UnresolvedSaid
Observation
“Worse, they are burning billions of dollars of month.”
Our reading: OpenAI is burning billions of dollars per month.
No horizon Evidence as of
Limitation: The first-half and full-year reports do not establish a common accounting basis or isolate July–September cash burn; subtracting their figures cannot settle the claim [e1, e2].
Source: OpenAI’s “Code Red”
Assessment: UnresolvedSaid
Forecast
“In the end LLMs will continue to exist, no matter what; they will get cheaper. And they will find some utility (indeed they are already have).”
Our reading: LLMs will continue to exist, get cheaper, and have some utility.
No deadline stated Evidence as of
Limitation: The deployment and efficiency evidence is vendor-reported.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Conditional forecast
“And if Sustekver and I (and at this point many others) are right that scaling is largely played out, a decline in demand is sure to follow.”
Our reading: If scaling is largely played out, a decline in demand for AI compute is sure to follow.
No deadline stated Evidence as of
Only if: if scaling is largely played out
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“More than that, any normal seven-year old develops a rich model of the world, including the objects they see, the people they interact with and so on, and even how their bodies work. GPT is still just faking it.”
Our reading: Unlike a seven-year-old, GPT lacks a rich model of the world and is just faking it.
No horizon Evidence as of
Limitation: The behavioral reading of 'faking it' is an evaluator interpretation.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Forecast
“But they will never be AGI, and they never be much more than commodity.”
Our reading: LLMs will never be AGI and will never be much more than a commodity.
No deadline stated Evidence as of
Limitation: One benchmark cannot settle AGI.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“In some ways, GPT-5 is more capable than most 7 or 8 year olds.”
Our reading: In some ways, such as trivia and writing python code, GPT-5 is more capable than most 7- or 8-year-olds.
No horizon Evidence as of
Limitation: The available model evidence is company-reported and predates the target period.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“Ultimately though, what LLMs are great at seeming is plausible – while being much less good at getting their facts straight.”
Our reading: LLMs are good at seeming plausible but much less good at getting facts right.
No horizon Evidence as of
Limitation: The located plausibility experiment concerns prompted rationales rather than ordinary factual answers.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“The reality is OpenAI has ridden to a $500 billion valuation on hopes, not profits. And those hopes are slipping away, as usage declines and others catch up.”
Our reading: OpenAI's $500 billion valuation rests on hopes rather than profits, and those hopes are slipping as usage declines and competitors catch up.
No horizon Evidence as of
Limitation: October 2025 is the contract's selected reporting period; the claim itself specifies no usage window.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“And I certainly don’t think that ChatGPT has lived up to expectations.”
Our reading: ChatGPT has not lived up to expectations.
No horizon Evidence as of
Limitation: The capability comparison is company-reported and limited to its internal benchmark.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Causal claim
“Too much industry hype is repeated with too little skepticism, driving up market valuations on analyses that often feel to me superficial.”
Our reading: Too much industry hype is repeated with too little skepticism, driving up market valuations on analyses that often feel to Marcus superficial.
No horizon Evidence as of
Limitation: Correlation between optimistic coverage and rising prices would not establish the direction of causation.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: SupportedSaid
Observation
“And LLMs themselves have already resorted to using lots of external tools like calculators and python interpreters; it is already clear that they are more of a partial solution than the full solution people imagine.”
Our reading: LLMs have already resorted to using lots of external tools such as calculators and Python interpreters, and it is already clear that they are more of a partial solution than the full solution people imagine.
No horizon Evidence as of
Limitation: Tool adoption does not by itself prove necessity, failure, or a permanent capability ceiling.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“Three years and nearly a trillion dollars later almost of all it remains true. And it’s not just that GPT-4 was pretty much as I anticipated but that the same problems have continued to plague every single model since”
Our reading: Almost all of his December 2022 predictions about GPT-4 remain true, and the same problems plague every model since, including GPT-5, Claude, Gemini, Grok, Llama and DeepSeek.
No horizon Evidence as of
Limitation: Representative failures cannot establish a universal claim about all model versions.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“Worse, it doesn’t prove that its premise—that exponentially increasing compute—will actually bring us to artificial general intelligence. He’s conflating costs going up exponentially with benefits going up exponentially.”
Our reading: Jensen Huang's claim of exponential compute demand does not show that exponentially increasing compute will bring AGI; he conflates exponential costs with exponential benefits.
No horizon Evidence as of
Limitation: This assesses the referenced argument, not the eventual feasibility of AGI.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Conditional forecast
“If the economy goes down, ChatGPT will be at the center of the mess.”
Our reading: If the economy goes down, ChatGPT will be at the center of the mess.
No deadline stated Evidence as of
Only if: if the economy goes down
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Forecast
“I think that the recurring core technical problems that we have seen (as discussed below) with LLMs aren’t going way; instead they inherent to the technology.”
Our reading: The recurring core technical problems of LLMs are inherent to the technology and will not go away.
No deadline stated Evidence as of
Limitation: The formal and practical meanings of hallucination differ.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“But time has shown that its results have held strong.”
Our reading: The MIT finding that only 5% of companies get a return on generative AI investment has held up.
No horizon Evidence as of
Limitation: Similar difficulties scaling AI do not independently corroborate a 5% return prevalence.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: SupportedSaid
Causal claim
“The results are disappointing because the underlying tech is unreliable, And that’s been obvious from the start.”
Our reading: Generative AI results are disappointing because the underlying technology is unreliable, which has been obvious from the start.
No horizon Evidence as of
Limitation: The studies cover particular occupations and deployments.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: MixedSaid
Observation
“productivity studies sometimes show a 30% gain, but none have come close to 10x (1000% gain).”
Our reading: No productivity studies of LLMs have shown anything close to a 10x gain; some show around 30%.
No horizon Evidence as of
Limitation: The counterexample involved six fictional cases and document production, not overall physician productivity.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Forecast
“I just don’t happen to think large language models like ChatGPT will get us there.”
Our reading: Large language models like ChatGPT will not get us to AGI.
No deadline stated Evidence as of
Limitation: AGI and the architectural boundary are not precisely defined.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: SupportedSaid
Observation
“The unfairly dismissed Apple reasoning paper laid bare the generalization issues; another paper called “Is Chain-of-Thought Reasoning of LLMs Mirage? A Data Distribution Lens” put a further nail in the LLM reasoning and generalization coffin.”
Our reading: Research papers (Apple's reasoning paper and the chain-of-thought mirage paper) demonstrated LLM reasoning and generalization failures.
No horizon Evidence as of
Limitation: The result depends on reading the claim as empirical task failures rather than universal incapacity.
Assessment: UnresolvedSaid
Observation
“The first and most obvious answer is money, which I estimate, back of the envelope as (roughly) a trillion dollars, much of it on Nvidia chips and massive salaries.”
Our reading: Supposing for the sake of argument that LLMs have run their course and new ideas are needed, Marcus estimates, back of the envelope, the money cost of the field taking so long to recognize this at roughly a trillion dollars, much of it on Nvidia chips and massive salaries.
No horizon Evidence as of
Limitation: The spending boundary and starting date remain ambiguous.
Assessment: SupportedSaid
Observation
“Each new bigger, more expensive model ekes out measurable improvements, but returns appear to be diminishing (that’s what Sutskever is saying about the Kaplan laws) and none of these experiments has solved core issues around hallucinations, generalization, planning and reasoning, as Sutskever too now recognizes.”
Our reading: No LLM scaling experiment has solved the core problems of hallucination, generalization, planning and reasoning.
No horizon Evidence as of
Limitation: 'Solved' has no stated threshold. Someone who counts IMO gold or big benchmark gains as solving reasoning would read the claim differently.
Assessment: SupportedSaid
Observation
“It has never worked.”
Our reading: Building ever larger LLMs has never succeeded in reaching AGI.
No horizon Evidence as of
Limitation: The conclusion depends on the ordinary public-demonstration reading of AGI.
Assessment: SupportedSaid
Causal claim
“A big part of the culprit? Maximizing metrics for user engagement.”
Our reading: OpenAI maximizing user-engagement metrics was a big part of the cause of ChatGPT users having mental health crises.
No horizon Evidence as of
Limitation: Causation rests on investigative reporting and company statements. No clinical study or court finding in the packet establishes that engagement-tuned updates caused specific users' crises.
Source: ChatGPT and delusions: an important new inside look at OpenAI
Assessment: UnresolvedSaid
Forecast
“AGI ain’t actually coming in 2027, and even Kokotajlo, known for his aggressive timelines, is now with me in thinking we probably won’t see AGI this decade.”
Our reading: AGI will not arrive in 2027.
Not yet due Evidence as of
Limitation: The AGI criterion is an ordinary-language interpretation rather than a test Marcus specified.
Source: BREAKING: The AI 2027 doomsday scenario has officially been postponed
Assessment: UnresolvedSaid
Forecast
“AGI is NOT in fact coming in Trump’s second term, and never was.”
Our reading: AGI will not arrive during Trump's second term (ending January 2029).
Not yet due Evidence as of
Limitation: No single benchmark establishes the broad AGI criterion.
Source: BREAKING: The AI 2027 doomsday scenario has officially been postponed
Assessment: UnresolvedSaid
Observation
“Suffice for now to say that we have built (and continue to build) our entire national policy (such as it is) around an idea that was never grounded in a sound analysis of AI.”
Our reading: The US has built, and continues to build, its entire national policy (such as it is) around an idea, in context the AI 2027 picture of imminent AGI, that was never grounded in a sound analysis of AI.
No horizon Evidence as of
Limitation: The packet lacks policy-development records and the technical analyses actually relied upon during January 20–November 21, 2025.
Source: BREAKING: The AI 2027 doomsday scenario has officially been postponed
Assessment: SupportedSaid
Observation
“Google announced their big new state-of-the-art model yesterday, Gemini 3, on most measures better than any other model out there.”
Our reading: Gemini 3 is better than any other model on most measures.
No horizon Evidence as of
Limitation: Much of the comparison comes from Google's own chosen benchmarks.
Source: Hot take on Google’s Gemini 3
Assessment: ContradictedSaid
Observation
“(It’s only when ChatGPT eclipsed Meta that LeCun began to be sharply, publicly critical of LLMs)”
Our reading: LeCun began to be sharply and publicly critical of LLMs only when ChatGPT eclipsed Meta.
No horizon Evidence as of
Limitation: The decisive essay was coauthored.
Assessment: SupportedSaid
Observation
“But he neither invented them nor was he the first to apply the back-propagation algorithm to learning their weights (though many people mistakenly believe this).”
Our reading: LeCun neither invented convolutional neural networks nor was the first to train their weights with back-propagation.
No horizon Evidence as of
Limitation: This result includes temporal convolutional networks; a claim restricted to two-dimensional image CNNs would require a separate priority comparison.
Assessment: SupportedSaid
Observation
“Oracle debt downgraded.”
Our reading: Oracle's debt has been downgraded, which is an ominous sign for generative AI.
No horizon Evidence as of
Limitation: The evidence is secondary coverage of the Barclays note, not the note itself.
Assessment: SupportedSaid
Observation
“Coreweave drops 20% in a week.”
Our reading: CoreWeave's stock fell 20% in one week, which is an ominous sign for generative AI.
No horizon Evidence as of
Limitation: The weekly figure comes from a news report, not a primary price-data feed.
Assessment: SupportedSaid
Observation
“Meta financing games revealed.”
Our reading: Meta's financing games have been revealed, presented as one of five recent ominous signs for generative AI.
No horizon Evidence as of
Limitation: The transaction identification and one-month lookback are evaluator interpretations.
Assessment: SupportedSaid
Observation
“Softbank sells entire Nvidia position.”
Our reading: SoftBank has sold its entire Nvidia position, which is an ominous sign for generative AI.
No horizon Evidence as of
Limitation: The 'ominous sign' framing is not scored.
Assessment: UnresolvedSaid
Observation
“In short, Altman was—likely in conjunction with Nvidia, which also just seemed to be laying groundwork for a bailout—launching a full court press for loan guarantees when he got caught with hands in the cookie jar.”
Our reading: Altman was launching a full court press for federal loan guarantees, likely in conjunction with Nvidia, which also seemed to be laying groundwork for a bailout.
No horizon Evidence as of
Limitation: The metaphor leaves the required intensity imprecise.
Assessment: UnresolvedSaid
Observation
“But it is no longer about lying to employees. It is about directly lying to the American public.”
Our reading: Sam Altman is now directly lying to the American public, not just to employees.
No horizon Evidence as of
Limitation: The submission does not expressly request guarantees for OpenAI's own data-center borrowing [e2].
Assessment: UnresolvedSaid
Conditional forecast
“Which means you, the taxpayer, will be footing the bill.”
Our reading: If the government goes along, taxpayers will end up paying for AI companies' borrowing.
No deadline stated Evidence as of
Only if: if the government goes along with the backstop
Source: OpenAI probably can’t make ends meet. That’s where you come in.
Assessment: MixedSaid
Observation
“What Altman couldn’t say then was that he has a plan, to reduce the cost of his borrowing … by having the American taxpayer (indirectly) foot the bill.”
Our reading: OpenAI plans to reduce its borrowing costs by having American taxpayers indirectly bear the cost.
No horizon Evidence as of
Limitation: The original remarks and clarification are reproduced by news outlets; a complete primary transcript was not obtained.
Source: OpenAI probably can’t make ends meet. That’s where you come in.
Assessment: UnresolvedSaid
Observation
“Nonetheless, a very large fraction of our economy is going into the world of price wars and little moat, massive, massive investments into companies that are all basically following the same strategy of investing more and more money into one single idea, viz. scaling ever larger language models, in hopes that something magical will emerge.”
Our reading: A very large fraction of the US economy is going into massive investments in companies that are all basically following the same strategy of scaling ever larger language models, in hopes that something magical will emerge.
No horizon Evidence as of
Limitation: The broad equipment, software, R&D and data-center categories do not isolate investment in larger-language-model development.
Assessment: UnresolvedSaid
Observation
“Yet investments in the same single approach grow every day.”
Our reading: Investments in the single LLM/transformer approach continue to grow.
No horizon Evidence as of
Limitation: The evidence lacks a comparable aggregate series isolating LLM/transformer investment in Q3 2025 and earlier periods.
Assessment: UnresolvedSaid
Observation
“Modest profits was maybe a bit optimistic, with most LLM developers losing money.”
Our reading: Most LLM developers are losing money.
No horizon Evidence as of
Limitation: The selected financial interval is an evaluator interpretation.
Assessment: SupportedSaid
Observation
“GPT-5 was disappointing, and didn’t come in 2024”
Our reading: GPT-5 was disappointing and was not released in 2024.
No horizon Evidence as of
Limitation: 'Disappointing' is evaluative; reactions were mixed, not unanimous.
Assessment: SupportedSaid
Observation
“Meanwhile we have entered an era, perhaps unprecedented, of extreme financial interdependence and circular investment:”
Our reading: The AI industry has entered a perhaps unprecedented era of extreme financial interdependence and circular investment.
No horizon Evidence as of
Limitation: Extreme is a qualitative judgment without a stated threshold; the result depends on the ordinary reading specified in the contract.
Assessment: SupportedSaid
Observation
“no robust solution to hallucinations”
Our reading: There is no robust solution to LLM hallucinations.
No horizon Evidence as of
Limitation: 'Robust' has no threshold; partial mitigations exist and rates are falling.
Assessment: UnresolvedSaid
Conditional forecast
“If and when the whole generative AI edifice comes crashing down, it will be in no small part because too many otherwise smart investors read too many graphs—from “LLM scaling” to usage statistics—too naively, assuming that things had been going up for a while would continue to go up at the same pace, indefinitely.”
Our reading: If generative AI collapses, a large part of the cause will be investors naively extrapolating scaling and usage trends.
No deadline stated Evidence as of
Only if: if and when the generative AI edifice comes crashing down
Assessment: UnresolvedSaid
Causal claim
“The problem, as always, lies in generalizing outside the training distribution.”
Our reading: The problem with vibe coding lies, as always, in generalizing outside the training distribution.
No horizon Evidence as of
Limitation: Qualitative reports do not measure causal effects.
Source: Is vibe coding dying?
Assessment: UnresolvedSaid
Forecast
“Remember how in October and in March I told you that vibe coding — in the sense of amateurs using large language models to write code to “build products that would have previously required teams of engineers” — would never be remotely reliable?”
Our reading: Vibe coding by amateurs using LLMs to build products that previously required engineering teams will never be remotely reliable.
No deadline stated Evidence as of
Limitation: Most of the evidence is anecdotal or comes from a vendor's CEO.
Source: Is vibe coding dying?
Assessment: SupportedSaid
Observation
“All that really happened is that GPT-5 (oversimplifying slightly) crawled the web for solutions to already-solved problems.”
Our reading: GPT-5 did not solve open Erdős problems; it essentially located existing solutions to already-solved problems on the web.
No horizon Evidence as of
Limitation: The result depends on treating the crawler wording as the acknowledged simplification.
Source: Erdosgate
Assessment: UnresolvedSaid
Observation
“I found that the claim of “solution” was pretty misleading, as I summed up in a tweet, and they refused to correct their misleading presentation:”
Our reading: In 2019 OpenAI's claim that its robot 'solved' the Rubik's cube was misleading, and OpenAI refused to correct it.
No horizon Evidence as of
Limitation: Disagreement over wording does not by itself prove a factual misrepresentation.
Source: Erdosgate
Assessment: UnresolvedSaid
Observation
“June, 2025: the Apple reasoning paper confirmed that even with “reasoning”, LLMs still can’t solve distribution shift, the core Achille’s heel in neural networks that I have been writing about for nearly 30 years.”
Our reading: Even with reasoning, LLMs still cannot handle distribution shift, as the June 2025 Apple paper confirmed.
No horizon Evidence as of
Limitation: The competing papers are evidence of a scientific dispute, not equally decisive demonstrations.
Source: Game over. AGI is not imminent, and LLMs are not the royal road to getting there.
Assessment: UnresolvedSaid
Observation
“We aren’t close to AGI, but we are reshaping our world on the premise that we are.”
Our reading: We are not close to AGI.
No horizon Evidence as of
Limitation: Benchmark failures do not measure time remaining to AGI.
Source: The Grand AGI Delusion
Assessment: UnresolvedSaid
Observation
“And schoolchildren using ChatGPT to do their homework aren’t enough to cover that.”
Our reading: Schoolchildren using ChatGPT to do their homework are not enough to cover OpenAI's immense costs and cash burn.
No horizon Evidence as of
Limitation: The reporting window and meaning of 'cover that' are not explicit.
Source: OpenAI’s future, foretold?
Assessment: SupportedSaid
Observation
“GenAI certainly sounds amazing – but it hasn’t always worked well enough to justify the costs.”
Our reading: Generative AI has not always worked well enough to justify its costs.
No horizon Evidence as of
Limitation: Support rests on preliminary, largely self-reported and anonymized deployment research.
Source: OpenAI’s future, foretold?
Assessment: SupportedSaid
Observation
“The notion on which this Substack was founded in May 2022 — that LLM scaling would not bring us to AGI — has gone mainstream.”
Our reading: The view that LLM scaling will not bring AGI has become mainstream.
No horizon Evidence as of
Limitation: The survey was academically weighted and was not a representative industry poll.
Assessment: UnresolvedSaid
Forecast
“that LLM scaling would not bring us to AGI”
Our reading: Scaling LLMs will not produce AGI.
No deadline stated Evidence as of
Limitation: There is no universally accepted AGI test.
Assessment: SupportedSaid
Observation
“The concerns about potential copyright violations that Reid Southen and I expressed here in and at greater length in the IEEE Spectrum in January 2024 continue almost two years later to be a problem.”
Our reading: Copyright violation concerns with generative AI remain a problem as of September 2025.
No horizon Evidence as of
Limitation: A lawsuit establishes a live allegation, not that the defendant infringed.
Assessment: UnresolvedSaid
Forecast
“Perhaps I also no longer need to warn people that the idea that coding would disappear soon is bullshit.”
Our reading: Human coding will not disappear soon due to AI.
No deadline stated Evidence as of
Limitation: This leaves the forecast open because of its timing, not because the evidence shows human coding disappeared.
Assessment: SupportedSaid
Observation
“No model could have done this in 2022. The graphics are vastly better than they used to be.”
Our reading: Image generation graphics quality has improved vastly since 2022, to a level no 2022 model could reach.
No horizon Evidence as of
Limitation: The evidence does not show that no 2022 model could produce a sharp, mechanically correct image of a rider. The 2022 test found some good outputs.
Source: Spinal Tap meets John Malkovich
Assessment: UnresolvedSaid
Observation
“Watch the tuners come and go! So much for object permanence.”
Our reading: Grok's generated video failed to keep objects consistent, with the bass tuners appearing and disappearing.
No horizon Evidence as of
Limitation: No evidence items show what the video contains.
Source: Spinal Tap meets John Malkovich
Assessment: UnresolvedSaid
Observation
“Over the last few years, graphics have gotten better and better. World models and comprehension less so.”
Our reading: Over the last few years, graphics in image generation systems have gotten better and better, while world models and comprehension have improved less.
No horizon Evidence as of
Limitation: The starting year and operational meanings of the capabilities are evaluator interpretations.
Source: What scaling does and doesn’t buy you: peeling back the hype surrounding Google‘s trendy Nano Banana
Assessment: UnresolvedSaid
Observation
“Suffice to say that I was able to break nano banana literally on my first try:”
Our reading: Marcus got Nano Banana to fail on his very first attempt.
No horizon Evidence as of
Limitation: The missing record does not show that Marcus's account was false.
Source: What scaling does and doesn’t buy you: peeling back the hype surrounding Google‘s trendy Nano Banana
Assessment: UnresolvedSaid
Observation
“And guess what, surprise, surprise, even the latest models still struggle with compositionality”
Our reading: Even the latest image generation models, Nano Banana among them, still struggle with compositionality.
No horizon Evidence as of
Limitation: The independent test is small and selected, and its outputs were not independently rescored.
Source: What scaling does and doesn’t buy you: peeling back the hype surrounding Google‘s trendy Nano Banana
Assessment: UnresolvedSaid
Observation
“For all the hype, Nano Banana is not all that much better at this sort of thing than ChatGPT-5:”
Our reading: Despite the hype, Nano Banana is not much better than ChatGPT-5 at drawing part-whole structures such as bicycles.
No horizon Evidence as of
Limitation: Selected failures from both systems would not establish comparable failure rates.
Source: What scaling does and doesn’t buy you: peeling back the hype surrounding Google‘s trendy Nano Banana
Assessment: UnresolvedSaid
Observation
“ChatGPT, by the way, also continues to struggle with the basic facts of life:”
Our reading: ChatGPT still struggles with basic facts of life in image generation.
No horizon Evidence as of
Limitation: The admitted evidence contains no tests from August 1–27, 2025.
Source: What scaling does and doesn’t buy you: peeling back the hype surrounding Google‘s trendy Nano Banana
Assessment: UnresolvedSaid
Causal claim
“Agency plus LLMs have already led multiple “vibe coders” to lose databases, with new reports coming in every day or two, stemming simply from the unreliability inherent in LLMs (that Gary has so often harped on).”
Our reading: Agency combined with LLMs has already led multiple vibe coders to lose databases, with new reports every day or two, and these losses stem simply from the unreliability built into LLMs.
No horizon Evidence as of
Limitation: The Cursor incident rests on a user allegation without execution logs or an independent postmortem [e2].
Assessment: UnresolvedSaid
Forecast
“The bad news is that although this one was stopped, many others won’t be. No one patch would be enough, or even a thousand; there are just too many variations on the overall theme”
Our reading: Many attacks on AI coding tools won't be stopped; not even a thousand patches would suffice given the number of variations.
No deadline stated Evidence as of
Limitation: The evidence concerns researcher demonstrations rather than a census of malicious attacks.
Assessment: MixedSaid
Observation
“Some customers even wanted the old models back. Almost nobody felt satisfied.”
Our reading: Almost no one was satisfied with GPT-5, and some customers wanted the old models back.
No horizon Evidence as of
Limitation: No representative satisfaction survey of GPT-5 users from the launch week was found, so the share of satisfied users is not known.
Assessment: UnresolvedSaid
Forecast
“But between the disappointments of GPT-5 and a new study from METR that shows that LLMs do markedly better on coding benchmarks than in real-word practice, I think it is safe to say that LLMs won’t lead the way.”
Our reading: LLMs will not be the approach that leads the way to AGI.
No deadline stated Evidence as of
Limitation: The first accepted AGI is an interpretation of 'lead the way', not a criterion Marcus explicitly supplied.
Assessment: SupportedSaid
Observation
“But between the disappointments of GPT-5 and a new study from METR that shows that LLMs do markedly better on coding benchmarks than in real-word practice, I think it is safe to say that LLMs won’t lead the way.”
Our reading: LLMs do markedly better on coding benchmarks than in real-world coding practice.
No horizon Evidence as of
Limitation: The study does not establish the same mismatch across all developers, tasks or LLMs; it allows for substantial productivity gains in other coding settings [e2].
Assessment: UnresolvedSaid
Causal claim
“Critically, as I argued at the end of June (and going back to 2019) LLMs never induce proper world models, which is why, for example, they still can’t even play chess reliably,1 and continue to make stupid, head-scratching errors with startling regularity.”
Our reading: LLMs never induce proper world models, which causes them to be unable to play chess reliably and to make frequent stupid errors.
No horizon Evidence as of
Limitation: The strongest intervention evidence concerns a specialist chess model.
Assessment: UnresolvedSaid
Causal claim
“ChatGPT gets them wrong because it is munging words together, not genuinely processing an (elementary) model of British PM history.”
Our reading: ChatGPT gets the British prime minister tasks wrong because it is munging words together rather than genuinely processing an elementary model of British PM history.
No horizon Evidence as of
Limitation: Evidence from Llama-2 cannot directly support or contradict a claim about GPT-5's internal processing.
Assessment: UnresolvedSaid
Observation
“They don’t induce abstract models of the world that they can reliably engage, even when those models are given to them on a silver platter (as with the rules of chess).”
Our reading: LLMs do not induce abstract world models they can reliably use, even when given the rules explicitly.
No horizon Evidence as of
Limitation: RuleArena tests practical regulations rather than chess.
Assessment: UnresolvedSaid
Observation
“Even with thinking mode activated, the output was filled with errors that could easily have been avoided, with a 5-second Wikipedia search that any sensible human could have performed.”
Our reading: GPT-5 in thinking mode produced a list of Canadian Prime Ministers full of easily avoidable errors.
No horizon Evidence as of
Limitation: The retrieved plain-text version of the post does not supply independently verifiable image provenance.
Assessment: ContradictedSaid
Observation
“They do not sanity check their own work.”
Our reading: LLMs do not sanity check their own work.
No horizon Evidence as of
Limitation: SCoRe used specialized training; its results do not establish that every model checks every answer [e1].
Assessment: MixedSaid
Observation
“Cherry-picked studies, benchmark-gaming, and now even vibe-coded graphs, with zero transparency about how systems work or how they have been trained; public science is in the rear view mirror.”
Our reading: AI has been marked by cherry-picked studies, benchmark-gaming and now even vibe-coded graphs, with zero transparency about how systems work or how they have been trained, and public science is in the rear view mirror.
No horizon Evidence as of
Limitation: The judgment depends on reading zero transparency as no substantive disclosure, rather than insufficient disclosure for reproduction.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“Allegedly godlike models that turn out to be incremental advances.”
Our reading: Models hyped as godlike turn out to be incremental advances.
No horizon Evidence as of
Limitation: A smaller aggregate benchmark gain can coexist with substantial gains in particular uses.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“That’s exactly what it means to hit a wall, and exactly the particular set of obstacles I described in my most notorious (and prescient) paper, in 2022. Real progress on some dimensions, but stuck in place on others.”
Our reading: LLM development has hit a wall: real progress on some dimensions but stuck on others.
No horizon Evidence as of
Limitation: The outcome depends on reading stuck as persistence of qualitative failures, as the surrounding passage indicates.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“The real news is a breaking study from Arizona State University that fully vindicates what I have told you for nearly 30 years—and more recently what Apple told you—about the core weakness of LLMs: their inability to generalize broadly.”
Our reading: LLMs' core weakness is their inability to generalize broadly, as confirmed by an ASU study.
No horizon Evidence as of
Limitation: This assesses the August 5 version available when Marcus wrote.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Causal claim
“Because expectations were through the roof, a huge number of people viewed GPT 5 as a major letdown. By the end of the night, OpenAI’s street cred had dramatically fallen.”
Our reading: Sky-high expectations caused many to see GPT-5 as a major letdown, and OpenAI's credibility fell dramatically on launch night.
No horizon Evidence as of
Limitation: The market movement is documented through news reporting rather than verified original records.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Forecast
“Pure scaling simply isn’t the path to AGI.”
Our reading: Pure scaling simply isn't the path to AGI and will not get us there.
No deadline stated Evidence as of
Limitation: Benchmark achievement is narrower than general intelligence.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“Driverless cars that still are only available in couple percent of the world’s cities.”
Our reading: Driverless cars are available in only a couple percent of the world's cities.
No horizon Evidence as of
Limitation: This does not establish that worldwide availability exceeded his estimate.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“Many of their best people have left.”
Our reading: Many of OpenAI's best people have left the company.
No horizon Evidence as of
Limitation: Documented prominence does not establish an objective ranking of OpenAI's best employees.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“What the mainstream media mostly hasn’t told you yet is that a few days later, hardly anybody is buying Altman’s story.”
Our reading: A few days after launch, hardly anybody believes Altman's claims that GPT-5 is like a PhD-level expert.
No horizon Evidence as of
Limitation: The evidence is news reports of individual reactions, not representative data.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“Promises for world-changing science that rarely materialize.”
Our reading: Promises that AI will produce world-changing science rarely materialize.
No horizon Evidence as of
Limitation: The searches were not an exhaustive census of AI science promises.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“People had grown to expect miracles, but GPT-5 is just the latest incremental advance.”
Our reading: GPT-5 is merely an incremental advance, not a radical leap.
No horizon Evidence as of
Limitation: The result depends on reading the comparison as against the immediately preceding models.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“Even some of the tech bros are waking up to the reality that “AGI in 2027” was marketing, not reality.”
Our reading: Even some tech bros are waking up to the reality that 'AGI in 2027' was marketing, not reality.
No horizon Evidence as of
Limitation: The observation concerns public attitudes by August 2025, not whether AGI ultimately arrives in 2027.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“The challenge of parts and wholes in generative images that Ernest Davis and I discussed here in December fared no better.”
Our reading: GPT-5 shows no improvement on the parts-and-wholes problem in generated images.
No horizon Evidence as of
Limitation: The only systematic test predates GPT-5 and used ChatGPT-4o.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“Dozens of shots on goal (Grok, Claude, Gemini) etc have invariably faced the same problems. Distribution shift has never been solved.”
Our reading: Dozens of shots on goal, such as Grok, Claude and Gemini, have invariably faced the same problems, and distribution shift has never been solved.
No horizon Evidence as of
Limitation: Support for recurring weaknesses is stronger than support for the universal wording.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“Elon is moving faster. Anthropic and Google and many others are nipping at their heels. Their relationship with Microsoft has frayed.”
Our reading: Elon Musk is moving faster than OpenAI, Anthropic and Google and many others are nipping at OpenAI's heels, and OpenAI's relationship with Microsoft has frayed.
No horizon Evidence as of
Limitation: Release cadence, benchmark rank and rate of capability improvement are different measures.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“And, indeed, much as I anticipated here two weeks ago, the problems I have been pointing out over the last quarter century still lingered.”
Our reading: GPT-5 still has the longstanding problems Marcus has identified, such as chess rule-following and world models.
No horizon Evidence as of
Limitation: The main evidence is a single demonstration, not a benchmark.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: ContradictedSaid
Observation
“The ASU team showed that exactly the same thing was true even in the latest, greatest models. Throw in every gadget invented since 1998, and the Achilles’ Heel I identified then still remains.”
Our reading: The ASU study showed that even the latest, greatest models cannot reliably extend universals outside their training space of examples, so the Achilles' heel Marcus identified in 1998 still remains despite every gadget invented since.
No horizon Evidence as of
Limitation: This does not establish that frontier models generalize reliably.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“It’s not like other systems are faring much better.”
Our reading: Other generative AI systems such as Grok are not faring much better than GPT-5.
No horizon Evidence as of
Limitation: The packet contains no verified primary comparison from the August 7–9 launch window.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“What was up with multiple graphs (at least two) in which the bars didn’t match the numbers?”
Our reading: In the GPT-5 announcement, OpenAI showed at least two graphs whose bars did not match the numbers.
No horizon Evidence as of
Limitation: The assessment relies on contemporaneous visual analysis and subsequent reporting because the primary video was not readable.
Source: GPT-5 hot take
Assessment: UnresolvedSaid
Observation
“Pricing is good, but profits may continue to be elusive; still no clear technical moat.”
Our reading: OpenAI still has no clear technical moat with GPT-5.
No horizon Evidence as of
Limitation: Competitor disclosures are self-reported.
Source: GPT-5 hot take
Assessment: SupportedSaid
Observation
“GPT-5 is obviously not AGI.”
Our reading: GPT-5 is not AGI.
No horizon Evidence as of
Limitation: The outcome depends on the stated interpretation of AGI.
Source: GPT-5 hot take
Assessment: SupportedSaid
Observation
“OpenAI conveniently forgot to include this comparison (ARC-AGI-2) in their livestream recital of benchmark progress, which left the livestream looking like marketing rather than science.”
Our reading: OpenAI left the ARC-AGI-2 comparison out of its GPT-5 livestream benchmarks, which made the livestream look like marketing rather than science.
No horizon Evidence as of
Limitation: The decisive source is contemporaneous specialist reporting rather than an independently inspected complete recording.
Source: GPT-5 hot take