Unresolved claims by Gary Marcus, page 3 of 8
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Unresolved claims by Gary Marcus (772)
The record was checked, and the evidence available by the cutoff does not settle it. Forecasts whose deadline has not arrived are listed apart, as not yet due.
Page 3 of 8: claims 201 to 300 of 772, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Observation
“That sentiment that has only grown over the last several months.”
Our reading: The sentiment that AI agents' reality doesn't match the hype has grown from March to August 2025.
No horizon Evidence as of
Limitation: Trust and disappointment with hype are different measures.
Assessment: UnresolvedSaid
Observation
“But, none, to my knowledge are reliable, except maybe in very narrow uses cases.”
Our reading: To his knowledge, none of the big companies' AI agents are reliable, except maybe in very narrow use cases.
No horizon Evidence as of
Limitation: The July reports do not provide representative, repeated trials or systematic rates of consequential errors and human intervention [e2, e3].
Assessment: UnresolvedSaid
Forecast
“For that, we will, I predict, need genuinely new architectures that incorporate explicit knowledge and world models at their very core.”
Our reading: Trustworthy AI will require genuinely new architectures with explicit knowledge and world models at their core.
No deadline stated Evidence as of
Limitation: Behavior on a bounded benchmark cannot establish broad trustworthiness.
Assessment: UnresolvedSaid
Forecast
“GPT-5 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
Our reading: GPT-5, like its predecessors, will be reckless and hard to control.
No deadline stated Evidence as of
Limitation: METR's task suite does not measure the prevalence or consequences of unintended behavior across ordinary deployments, and its detection methods may miss reward hacking [e1].
Assessment: UnresolvedSaid
Forecast
“Without external aids it won’t be able beat Meta’s Cicero in Diplomacy; it won’t be able to drive a car reliably; it won’t be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.”
Our reading: Without external aids, GPT-5 will not be able to beat Meta's Cicero at Diplomacy, will not be able to drive a car reliably, and will not be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.
No deadline stated Evidence as of
Limitation: This compound claim cannot be supported from evidence about robotics alone.
Assessment: UnresolvedSaid
Forecast
“Its natural language output still won’t be something that one can reliably hook up to downstream programs; it won’t be something, for example, that you can simply and directly hook up to a database or virtual assistant, with predictable results.”
Our reading: GPT-5's natural language output will not be something one can reliably hook up to downstream programs, for example simply and directly to a database or virtual assistant, with predictable results.
No deadline stated Evidence as of
Limitation: Distinguishing natural language from plaintext code materially affects the assessment.
Assessment: UnresolvedSaid
Conditional forecast
“Trustworthy, general artificial intelligence, aligned with human values, will come, when it does, from systems that are more structured, with more built-in knowledge, and will incorporate at least some degree of explicit tools for reasoning and planning, as well as explicit knowledge, that are lacking in systems like GPT.”
Our reading: When trustworthy general AI aligned with human values comes, it will come from systems that are more structured, have more built-in knowledge, and incorporate at least some degree of explicit tools for reasoning and planning, as well as explicit knowledge, that systems like GPT lack.
No deadline stated Evidence as of
Only if: when it does come
Assessment: UnresolvedSaid
Forecast
“GPT-5 will not have reliable models of the things that it talks about that are accessible to external programmers in a way that reliably feeds downstream processes.”
Our reading: GPT-5 will not have reliable models of the things it talks about that external programmers can access in a way that reliably feeds downstream processes.
No deadline stated Evidence as of
Limitation: The meaning of an accessible subject model materially affects what evidence would count.
Assessment: UnresolvedSaid
Observation
“I was right on the money then.”
Our reading: Marcus's December 2022 predictions about GPT-4 were accurate.
No horizon Evidence as of
Limitation: The assessment reads the retrospective as referring to all seven predictions; its intended scope is not explicit.
Assessment: UnresolvedSaid
Forecast
““Scaling” alone—building bigger and models until they absorb the entire internet — will prove useful, but only to a point.”
Our reading: Scaling alone will prove useful only up to a point.
No deadline stated Evidence as of
Limitation: Finite experiments cannot establish what all future increases in scale will achieve.
Assessment: UnresolvedSaid
Observation
“Neither DeepMind nor OpenAI has given any substantial information about how their systems are designed and trained.”
Our reading: Neither DeepMind nor OpenAI has given substantial information on how their IMO systems are designed and trained.
No horizon Evidence as of
Limitation: Substantial is a judgment threshold, not a specified quantity.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Forecast
“Eric Schmidt’s recent prediction (made before IMO 2025), that there will be AI-based “world-class mathematicians” within a year, is pure fantasy.”
Our reading: There will not be AI-based world-class mathematicians within a year (by about July 2026).
Deadline passed Evidence as of
Limitation: The evidence supplies no agreed standard connecting one exceptional result to world-class research capability.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Forecast
“Tying for 27th place in a highly elite high school mathematics competition will not immediately change that.”
Our reading: Tying for 27th place at the IMO will not immediately change the fact that only one theorem of middling importance (the Robbins conjecture, in 1996) has ever been proved autonomously by an AI system.
No deadline stated Evidence as of
Limitation: “Immediately” has no operational endpoint; open_horizon does not extend it indefinitely.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Observation
“The system is so bound to Musk it even asked his opinion on pizza.”
Our reading: Grok is so bound to Musk that it sometimes searches for Musk's opinion before answering, and in one case even asked his opinion on pizza.
No horizon Evidence as of
Limitation: The available demonstration establishes occurrence, not how often the behavior happened or its underlying cause.
Assessment: UnresolvedSaid
Observation
“From this I infer that xAI’s main methodology for handling alignment is trial-and-error.”
Our reading: From xAI's account of the incident, Marcus infers that xAI's main methodology for handling alignment is trial-and-error.
No horizon Evidence as of
Limitation: Public disclosures may omit important internal practices.
Assessment: UnresolvedSaid
Observation
“My p(doom) is still lower than most industry people’s.”
Our reading: Marcus's p(doom) is lower than that of most AI industry people.
No horizon Evidence as of
Limitation: A researcher sample is not automatically representative of the AI industry.
Assessment: UnresolvedSaid
Observation
“In contrast, o3, when it draws on symbolic code (which it sometimes displays explicitly) can build crossword grids like this far faster than I could (short of my writing my own code).”
Our reading: When o3 draws on symbolic code, it can build crossword grids like this one far faster than Marcus could, short of writing his own code.
No horizon Evidence as of
Limitation: The informal experiment may have private records that are not publicly available.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“OpenAI quietly began doing this to some degree in 2023 with a system called “code interpreter”, in which LLMs (themselves neural networks) call (purely symbolic) Python interpreters. This is literally “putting programs inside”; exactly what Hinton said was a huge mistake.”
Our reading: In 2023, OpenAI quietly began, to some degree, 'putting programs inside' its systems with Code Interpreter, in which LLMs call purely symbolic Python interpreters. Marcus says this is exactly what Hinton called a huge mistake.
No horizon Evidence as of
Limitation: The launch and functionality component is supported.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“I argued that scaling as it was then defined (pretraining data and compute) alone was insufficient to solve challenges with reasoning, misalignment and hallucinations. (Spoiler alert: it still hasn’t.)”
Our reading: Scaling pretraining data and compute alone has not solved reasoning, misalignment and hallucination problems.
No horizon Evidence as of
Limitation: This is an evidence limitation, not evidence that pretraining alone had solved the problems.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“Yet nearly all funding has been aimed simply at making them larger.”
Our reading: Nearly all AI funding has gone to making LLMs larger.
No horizon Evidence as of
Limitation: The selected annual period cannot settle an unspecified cumulative historical claim.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“OpenAI, without any sort of of public acknowledgement whatsoever, has accidentally vindicated neurosymbolic AI.”
Our reading: Without any public acknowledgement whatsoever, OpenAI has accidentally vindicated neurosymbolic AI.
No horizon Evidence as of
Limitation: Company-reported improvements establish limited practical benefits, not the necessity of neurosymbolic AI for general intelligence.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“A significant fraction of the money that people have paid for Generative AI has been for coding assistance.”
Our reading: A significant fraction of the money people have paid for generative AI has been for coding assistance.
No horizon Evidence as of
Limitation: The evidence supplies no matched coding-payment numerator and total generative AI-payment denominator covering the same period and market.
Source: BREAKING NEWS: AI coding may not be helping as much as you think
Assessment: UnresolvedSaid
Observation
“When I point to the positive uses of Generative AI, I invariably point to how GenAI serves as kind of valuable autocomplete for coding.”
Our reading: When pointing to positive uses of generative AI, Marcus invariably points to how it serves as kind of a valuable autocomplete for coding.
No horizon Evidence as of
Limitation: Search coverage is incomplete and some interviews are edited.
Source: BREAKING NEWS: AI coding may not be helping as much as you think
Assessment: UnresolvedSaid
Observation
“Many of us in the field have started to get emails regularly from people who believe that they have seen signs of consciousness in LLMs.”
Our reading: Many AI researchers have started regularly receiving emails from people who believe they have seen signs of consciousness in LLMs.
No horizon Evidence as of
Limitation: Private correspondence is difficult to verify through public sources.
Assessment: UnresolvedSaid
Observation
“It took me less than an hour to find half a dozen clear counterexamples in which ChatGPT 4o failed with respect to compositionality, evident even to a teenager.”
Our reading: In less than an hour, Marcus found half a dozen clear counterexamples in which ChatGPT 4o failed with respect to compositionality, failures evident even to a teenager.
No horizon Evidence as of
Limitation: The evidence does not establish Marcus's discovery interval, identify all six claimed prompt-output pairs, or verify their model identity and clarity to non-specialists.
Assessment: UnresolvedSaid
Conditional forecast
“As the number of propositions and the complexity of what is represented increases (e.g., to include logical quantifiers), I expect encoding too will turn out to be unreliable.”
Our reading: As propositions become more numerous and complex, LLMs' encoding of them will also prove unreliable.
No deadline stated Evidence as of
Only if: as number and complexity of propositions increases
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Observation
“Game play continued, and I was able to beat the “smartest AI on earth” in four moves.”
Our reading: Grok 3 lost to Marcus in four moves at a tic-tac-toe variant that used y's and z's.
No horizon Evidence as of
Limitation: A public record may exist outside the indexed sources searched.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“LLMs try — to their peril — to live without classical world models.”
Our reading: LLMs try to function without classical world models, and this harms them.
No horizon Evidence as of
Limitation: The cited experiment concerns planning rather than the full range of LLM failures.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“In short ChatGPT can approximate the game of chess, but it can’t play it reliably, precisely because (despite immense relevant evidence) it never induces proper world model of the board and the rules.”
Our reading: ChatGPT cannot play chess reliably because it never induces a proper world model of the board and the rules.
No horizon Evidence as of
Limitation: The available intervention studies concern earlier ChatGPT experiments or different language models.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“The fact that they can get by as far as they can without explicit world models is actually astonishing. But much of what ails them comes from that design choice.”
Our reading: Much of what goes wrong with LLMs comes from the design choice to do without explicit world models.
No horizon Evidence as of
Limitation: Much and the relevant population of failures remain unspecified.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“And that is why they hallucinate, frequently. They wouldn’t need to do so if they actually had reliable access to relevant databases.”
Our reading: LLMs hallucinate often because they rebuild facts stochastically from fragments of text instead of using structured databases.
No horizon Evidence as of
Limitation: The supporting experiment concerns structured workflow generation.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“A huge number of peculiar LLM errors can be understood in this way.”
Our reading: The lack of world models explains a huge number of peculiar LLM errors.
No horizon Evidence as of
Limitation: The claim supplies no threshold for huge number.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Observation
“Text in general is better now (though not in this image) but watch comprehension of any time other than 10:10 (most frequent in advertisements) remains dreadful.”
Our reading: Imagen 4 is better at text in general now, but its handling of watch times other than 10:10 remains dreadful.
No horizon Evidence as of
Limitation: The packet contains no representative watch-time evaluation matched to the June 2025 Imagen 4 version.
Assessment: UnresolvedSaid
Observation
“Now? Here’s what I got out of Imagen 4, showing failure at the feature “binding problem” that some of us have been harping on for three decades:”
Our reading: Imagen 4 still fails at the feature binding problem on the stacked colored blocks prompt.
No horizon Evidence as of
Limitation: The decisive artifact is an image that the retrieved plain text does not make inspectable.
Assessment: UnresolvedSaid
Observation
“Not much better on some other tests of noncanonical images (which push language understanding harder), such as this one, adapted from an example Sandeep Manudhane sent me:”
Our reading: Imagen 4 is not much better on some other tests of noncanonical images, which push language understanding harder.
No horizon Evidence as of
Limitation: The comparison and degree of improvement are imprecise.
Assessment: UnresolvedSaid
Observation
“Anthropic literally and explicitly tells their system in the system prompt, don’t help user build biological weapons, and yet, with a bit of jailbreaking they can readily be coaxed into doing that:”
Our reading: Despite system-prompt prohibitions, Claude can readily be jailbroken into helping build biological weapons.
No horizon Evidence as of
Limitation: The available evidence supports a narrower capability claim more strongly than the complete statement.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: UnresolvedSaid
Causal claim
“No LLM is fully coherent in it’s “beliefs”, because by nature they parrot a variety conflicting things from their training, which is itself inconsistent, rather than reasoning their way a coherent view the way a rational person might.”
Our reading: No LLM has fully coherent beliefs, because LLMs repeat conflicting material from inconsistent training data instead of reasoning their way to a coherent view.
No horizon Evidence as of
Limitation: 'Parrot' versus 'reason' is partly a theoretical framing that studies do not measure directly.
Source: 1984, but with LLM’s
Assessment: UnresolvedSaid
Causal claim
“But downloading code without conceptual understanding is of less help in the case of new problems, dynamically changing environments, and so on.”
Our reading: LLMs retrieving code without conceptual understanding help less on new problems and dynamically changing environments.
No horizon Evidence as of
Limitation: The meaning of conceptual understanding remains unsettled in this claim.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: UnresolvedSaid
Forecast
“Mark my words: in time, we will see scores of papers reinforcing the Apple results.”
Our reading: Over time, scores of papers will reinforce the Apple paper's findings on reasoning model limitations.
No deadline stated Evidence as of
Limitation: The bibliography search is not exhaustive.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: UnresolvedSaid
Observation
“But also, we need to do so reliably, and in a general way and we haven’t yet crossed that threshold.”
Our reading: Current AI does not yet integrate neural and symbolic methods reliably and generally.
No horizon Evidence as of
Limitation: The claim supplies neither a reliability measure nor a required range of tasks.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: UnresolvedSaid
Observation
“Claude, on the other hand, can barely do 7 discs, getting less than 80% accuracy, left bottom panel below, and pretty much can’t get 8 correct at all.”
Our reading: Claude gets under 80% accuracy on 7-disc Tower of Hanoi and essentially cannot solve 8 discs.
No horizon Evidence as of
Limitation: The original PDFs exceeded the reading tool's size limit.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Observation
“(They also can’t play chess as well as conventional algorithms, can’t fold proteins like special-purpose neurosymbolic hybrids, can’t run databases as well as conventional databases, etc.)”
Our reading: LLMs can't play chess, fold proteins, or run databases as well as conventional or special-purpose systems.
No horizon Evidence as of
Limitation: The packet supplies no qualifying protein-structure or database-execution comparison.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Forecast
“But anybody who thinks LLMs are a direct route to the sort AGI that could fundamentally transform society for the good is kidding themselves.”
Our reading: Anybody who thinks LLMs are a direct route to the sort of AGI that could fundamentally transform society for the good is kidding themselves.
No deadline stated Evidence as of
Limitation: The result depends on an evaluator-supplied reading of AGI and 'direct route'.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Observation
“[Aside from this, models like o3 actually hallucinate a bunch more than attentive humans, struggle heavily with drawing reliable diagrams, etc; they happen to share a few weakness with humans, but on a bunch of dimensions they actually fall short.]”
Our reading: Models like o3 hallucinate much more than attentive humans and struggle with reliable diagrams.
No horizon Evidence as of
Limitation: The claim combines two distinct empirical assertions.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Observation
“In all, Charlotin’s database lists 112 cases, for an average of a bit less than one publicly reported case a week since ChatGPT become popular. And if May is representative, the problem is getting worse.”
Our reading: Charlotin's database lists 112 cases, an average of a bit less than one publicly reported case a week since ChatGPT became popular, and if May is representative, the problem is getting worse.
No horizon Evidence as of
Only if: if May is representative
Source: AI literacy, hallucinations, and the law: A case study
Assessment: UnresolvedSaid
Conditional forecast
“A camera and microphone. You would be monitored all the time. Everything you say, everything you do, and not just recorded, but analyzed, interpreted and turned into data, which of course OpenAI train their models on.”
Our reading: If OpenAI's planned io gadget is, as it would seem, an always-worn device with a camera and microphone, its users would be monitored all the time, with everything they say and do recorded, analyzed, interpreted and turned into data that OpenAI trains its models on.
No deadline stated Evidence as of
Only if: if the gadget is, as it would seem, worn all the time with a camera and microphone
Source: Black Mirror was a warmup act
Assessment: UnresolvedSaid
Observation
“The US has done almost nothing legislatively about the risks of AI (aside from deepfake porn).”
Our reading: The US has passed almost no legislation addressing AI risks, apart from deepfake porn.
No horizon Evidence as of
Limitation: The evidence items do not provide a complete inventory of federal protections enacted through May 22, 2025.
Assessment: UnresolvedSaid
Causal claim
“Their stories about this are, in my view, greatly flawed, but having outside groups with science fiction chops writing stuff like this distracts away from those flaws, and gives more power to the very companies trying hardest to race towards AGI.”
Our reading: OpenAI's and Anthropic's AGI-imminence stories are greatly flawed, and outside scenarios like AI 2027 distract from those flaws and empower those companies.
No horizon Evidence as of
Limitation: The criticism of the company stories is underspecified and partly evaluative.
Assessment: UnresolvedSaid
Observation
“And so far, anyway, the impact on the labor market of Generative AI has been modest; ditto for productivity in any measurable statistics.”
Our reading: So far, generative AI's impact on the labor market and on measurable productivity has been modest.
No horizon Evidence as of
Limitation: Geographic and occupational coverage is limited.
Assessment: UnresolvedSaid
Causal claim
“AI 2027 isn’t slowing them down; it’s putting wind (and money, and political power) in their sails.”
Our reading: AI 2027 is giving AI companies like OpenAI and Anthropic more money and political power rather than slowing them.
No horizon Evidence as of
Limitation: Investor motives and private political discussions may be unavailable.
Assessment: UnresolvedSaid
Forecast
“It’s also encouraging the world to make short-term choices about AI (e.g., making plans around export controls when China will inevitably catch up) rather than longer-term choices (investing in research to develop safer, more alignable approaches to AI).”
Our reading: China will inevitably catch up in AI despite export controls.
No deadline stated Evidence as of
Limitation: The interpretation of catch up materially affects the result.
Assessment: UnresolvedSaid
Causal claim
“Tall tales about the imminence of AGI aren’t slowing down the AI race dynamic that the authors of AI-2027 want to mitigate; they are speeding that very dynamic up.”
Our reading: Stories claiming AGI is imminent are speeding up, not slowing, the AI race dynamic.
No horizon Evidence as of
Limitation: The accessible experimental evidence measures attention, not development decisions.
Assessment: UnresolvedSaid
Observation
“Cell phones and the internet were pretty big, but it’s not obvious their impact was greater than the industrial revolution, and so far AI’s impact has (despite all the PR) been far less than either cell phones or the internet.”
Our reading: So far, AI's impact has been far less than that of cell phones or the internet.
No horizon Evidence as of
Limitation: The evidence provides no common measure of accumulated impact or threshold for 'far less'.
Assessment: UnresolvedSaid
Observation
“We have no concretely implemented international AI governance, no national AI agency; we are no longer even positioned well to detect and address AI-escalated cybercrime.”
Our reading: As of May 2025, there is no concretely implemented international AI governance and no US national AI agency, and the US is no longer even well positioned to detect and address AI-escalated cybercrime.
No horizon Evidence as of
Limitation: The OECD article is retrospective and carries an authors' views disclaimer.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: UnresolvedSaid
Observation
“By many accounts, AI-fueled scams have exploded, too, and again there is no coherent federal response.”
Our reading: By many accounts, AI-fueled scams have exploded, and there is no coherent US federal response to them.
No horizon Evidence as of
Limitation: Vendor detection data can change with customer coverage and detection methods.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: UnresolvedSaid
Causal claim
“Because both countries are both building essentially the same kind of models with the same kinds of techniques using the same kinds of data, the results from the two nations are essentially converging on the same outcomes.”
Our reading: US and Chinese AI results are converging because both use the same kinds of models, techniques, and data.
No horizon Evidence as of
Limitation: A supported assessment of benchmark convergence alone would drop the claim's causal commitment.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Causal claim
“With no regulatory pressure, tech companies would have little incentive to prioritize safety, transparency, or ethical design; any costs to society would be borne by society.”
Our reading: With no regulatory pressure, tech companies would have little incentive to prioritize safety, transparency, or ethical design, and any costs to society would be borne by society.
No horizon Evidence as of
Only if: with no regulatory pressure
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Conditional forecast
“First, if current trends continue, the “race” with China will not end in victory, for either side.”
Our reading: If current trends continue, neither the US nor China will win the AI race.
No deadline stated Evidence as of
Only if: if current trends continue
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Observation
“So-called leaderboards are no longer dominated by any one country.”
Our reading: AI leaderboards are no longer dominated by any single country.
No horizon Evidence as of
Limitation: Several categories from one benchmark operator are not independent measures.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Forecast
“With enough money you can have TAs and smaller sections of say 20 students each, taught by teaching assistants, but with university budgets being slashed, few universities will have enough money to make that work.”
Our reading: With university budgets being cut, few universities will have enough money to replace term papers with small TA-led sections.
No deadline stated Evidence as of
Limitation: US universities are an interpretive population choice.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Causal claim
“GenAI has nuked all of this.”
Our reading: GenAI has destroyed term papers' role in giving writing practice, prompting reflection and evaluating students at scale.
No horizon Evidence as of
Limitation: The strongest negative experiment examines take-home examinations rather than term papers.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Causal claim
“The creators and distributors of GenAI, who have brought this huge problem to society?”
Our reading: The creators and distributors of GenAI have brought this huge problem, GenAI undermining college education and evaluation, to society.
No horizon Evidence as of
Limitation: The strongest direct experiment covers five psychology modules at one UK university.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Causal claim
“(Those systems also can’t reliably stick to the rules of chess despite having them – and millions of games – in their database, a manifestation of the related problem of extracting statistical tendencies without every fully deriving and apprehending the correct abstractions).”
Our reading: LLMs can't reliably stick to the rules of chess despite having the rules and millions of games in their data, and this is a manifestation of the related problem of extracting statistical tendencies without ever fully deriving and grasping the correct abstractions.
No horizon Evidence as of
Limitation: Puzzle results, full-game results, and playing strength are not interchangeable.
Assessment: UnresolvedSaid
Observation
“They don’t have a database of records like any proper business would (which would be a strong basis to solve the problem); and they don’t have what people like Yann LeCun or Judea Pearl or I would call a world model.”
Our reading: LLMs don't have a database of records like any proper business would, and they don't have what people like Yann LeCun, Judea Pearl, or Marcus would call a world model.
No horizon Evidence as of
Limitation: One documented architecture cannot establish a claim about all LLMs.
Assessment: UnresolvedSaid
Forecast
“We will eventually get to something better, but continuing to put all our eggs in the Henrietta’s LLM basket is absurd to the point of being delusional.”
Our reading: AI will eventually get to something better than LLMs.
No deadline stated Evidence as of
Limitation: The result depends on how broadly 'something better' is read.
Assessment: UnresolvedSaid
Observation
“And it would be absurd to say that o3 can do all tasks that humans can do in 1.7 hours or that GPT-3 could do anything that a person can do in 15 seconds”
Our reading: o3 cannot do all tasks humans can do in 1.7 hours, and GPT-3 could not do everything a person can do in 15 seconds.
No horizon Evidence as of
Limitation: Unresolved reflects an evidence gap, not evidence that either model has universal capability.
Source: The latest AI scaling graph - and why it hardly makes sense
Assessment: UnresolvedSaid
Causal claim
“Claude does what LaMDA does better because it has more data, but I don’t see any really argument that Claude is any more sentient than a web browser.”
Our reading: Claude does what LaMDA does better because it has more data, but Marcus says he does not see any real argument that Claude is any more sentient than a web browser.
No horizon Evidence as of
Limitation: Private training information limits causal assessment.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: UnresolvedSaid
Observation
“What these systems do, no more and no less, is to put together sequences of words, but without any coherent understanding of the world behind them, like foreign language Scrabble players who use English words as point-scoring tools, without any clue about what that mean.”
Our reading: LLMs only string together sequences of words and have no coherent understanding of the world behind them.
No horizon Evidence as of
Limitation: The positive mechanistic evidence comes from the model developer and has acknowledged methodological limits.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: UnresolvedSaid
Observation
“NVidia is down significantly more this year (23.4%) than the S&P (11.5%).”
Our reading: NVidia's stock is down 23.4% in 2025 so far, significantly more than the S&P's 11.5% decline.
No horizon Evidence as of
Limitation: Primary historical records sought were inaccessible or returned the wrong period.
Source: Poor ROI for GenAI
Assessment: UnresolvedSaid
Observation
“Real world customers, many of whom are disappointed, are rarely mentioned.”
Our reading: Real-world customers, many of whom are disappointed, are rarely mentioned in the pundits' discussion of AI progress.
No horizon Evidence as of
Limitation: The survey was commissioned by an AI vendor and measures adoption disappointment, which can include organizational problems as well as product shortcomings.
Source: Deep Learning, Deep Scandal
Assessment: UnresolvedSaid
Observation
“Deep learning is indeed finally hitting a wall, in the sense of reaching a point of diminishing results.”
Our reading: Deep learning is hitting a wall in the sense of reaching diminishing returns.
No horizon Evidence as of
Limitation: METR evaluated an earlier GPT-4.5 checkpoint in a short, preliminary evaluation.
Source: Deep Learning, Deep Scandal
Assessment: UnresolvedSaid
Observation
“An LLM just spits back (sometimes in transmogrified form) what it has read.”
Our reading: An LLM just spits back what it has read, sometimes in transmogrified form.
No horizon Evidence as of
Limitation: Unresolved does not mean that the evidence favors the claim.
Assessment: UnresolvedSaid
Observation
“Humans are (usually, if well-motivated) smart enough to sort things like jokes and fiction from reality, but current AI doesn’t have a clue.”
Our reading: Current AI systems cannot tell jokes and fiction apart from reality, while well-motivated humans usually can.
No horizon Evidence as of
Limitation: The cited studies do not establish either representative failures or a usable ability on the specified fiction-versus-fact task during the assessment period.
Assessment: UnresolvedSaid
Observation
“I do genuinely believe today’s Turing Test takers are better AI systems than an earlier generation. I hardly think that means anything is “over”.”
Our reading: Today's Turing Test-taking AI systems are better than the earlier generation, but that does not mean anything is 'over'.
No horizon Evidence as of
Limitation: This is an evidence gap, not evidence that contemporary systems lack broader improvements.
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Conditional forecast
“But I am willing to wager that if someone put in that kind of effort and carefully built out the test, no current system would come close to human performance.”
Our reading: If a careful comprehension challenge test were built, no current AI system would come close to human performance on it.
No deadline stated Evidence as of
Only if: if someone carefully built out the comprehension challenge test
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Observation
“Unfortunately, nobody has ever fully implemented the comprehension challenge test that I proposed.”
Our reading: Nobody has fully implemented Marcus's proposed comprehension challenge test.
No horizon Evidence as of
Limitation: The boundary between a related benchmark and full implementation is not formally defined.
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Forecast
“Agents will be a disappointment, precisely because of all of the ongoing reliability and hallucination issues.”
Our reading: AI agents will be a disappointment because of ongoing reliability and hallucination problems.
No deadline stated Evidence as of
Limitation: The evidence mixes enterprise decision-makers, agent developers and deployed-system practitioners.
Assessment: UnresolvedSaid
Forecast
“For most GenAI model building companies, profits will be meager.”
Our reading: Most GenAI model-building companies will have meager or no profits.
No deadline stated Evidence as of
Limitation: The evidence concerns one company's projected losses before the claim was published, not comparable realized results for a defined population [e1].
Assessment: UnresolvedSaid
Observation
“This And the promised major jumps in economic productivity simply haven’t emerged.”
Our reading: The promised major gains in economic productivity from GenAI have not materialized.
No horizon Evidence as of
Limitation: The US-focused evidence does not establish a worldwide result or isolate GenAI's aggregate contribution in the target quarter [e1, e2].
Assessment: UnresolvedSaid
Observation
“(Or nil; most are losing money; few have made any profits at all).”
Our reading: Most GenAI model-building companies are losing money and few have made any profit.
No horizon Evidence as of
Limitation: Anthropic's historical results were disclosed after the post but before the evaluation date.
Assessment: UnresolvedSaid
Forecast
“The end game here is not that LLMs will vanish, but that LLMs will be virtually free.”
Our reading: LLMs will not vanish but will ultimately become virtually free.
No deadline stated Evidence as of
Limitation: Consumer chat, API inference and downloadable weights have different access costs.
Assessment: UnresolvedSaid
Observation
“User disillusionment continues to grow.”
Our reading: User disillusionment with generative AI is continuing to grow.
No horizon Evidence as of
Limitation: The satisfaction comparison covers shopping-related use in 2024, not the target period in early 2025 [e1].
Assessment: UnresolvedSaid
Observation
“NVidia, the largest company actually making serious profits from AI, is down 17.75% for the year, and (after a meteoric rise) clearly facing headwinds.”
Our reading: Nvidia, the largest company making serious AI profits, is down 17.75% year-to-date in 2025 and clearly facing headwinds.
No horizon Evidence as of
Limitation: Several promising market-data records could not be read with the required tool.
Assessment: UnresolvedSaid
Forecast
“GenAI is never going to disappear.”
Our reading: Generative AI will never disappear.
No deadline stated Evidence as of
Limitation: The survey covers US Gen Z respondents, not all populations or deployments [e1].
Assessment: UnresolvedSaid
Forecast
“AI will be a tool to help people write code, just as spell-checkers are a tool to help authors write articles and novel, but AI will not soon replace people who understand how to conceive of, write, and debug code.”
Our reading: AI will serve as a tool to help people write code but will not soon replace people who know how to conceive of, write, and debug code.
No deadline stated Evidence as of
Limitation: The available employment research does not directly measure responsibility for conceiving and debugging software.
Source: AI Coding Fantasy meets Pac-Man
Assessment: UnresolvedSaid
Observation
“And they don’t magically resolve the challenges of maintaining complex code bases over time.”
Our reading: Current AI coding tools do not resolve the challenges of maintaining complex code bases over time.
No horizon Evidence as of
Limitation: The study does not establish whether its findings apply to target-period tools [e1].
Source: AI Coding Fantasy meets Pac-Man
Assessment: UnresolvedSaid
Forecast
“The idea that coders (and more generally, software architects) are on their way out is absurd.”
Our reading: Coders and software architects are not on their way out due to AI.
No deadline stated Evidence as of
Limitation: The ordinary occupational-obsolescence reading differs from a claim that employment will never decline.
Source: AI Coding Fantasy meets Pac-Man
Assessment: UnresolvedSaid
Observation
“A few days ago he triumphantly claimed (without evidence) that nearly all code would be written by AI within the next year:”
Our reading: Dario Amodei's claim that nearly all code would be written by AI within the next year was made without evidence.
No horizon Evidence as of
Limitation: The readable transcription is unofficial.
Source: AI Coding Fantasy meets Pac-Man
Assessment: UnresolvedSaid
Observation
“That is not going to magically change next year, at a time when AI still can’t reliably solving river-crossing problems.”
Our reading: Current AI cannot reliably solve river-crossing puzzles.
No horizon Evidence as of
Limitation: Unresolved does not mean the models were reliable.
Assessment: UnresolvedSaid
Observation
“Nobel-calibre science requires sound logic, reliable causal reasoning, and originality, and none of those are firmly in the grasp of current AI.”
Our reading: Current AI does not firmly have the sound logic, reliable causal reasoning, and originality that Nobel-calibre science requires.
No horizon Evidence as of
Limitation: The two benchmark results are preprints and concern particular tasks and model samples.
Assessment: UnresolvedSaid
Observation
“• AGI should have a reliable ability to understand complex syntactic and semantic structure. (Failings here are also often reflected in quirky errors.”
Our reading: Current AI systems lack a reliable ability to understand complex syntactic and semantic structure, and these failings are often reflected in quirky errors.
No horizon Evidence as of
Limitation: This is an evidence gap, not a finding that the alleged errors do not occur.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Observation
“Especially when we just saw GPT-5 (ultimately relabeled as GPT 4.5) fail despite more than two years of hard and expensive work.”
Our reading: OpenAI's planned GPT-5, relabeled GPT-4.5, failed despite more than two years of hard and expensive work.
No horizon Evidence as of
Limitation: Product improvements can coexist with missed internal expectations.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Causal claim
“But it is not just that I think Klein is wrong, or that he did a disservice to the other side of the argument, but that words like his are motivating a lot of dodgy bets being placed on the imminent AGI assumption — in ways that may not make sense.”
Our reading: Claims like Klein's that AGI is imminent are motivating many dodgy bets that may not make sense.
No horizon Evidence as of
Limitation: Public explanations may omit internal motives.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Observation
“What that means is that a quarter century has passed without a principled solution to even one of those problems.”
Our reading: In the quarter century since 2001, none of the listed AGI prerequisites (generalization, abstract reasoning, tracking individuals, understanding structure) has received a principled solution.
No horizon Evidence as of
Limitation: A narrow successful algorithm would not refute the contextual claim about general capabilities.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Causal claim
“(My view, as I explained in Taming Silicon Valley, is that industry has a vested interest in making people believe that AGI is imminent — how else to justify the lavish valuations - and that they have fooled a lot of people in government who aren’t technical experts into believing that AGI is closer than it really is).”
Our reading: The AI industry has a vested interest in claiming AGI is imminent to justify lavish valuations, and has misled many non-technical government officials into thinking AGI is closer than it is.
No horizon Evidence as of
Limitation: One company's announcement cannot establish an industry-wide valuation mechanism.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Observation
“The fact that Grok 3 was 15x the size of Grok 2, and also generally viewed as disappointing.”
Our reading: Grok 3 was 15 times the size of Grok 2 and was generally viewed as disappointing.
No horizon Evidence as of
Limitation: The scale judgment depends on what size means.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Observation
“The fact that GPT 4.5 received very mixed reviews — despite years of labor and billions investment — and could in no reasonable way be considered a major (rather than minor, incremental) step forwards towards AGI.”
Our reading: GPT-4.5 received very mixed reviews and is at most a minor, incremental step toward AGI, despite years of work and billions invested.
No horizon Evidence as of
Limitation: Company benchmarks and limited pre-release evaluations do not exhaust a model's capabilities.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Forecast
“And these complex systems, built on the unreliable foundations of LLMs, are going to prove to be massively difficult to debug.”
Our reading: Complex systems built on LLMs, such as deep research agents, will prove massively difficult to debug.
No deadline stated Evidence as of
Limitation: The production account is a company report without quantified debugging effort.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Forecast
“But there’s also no reason to think that any of them are logically insolvable in principle. So we will get there someday.”
Our reading: Because there is no reason to think any of the remaining problems are logically unsolvable in principle, AGI will be achieved someday.
No deadline stated Evidence as of
Limitation: The search cannot establish that every possible demonstration has been examined.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Forecast
“I can’t imagine that noncoders are really going to get those systems to work well enough for prime time (except perhaps in very very narrow use cases), let alone know what do when the back end internet calls break.”
Our reading: Marcus says he can't imagine noncoders getting systems built by automated code tools to work well enough for prime time, except perhaps in very narrow use cases.
No deadline stated Evidence as of
Limitation: Both sources are company accounts without independent operational verification.