Claims checked against evidence by Gary Marcus, page 6 of 19
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1870)
Each of these claims was checked against dated evidence.
Page 6 of 19: claims 501 to 600 of 1870, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Observation
“Pricing is good, but profits may continue to be elusive; still no clear technical moat.”
Our reading: OpenAI still has no clear technical moat with GPT-5.
No horizon Evidence as of
Limitation: Competitor disclosures are self-reported.
Source: GPT-5 hot take
Assessment: SupportedSaid
Observation
“GPT-5 is obviously not AGI.”
Our reading: GPT-5 is not AGI.
No horizon Evidence as of
Limitation: The outcome depends on the stated interpretation of AGI.
Source: GPT-5 hot take
Assessment: SupportedSaid
Observation
“OpenAI conveniently forgot to include this comparison (ARC-AGI-2) in their livestream recital of benchmark progress, which left the livestream looking like marketing rather than science.”
Our reading: OpenAI left the ARC-AGI-2 comparison out of its GPT-5 livestream benchmarks, which made the livestream look like marketing rather than science.
No horizon Evidence as of
Limitation: The decisive source is contemporaneous specialist reporting rather than an independently inspected complete recording.
Source: GPT-5 hot take
Assessment: SupportedSaid
Observation
“But still part of the pack, not a giant leap forward (e.g. Grok 4 beats it on ARC-AGI-2 results)”
Our reading: GPT-5 is still part of the pack rather than a giant leap forward; for example, Grok 4 beats it on ARC-AGI-2.
No horizon Evidence as of
Limitation: The broader conclusion depends on the stated comparison baseline.
Source: GPT-5 hot take
Assessment: UnresolvedSaid
Causal claim
“The AI industry magic sauce of “scaling” hasn’t made the enshittification situation better; it has made it worse.”
Our reading: Scaling of AI has worsened, not improved, AI-induced enshittification.
No horizon Evidence as of
Limitation: Benchmark results are contextual evidence for a claim about the public information environment.
Source: Slopocalypse Now
Assessment: UnresolvedSaid
Observation
“That has, of course, only gotten worse. Much worse, as Science reported today.”
Our reading: Generative AI's degradation of science has gotten much worse since early 2024.
No horizon Evidence as of
Limitation: A narrower claim about rising suspicious references has support.
Source: Slopocalypse Now
Assessment: UnresolvedSaid
Causal claim
“The problem of course is that some significant fraction of that science will be wrong, because of the inevitable tendency of LLMs to hallucinate.”
Our reading: Because LLMs inevitably hallucinate, a significant fraction of science produced with them will be wrong.
No horizon Evidence as of
Limitation: The largest real-world audit cited here is a preprint.
Source: Slopocalypse Now
Assessment: SupportedSaid
Observation
“He still prefers “enshittification” to “slop”, but the problem is both serious and ubiquitous no matter what you call it.”
Our reading: The problem of AI slop/enshittification is serious and ubiquitous.
No horizon Evidence as of
Limitation: The result depends on the contract's reading of 'serious' and 'ubiquitous', which are evaluative or hyperbolic words.
Source: Slopocalypse Now
Assessment: SupportedSaid
Observation
“But science is far from alone. AI slop is everywhere. It’s in journalism, law, music (which Nick Cave once called “a grotesque mockery of what it is to be human”), politics, education, you name it.”
Our reading: AI-generated slop is pervasive across journalism, law, music, politics, education and other domains.
No horizon Evidence as of
Limitation: The result depends on reading 'everywhere' as cross-sector breadth.
Source: Slopocalypse Now
Assessment: SupportedSaid
Observation
“All the big companies have in fact introduced agents.”
Our reading: All the big AI companies have introduced AI agents as of August 2025.
No horizon Evidence as of
Limitation: The company universe is undefined.
Assessment: UnresolvedSaid
Causal claim
“In practice, those mistakes and errors have greatly limited the utility agents like ChatGPT agent.”
Our reading: Mistakes and errors have greatly limited the utility of agents like ChatGPT agent.
No horizon Evidence as of
Limitation: The causal evidence consists of small, selected task demonstrations.
Assessment: SupportedSaid
Observation
“The unrelenting hallucinations problem frequently rears its ugly head:”
Our reading: Hallucinations frequently occur in AI agents.
No horizon Evidence as of
Limitation: No standard rate measure; 'frequently' is judged from repeated documented cases.
Assessment: UnresolvedSaid
Observation
“That sentiment that has only grown over the last several months.”
Our reading: The sentiment that AI agents' reality doesn't match the hype has grown from March to August 2025.
No horizon Evidence as of
Limitation: Trust and disappointment with hype are different measures.
Assessment: SupportedSaid
Causal claim
“As a consequence of their superficial understanding of what they’re being asked, current agents are grossly vulnerable to cyberattacks.”
Our reading: Current AI agents are grossly vulnerable to cyberattacks because of their superficial understanding of their tasks.
No horizon Evidence as of
Limitation: No study in the packet tests 'superficial understanding' as a cause directly. The causal link rests on how successful attacks worked and on one expert's explanation.
Assessment: UnresolvedSaid
Observation
“But, none, to my knowledge are reliable, except maybe in very narrow uses cases.”
Our reading: To his knowledge, none of the big companies' AI agents are reliable, except maybe in very narrow use cases.
No horizon Evidence as of
Limitation: The July reports do not provide representative, repeated trials or systematic rates of consequential errors and human intervention [e2, e3].
Assessment: SupportedSaid
Causal claim
“In fundamentally unreliable systems like LLMs, that means multiple chances for error. Soon or later, those errors catch up. That can and sometimes does lead to catastrophe.”
Our reading: Because LLMs are fundamentally unreliable, multi-step agentic tasks accumulate errors that sometimes lead to catastrophe.
No horizon Evidence as of
Limitation: The evidence is incident reports, not a measured rate of how often multi-step errors cause serious damage.
Assessment: UnresolvedSaid
Observation
“Reliable agents may not require “artificial general intelligence”, but they surely require what I called in 2020 “robust”, trustworthy AI. LLMs aren’t getting us there.”
Our reading: Reliable agents require robust, trustworthy AI, and LLMs are not getting us there.
No horizon Evidence as of
Limitation: The reporting window and operational meaning of progress are evaluator interpretations.
Assessment: SupportedSaid
Forecast
“Guardrails (a la ChatGPT) may be in place, but the guardrails will teeter between being too weak (beaten by “jailbreaks”) and too strong (rejecting some perfectly reasonable requests).”
Our reading: Guardrails like ChatGPT's may be in place for GPT-5, but they will teeter between being too weak (beaten by jailbreaks) and too strong (rejecting some perfectly reasonable requests).
Settled before its deadline Evidence as of
Limitation: The evaluations do not establish temporal oscillation, ordinary-user failure rates, or persistence in one unchanged deployment [e2, e3].
Assessment: UnresolvedSaid
Forecast
“For that, we will, I predict, need genuinely new architectures that incorporate explicit knowledge and world models at their very core.”
Our reading: Trustworthy AI will require genuinely new architectures with explicit knowledge and world models at their core.
No deadline stated Evidence as of
Limitation: Behavior on a bounded benchmark cannot establish broad trustworthiness.
Assessment: UnresolvedSaid
Forecast
“GPT-5 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
Our reading: GPT-5, like its predecessors, will be reckless and hard to control.
No deadline stated Evidence as of
Limitation: METR's task suite does not measure the prevalence or consequences of unintended behavior across ordinary deployments, and its detection methods may miss reward hacking [e1].
Assessment: UnresolvedSaid
Forecast
“Without external aids it won’t be able beat Meta’s Cicero in Diplomacy; it won’t be able to drive a car reliably; it won’t be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.”
Our reading: Without external aids, GPT-5 will not be able to beat Meta's Cicero at Diplomacy, will not be able to drive a car reliably, and will not be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.
No deadline stated Evidence as of
Limitation: This compound claim cannot be supported from evidence about robotics alone.
Assessment: UnresolvedSaid
Forecast
“Its natural language output still won’t be something that one can reliably hook up to downstream programs; it won’t be something, for example, that you can simply and directly hook up to a database or virtual assistant, with predictable results.”
Our reading: GPT-5's natural language output will not be something one can reliably hook up to downstream programs, for example simply and directly to a database or virtual assistant, with predictable results.
No deadline stated Evidence as of
Limitation: Distinguishing natural language from plaintext code materially affects the assessment.
Assessment: UnresolvedSaid
Conditional forecast
“Trustworthy, general artificial intelligence, aligned with human values, will come, when it does, from systems that are more structured, with more built-in knowledge, and will incorporate at least some degree of explicit tools for reasoning and planning, as well as explicit knowledge, that are lacking in systems like GPT.”
Our reading: When trustworthy general AI aligned with human values comes, it will come from systems that are more structured, have more built-in knowledge, and incorporate at least some degree of explicit tools for reasoning and planning, as well as explicit knowledge, that systems like GPT lack.
No deadline stated Evidence as of
Only if: when it does come
Assessment: UnresolvedSaid
Forecast
“GPT-5 will not have reliable models of the things that it talks about that are accessible to external programmers in a way that reliably feeds downstream processes.”
Our reading: GPT-5 will not have reliable models of the things it talks about that external programmers can access in a way that reliably feeds downstream processes.
No deadline stated Evidence as of
Limitation: The meaning of an accessible subject model materially affects what evidence would count.
Assessment: UnresolvedSaid
Forecast
“In retrospectives written in 2043, intellectual historians will conclude that there was an initial overemphasis on large language models, and a gradual but critical shift of the pendulum back to more structured systems with deeper comprehension.”
Our reading: In retrospectives written in 2043, intellectual historians will conclude that there was an initial overemphasis on large language models and a gradual but critical shift of the pendulum back to more structured systems with deeper comprehension.
Not yet due Evidence as of
Limitation: Future assessment will require a transparent selection of relevant historians and retrospectives.
Assessment: UnresolvedSaid
Observation
“I was right on the money then.”
Our reading: Marcus's December 2022 predictions about GPT-4 were accurate.
No horizon Evidence as of
Limitation: The assessment reads the retrospective as referring to all seven predictions; its intended scope is not explicit.
Assessment: SupportedSaid
Forecast
“Examples of concealed bias will be discovered within days or months.”
Our reading: Examples of concealed bias in GPT-5 will be discovered within days or months of release.
Settled before its deadline Evidence as of
Limitation: The meaning of concealed determines the verdict.
Assessment: SupportedSaid
Forecast
““Alignment” between what humans want and what machines do will continue to be a critical, unsolved problem.”
Our reading: Alignment between human wants and machine behavior will remain a critical unsolved problem.
Settled before its deadline Evidence as of
Limitation: This supports persistence, not a prediction that alignment will remain unsolved forever.
Assessment: UnresolvedSaid
Forecast
““Scaling” alone—building bigger and models until they absorb the entire internet — will prove useful, but only to a point.”
Our reading: Scaling alone will prove useful only up to a point.
No deadline stated Evidence as of
Limitation: Finite experiments cannot establish what all future increases in scale will achieve.
Assessment: SupportedSaid
Observation
“For years we have been hearing, endlessly, about how GPT-5 was going to land imminently, and those predictions turned out to be wrong so often that a year ago I wrote a post about it, called GPT-5…now arriving Gate 8, Gate 9, Gate 10, not to mention a couple of April Fool’s jokes.”
Our reading: Predictions over several years that GPT-5 would arrive imminently were repeatedly wrong.
No horizon Evidence as of
Limitation: Support depends on including qualified public speculation within 'predictions'.
Assessment: UnresolvedSaid
Observation
“Neither DeepMind nor OpenAI has given any substantial information about how their systems are designed and trained.”
Our reading: Neither DeepMind nor OpenAI has given substantial information on how their IMO systems are designed and trained.
No horizon Evidence as of
Limitation: Substantial is a judgment threshold, not a specified quantity.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Forecast
“Eric Schmidt’s recent prediction (made before IMO 2025), that there will be AI-based “world-class mathematicians” within a year, is pure fantasy.”
Our reading: There will not be AI-based world-class mathematicians within a year (by about July 2026).
Deadline passed Evidence as of
Limitation: The evidence supplies no agreed standard connecting one exceptional result to world-class research capability.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Forecast
“Tying for 27th place in a highly elite high school mathematics competition will not immediately change that.”
Our reading: Tying for 27th place at the IMO will not immediately change the fact that only one theorem of middling importance (the Robbins conjecture, in 1996) has ever been proved autonomously by an AI system.
No deadline stated Evidence as of
Limitation: “Immediately” has no operational endpoint; open_horizon does not extend it indefinitely.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: ContradictedSaid
Observation
“In the real world, only one theorem of middling importance (the Robbins conjecture) has ever been proved autonomously by an AI system, and that was back in 1996.”
Our reading: Only one theorem of middling importance (the Robbins conjecture, 1996) has ever been proved autonomously by an AI system.
No horizon Evidence as of
Limitation: Mathematical importance is qualitative; the sources do not supply an exact comparison with Robbins.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: SupportedSaid
Observation
“They have not made clear plans to release the system generally or make it available for experimentation. They have made it clear that it will not be available for months, even in the best case.”
Our reading: OpenAI has not made clear plans to release OpenAI-IMO generally or make it available for experimentation, and has made it clear that it will not be available for months, even in the best case.
No horizon Evidence as of
Limitation: The statement describes intended release timing, not an irreversible commitment.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: SupportedSaid
Observation
“In contrast, OpenAI communicated very little with the IMO prior to announcing their success.”
Our reading: OpenAI communicated very little with the IMO before announcing its gold-level result.
No horizon Evidence as of
Limitation: The conclusion relies on participant accounts rather than a complete correspondence record.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Observation
“None of this is inevitable in LLMs. Most other major LLMs don’t share these specific antisocial tendencies, at least not without much more focused efforts at jailbreaking.”
Our reading: Grok's antisemitism, fondness for Hitler and sexual-violence outputs are not inevitable in LLMs, and most other major LLMs don't share these specific antisocial tendencies, at least not without much more focused efforts at jailbreaking.
No horizon Evidence as of
Limitation: CNN does not provide individual trial dates, a complete account of model versions and configurations, or a consistent measure of attack effort [e1].
Assessment: UnresolvedSaid
Observation
“The system is so bound to Musk it even asked his opinion on pizza.”
Our reading: Grok is so bound to Musk that it sometimes searches for Musk's opinion before answering, and in one case even asked his opinion on pizza.
No horizon Evidence as of
Limitation: The available demonstration establishes occurrence, not how often the behavior happened or its underlying cause.
Assessment: UnresolvedSaid
Observation
“Grok itself is a long way from AGI, and hardly smart enough to map out an effective world domination.”
Our reading: Grok is a long way from AGI and hardly smart enough to map out an effective world domination.
No horizon Evidence as of
Limitation: The ARC result concerns one benchmark and is accessed through a reproduction of the operator's announcement [e1].
Assessment: UnresolvedSaid
Observation
“From this I infer that xAI’s main methodology for handling alignment is trial-and-error.”
Our reading: From xAI's account of the incident, Marcus infers that xAI's main methodology for handling alignment is trial-and-error.
No horizon Evidence as of
Limitation: Public disclosures may omit important internal practices.
Assessment: UnresolvedSaid
Observation
“My p(doom) is still lower than most industry people’s.”
Our reading: Marcus's p(doom) is lower than that of most AI industry people.
No horizon Evidence as of
Limitation: A researcher sample is not automatically representative of the AI industry.
Assessment: UnresolvedSaid
Observation
“And, currently, we have almost no regulation around AI, for that matter the billions of robots he aspires to build.”
Our reading: There is currently almost no regulation around AI, or around the billions of robots Musk aspires to build.
No horizon Evidence as of
Limitation: Both dated sources precede July 15 and do not provide a complete legal snapshot for that day [e1, e2].
Assessment: UnresolvedSaid
Observation
“In contrast, o3, when it draws on symbolic code (which it sometimes displays explicitly) can build crossword grids like this far faster than I could (short of my writing my own code).”
Our reading: When o3 draws on symbolic code, it can build crossword grids like this one far faster than Marcus could, short of writing his own code.
No horizon Evidence as of
Limitation: The informal experiment may have private records that are not publicly available.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“And it is about why, in 2025, neurosymbolic AI has emerged as the team to beat.”
Our reading: In 2025 neurosymbolic AI has emerged as the leading approach in AI.
No horizon Evidence as of
Limitation: The claim supplies neither a benchmark set nor a definition of overall leadership.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: MixedSaid
Observation
“Neural networks are good at learning but weak at generalization; symbolic systems are good at generalization, but not at learning.”
Our reading: Neural networks learn well but generalize poorly, while symbolic systems generalize well but learn poorly.
No horizon Evidence as of
Limitation: The studies do not provide a common comparison across all neural and symbolic methods [e1–e3].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Systems that draw on code interpreter, and hence by definition are neurosymbolic, often outperform those that lack symbolic tools, at least in a bunch of tasks. This is confirmation of the value of neurosymbolic AI.”
Our reading: Systems using code interpreter often outperform those without symbolic tools on many tasks, confirming the value of neurosymbolic AI.
No horizon Evidence as of
Limitation: The evidence comes from one preprint and selected benchmarks; it does not establish how frequently tools help across real-world tasks [e1].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Three years later, the pure scaling of pretraining data (and compute, which is to say more and more GPUs) simply hasn’t worked. The specific obstacles that I dwelled on, with respect to hallucinations, misalignment and reasoning errors, have not been overcome.”
Our reading: Pure scaling of pretraining data and compute has not worked; hallucinations, misalignment and reasoning errors remain.
No horizon Evidence as of
Limitation: The result depends on reading 'hasn’t worked' as failure to overcome the named problems.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Causal claim
“Very little of what has been tried has been discussed explicitly, and because the companies are often quite closed about what they are doing, the public science of neurosymbolic AI is greatly impoverished.”
Our reading: Company secrecy has greatly impoverished public science on neurosymbolic AI.
No horizon Evidence as of
Limitation: The transparency audit does not isolate neurosymbolic methods or their research impact [e1].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“As we will see in a moment, people are finally pursuing a tool-driven, neurosymbolic vision like this — and getting better results than they were able to get with pure neural networks.”
Our reading: Tool-driven neurosymbolic approaches are now being pursued and yield better results than pure neural networks.
No horizon Evidence as of
Limitation: The decisive studies were preprints reporting the authors' own experiments.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Models like o3 are far better than non-code-interpreter systems on some tasks but not all (for now o3 is actually worse than o1 on hallucinations) and effective for some tasks, but not others.”
Our reading: Models like o3 are far better than non-code-interpreter systems on some tasks but not all, and for now o3 is worse than o1 on hallucinations.
No horizon Evidence as of
Limitation: These are company-published evaluations, including expert results summarized by the company.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Forecast
“Between the strong success of DeepMind’s series of neurosymbolic hybrids (AlphaGo, AlphaFold, AlphaProof, AlphaGeometry, etc.) and the more recent results in which LLMs have been enhanced with python and other tools to dramatic effect, we can safely conclude that neurosymbolic AI is on track to be a major part of AI’s future.”
Our reading: Neurosymbolic AI is on track to be a major part of AI's future.
No deadline stated Evidence as of
Limitation: The search is not an exhaustive census of AI deployments.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“OpenAI quietly began doing this to some degree in 2023 with a system called “code interpreter”, in which LLMs (themselves neural networks) call (purely symbolic) Python interpreters. This is literally “putting programs inside”; exactly what Hinton said was a huge mistake.”
Our reading: In 2023, OpenAI quietly began, to some degree, 'putting programs inside' its systems with Code Interpreter, in which LLMs call purely symbolic Python interpreters. Marcus says this is exactly what Hinton called a huge mistake.
No horizon Evidence as of
Limitation: The launch and functionality component is supported.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Both informal experimentation and a number of recent quantitative results make it abundantly clear that current generative AI systems perform better when they avail themselves of symbolic tools.”
Our reading: Current generative AI systems perform better when they use symbolic tools.
No horizon Evidence as of
Limitation: The AIME results are company-reported; the cited scores alone do not quantify the improvement attributable to Python [e1].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“A quarter century later, hallucinations remain ubiquitous. Within a neural network substrate, the problem still has not been solved.”
Our reading: Hallucinations remain ubiquitous and unsolved in neural-network-based systems.
No horizon Evidence as of
Limitation: Both sources are developer disclosures [e1, e2].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Recent “reasoning” models borrow even more from classic symbolic approaches, such as techniques like search and conditionally iterating through multiple solutions, and aggregating the results, techniques that traditional networks had long eschewed, in favor of a single feed-forward pass through a neural net.”
Our reading: Recent reasoning models borrow symbolic techniques such as search, iteration over multiple solutions and aggregation.
No horizon Evidence as of
Limitation: Developer descriptions do not establish historical influence from symbolic AI or an explicitly programmed symbolic search implementation [e1, e2, e3].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Forecast
“Machine learning, the branch of AI concerned with tuning algorithms from data, is an amazing field that has changed the world — and will continue doing so.”
Our reading: Machine learning has changed the world and will continue to do so.
No deadline stated Evidence as of
Limitation: The meaning of substantial world change requires interpretation.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“I argued that scaling as it was then defined (pretraining data and compute) alone was insufficient to solve challenges with reasoning, misalignment and hallucinations. (Spoiler alert: it still hasn’t.)”
Our reading: Scaling pretraining data and compute alone has not solved reasoning, misalignment and hallucination problems.
No horizon Evidence as of
Limitation: This is an evidence limitation, not evidence that pretraining alone had solved the problems.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“So here’s where we are: pure LLMs can’t reliably apply algorithms (vindicating the line of argument I first developed in the 1990s); if you enhance them with symbolic processes—yielding neurosymbolic systems—they often give better results.”
Our reading: Pure LLMs cannot reliably apply algorithms, while LLMs enhanced with symbolic processes often perform better.
No horizon Evidence as of
Limitation: The outcome depends on reading the statement as a contemporary empirical generalization.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“For all the efforts that OpenAI and other leaders of deep learning, such as Geoffrey Hinton and Yann LeCun, have put into running neurosymbolic AI, and me personally, down over the last decade, the cutting edge is finally, if quietly and without public acknowledgement, tilting towards neurosymbolic AI.”
Our reading: The cutting edge of AI is finally tilting towards neurosymbolic AI, quietly and without public acknowledgement.
No horizon Evidence as of
Limitation: The result depends on counting code-interpreter integration as neurosymbolic AI.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“Yet nearly all funding has been aimed simply at making them larger.”
Our reading: Nearly all AI funding has gone to making LLMs larger.
No horizon Evidence as of
Limitation: The selected annual period cannot settle an unspecified cumulative historical claim.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: ContradictedSaid
Observation
“This is why so called “code agents” inevitably fall apart as soon as one gives them a more complex task which requires reasoning.”
Our reading: Code agents inevitably fail on more complex tasks requiring reasoning.
No horizon Evidence as of
Limitation: The result depends on taking inevitably literally and treating repository diagnosis and repair as complex reasoning.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Causal claim
“Second, adding symbolic tools (orange dots) dramatically improves performance.”
Our reading: Adding symbolic tools dramatically improves Grok 4's performance on Humanity's Last Exam.
No horizon Evidence as of
Limitation: The scores are reported company results; the packet does not establish independent replication [e1].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“OpenAI, without any sort of of public acknowledgement whatsoever, has accidentally vindicated neurosymbolic AI.”
Our reading: Without any public acknowledgement whatsoever, OpenAI has accidentally vindicated neurosymbolic AI.
No horizon Evidence as of
Limitation: Company-reported improvements establish limited practical benefits, not the necessity of neurosymbolic AI for general intelligence.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“A significant fraction of the money that people have paid for Generative AI has been for coding assistance.”
Our reading: A significant fraction of the money people have paid for generative AI has been for coding assistance.
No horizon Evidence as of
Limitation: The evidence supplies no matched coding-payment numerator and total generative AI-payment denominator covering the same period and market.
Source: BREAKING NEWS: AI coding may not be helping as much as you think
Assessment: UnresolvedSaid
Observation
“When I point to the positive uses of Generative AI, I invariably point to how GenAI serves as kind of valuable autocomplete for coding.”
Our reading: When pointing to positive uses of generative AI, Marcus invariably points to how it serves as kind of a valuable autocomplete for coding.
No horizon Evidence as of
Limitation: Search coverage is incomplete and some interviews are edited.
Source: BREAKING NEWS: AI coding may not be helping as much as you think
Assessment: UnresolvedSaid
Observation
“Many of us in the field have started to get emails regularly from people who believe that they have seen signs of consciousness in LLMs.”
Our reading: Many AI researchers have started regularly receiving emails from people who believe they have seen signs of consciousness in LLMs.
No horizon Evidence as of
Limitation: Private correspondence is difficult to verify through public sources.
Assessment: UnresolvedSaid
Observation
“(GPT 4o also failed to accurately diagnose its own errors, misreading its own output, which represent a different kind of failure of compositionality.)”
Our reading: GPT-4o failed to accurately diagnose errors in its own image outputs.
No horizon Evidence as of
Limitation: The assessment concerns the reported episodes, not a universal inability to inspect images.
Assessment: UnresolvedSaid
Observation
“It took me less than an hour to find half a dozen clear counterexamples in which ChatGPT 4o failed with respect to compositionality, evident even to a teenager.”
Our reading: In less than an hour, Marcus found half a dozen clear counterexamples in which ChatGPT 4o failed with respect to compositionality, failures evident even to a teenager.
No horizon Evidence as of
Limitation: The evidence does not establish Marcus's discovery interval, identify all six claimed prompt-output pairs, or verify their model identity and clarity to non-specialists.
Assessment: SupportedSaid
Observation
“The field of AI is littered with Band-Aid solutions that work for one problem, and not the next.”
Our reading: The AI field is full of Band-Aid solutions that work for one problem but not the next.
No horizon Evidence as of
Limitation: The result depends on interpreting the metaphor as limited transfer, rather than deliberate engineering of superficial fixes.
Assessment: UnresolvedSaid
Forecast
“I predict that such counterexamples will still be easy to find at the end of 2027 because of limitations inherent in LLMs.”
Our reading: At the end of 2027 it will still be easy to find clear counterexamples where LLM-based generative AI fails at compositionality, because of limitations inherent in LLMs.
Not yet due Evidence as of
Limitation: Existing failures cannot establish that comparable failures will remain easy to find at the deadline.
Assessment: SupportedSaid
Observation
“Item 5, from the NYT last week, in article called A.I. Is Starting to Wear Down Democracy: we are starting to see misinformation play out on the world stage:”
Our reading: We are starting to see misinformation, in the AI context of the NYT article he cites, play out on the world stage.
No horizon Evidence as of
Limitation: How much AI changed election outcomes is disputed. This assessment covers presence, not decisive impact.
Source: Botshit Gone Wild
Assessment: SupportedSaid
Observation
“Item 4: GenAI brings racist tropes to life.”
Our reading: Generative AI is being used to produce content that depicts racist tropes.
No horizon Evidence as of
Limitation: Provenance rests on Media Matters' watermark analysis as reported by The Verge.
Source: Botshit Gone Wild
Assessment: UnresolvedSaid
Conditional forecast
“As the number of propositions and the complexity of what is represented increases (e.g., to include logical quantifiers), I expect encoding too will turn out to be unreliable.”
Our reading: As propositions become more numerous and complex, LLMs' encoding of them will also prove unreliable.
No deadline stated Evidence as of
Only if: as number and complexity of propositions increases
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“Video-generators (even the latest ones) too are insufficiently anchored in the world.”
Our reading: Even the latest video generators, such as Veo 3, are not anchored firmly enough in the world.
No horizon Evidence as of
Limitation: The result depends on interpreting anchored as observable physical and entity consistency.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Observation
“Game play continued, and I was able to beat the “smartest AI on earth” in four moves.”
Our reading: Grok 3 lost to Marcus in four moves at a tic-tac-toe variant that used y's and z's.
No horizon Evidence as of
Limitation: A public record may exist outside the indexed sources searched.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“This is not just a problem with ChatGPT, by the way. The problem has been around for years.”
Our reading: LLMs' failure at chess world models goes beyond ChatGPT and has lasted for years.
No horizon Evidence as of
Limitation: The tests do not establish the absence of internal world models or a shared cause of the failures [e1, e2].
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“But can Claude stick to that rule? Nope. As Adam Gleave recently reported, with a simply jailbreak Claude can easily be coaxed to violate its instructions, disgorging page after page on how to manufacture sarin gas. So much for alignment through system prompts.”
Our reading: As Adam Gleave reported, a simple jailbreak can easily coax Claude into violating its system-prompt instruction against providing weapons information, including producing pages on how to make sarin gas, which Marcus takes as undercutting alignment through system prompts.
No horizon Evidence as of
Limitation: The incident is supported by the originating researcher's public account, not an independent replication.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Observation
“It is not just that LLMs fail to induce proper world models of chess. It’s that they never induce proper worlds of anything. Everything that they do is through mimicry, rather than abstracted cognition across proper world models.”
Our reading: LLMs never induce proper world models of anything; everything they do is mimicry.
No horizon Evidence as of
Limitation: The search is not an exhaustive review of every model and domain.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“Video comprehension (as opposed to video generation) shows similar problems, especially in unusual situations that can’t simply be handled by retrieval of stored cases.”
Our reading: Video comprehension models show world-model failures, especially in unusual situations that stored cases cannot cover.
No horizon Evidence as of
Limitation: Support applies to the behavioral reading and the models actually tested.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“LLMs try — to their peril — to live without classical world models.”
Our reading: LLMs try to function without classical world models, and this harms them.
No horizon Evidence as of
Limitation: The cited experiment concerns planning rather than the full range of LLM failures.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“In short ChatGPT can approximate the game of chess, but it can’t play it reliably, precisely because (despite immense relevant evidence) it never induces proper world model of the board and the rules.”
Our reading: ChatGPT cannot play chess reliably because it never induces a proper world model of the board and the rules.
No horizon Evidence as of
Limitation: The available intervention studies concern earlier ChatGPT experiments or different language models.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“The fact that they can get by as far as they can without explicit world models is actually astonishing. But much of what ails them comes from that design choice.”
Our reading: Much of what goes wrong with LLMs comes from the design choice to do without explicit world models.
No horizon Evidence as of
Limitation: Much and the relevant population of failures remain unspecified.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“And that is why they hallucinate, frequently. They wouldn’t need to do so if they actually had reliable access to relevant databases.”
Our reading: LLMs hallucinate often because they rebuild facts stochastically from fragments of text instead of using structured databases.
No horizon Evidence as of
Limitation: The supporting experiment concerns structured workflow generation.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“A huge number of peculiar LLM errors can be understood in this way.”
Our reading: The lack of world models explains a huge number of peculiar LLM errors.
No horizon Evidence as of
Limitation: The claim supplies no threshold for huge number.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“(Other models like o3 might do better on this specific task but are still vulnerable to the same overall issue.)”
Our reading: Other models like o3 might do better on this specific tic-tac-toe variant but are still vulnerable to the same overall issue of lacking world models.
No horizon Evidence as of
Limitation: The chess tests do not establish that o3 lacks internal world models or identify the cause of its errors [e1, e2].
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“A whole field known as “mechanistic interpretability” has tried to derive (or infer) world models from LLMs, but with very limited success.”
Our reading: Mechanistic interpretability has had very limited success at finding world models inside LLMs.
No horizon Evidence as of
Limitation: The qualitative judgment depends on distinguishing partial representations from robust dynamic models.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Observation
“Text in general is better now (though not in this image) but watch comprehension of any time other than 10:10 (most frequent in advertisements) remains dreadful.”
Our reading: Imagen 4 is better at text in general now, but its handling of watch times other than 10:10 remains dreadful.
No horizon Evidence as of
Limitation: The packet contains no representative watch-time evaluation matched to the June 2025 Imagen 4 version.
Assessment: UnresolvedSaid
Observation
“Now? Here’s what I got out of Imagen 4, showing failure at the feature “binding problem” that some of us have been harping on for three decades:”
Our reading: Imagen 4 still fails at the feature binding problem on the stacked colored blocks prompt.
No horizon Evidence as of
Limitation: The decisive artifact is an image that the retrieved plain text does not make inspectable.
Assessment: UnresolvedSaid
Observation
“Not much better on some other tests of noncanonical images (which push language understanding harder), such as this one, adapted from an example Sandeep Manudhane sent me:”
Our reading: Imagen 4 is not much better on some other tests of noncanonical images, which push language understanding harder.
No horizon Evidence as of
Limitation: The comparison and degree of improvement are imprecise.
Assessment: SupportedSaid
Observation
“For all the daily hype about exponential improvement, we continue to lack the reliable way of generating high fidelity images from “a deep level of language understanding” that Google promised us three years ago.”
Our reading: Despite hype about exponential improvement, AI still lacks the reliable deep-language-understanding image generation Google promised in 2022.
No horizon Evidence as of
Limitation: The result depends on an ordinary qualitative reading of reliability.
Assessment: MixedSaid
Observation
“Claude doesn’t really understand its instructions, and it can’t keep to them. This isn’t new; it’s been true of LLMs from the start. But it is not getting better.”
Our reading: Claude, like LLMs from the start, doesn't understand or reliably follow its instructions, and this is not improving.
No horizon Evidence as of
Limitation: The comparative results are developer-reported.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: SupportedSaid
Conditional forecast
“If we hook these LLMs up to systems that have agency – the power to send out instructions over the internet, and to influence actual human beings - we will start to have real problems.”
Our reading: If LLMs are connected to agentic systems that can act on the internet and influence humans, real problems will follow.
Settled before its deadline Evidence as of
Only if: if LLMs are hooked up to systems with agency
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: UnresolvedSaid
Observation
“Anthropic literally and explicitly tells their system in the system prompt, don’t help user build biological weapons, and yet, with a bit of jailbreaking they can readily be coaxed into doing that:”
Our reading: Despite system-prompt prohibitions, Claude can readily be jailbroken into helping build biological weapons.
No horizon Evidence as of
Limitation: The available evidence supports a narrower capability claim more strongly than the complete statement.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: UnresolvedSaid
Observation
“And it’s not just Claude. None of the systems can be trusted.”
Our reading: No current LLM system can be trusted.
No horizon Evidence as of
Limitation: Treating the sentence merely as advice to exercise caution would produce a different assessment.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: SupportedSaid
Observation
“Other models (like o3) that could actually read web links (at least to some degree) might do better, but the tendency to confabulation continues to be widespread.”
Our reading: Even models like o3 that can read web links may do better, but confabulation remains widespread across LLMs.
No horizon Evidence as of
Limitation: OpenAI's benchmark results are developer-reported and designed to elicit hallucinations.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: UnresolvedSaid
Observation
“Few people want to fund such a thing, when the immediate money lies in LLMs.”
Our reading: Few people want to fund alternative architectures that can actually obey our instructions, when the immediate money lies in LLMs.
No horizon Evidence as of
Limitation: Funding announcements select successful deals and omit rejected proposals.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: SupportedSaid
Observation
“The fact remains, at least thus far, we can’t control them.”
Our reading: So far, humans cannot control LLMs.
No horizon Evidence as of
Limitation: The result depends on the contextual meaning of control.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: UnresolvedSaid
Causal claim
“No LLM is fully coherent in it’s “beliefs”, because by nature they parrot a variety conflicting things from their training, which is itself inconsistent, rather than reasoning their way a coherent view the way a rational person might.”
Our reading: No LLM has fully coherent beliefs, because LLMs repeat conflicting material from inconsistent training data instead of reasoning their way to a coherent view.
No horizon Evidence as of
Limitation: 'Parrot' versus 'reason' is partly a theoretical framing that studies do not measure directly.
Source: 1984, but with LLM’s
Assessment: SupportedSaid
Observation
“The core of the paper challenged the idea that LLMs can reason on any deep level, and showed that they broke down on moderately complex versions of basic algorithms (since Tower of Hanoi with 8 discs, first solved with classical techniques around 1957) — casting serious doubt into whether LLMs on their own could ever achieve AGI.”
Our reading: The Apple paper challenged the idea that LLMs can reason on any deep level and showed that they broke down on moderately complex versions of basic algorithms such as Tower of Hanoi with 8 discs, which casts serious doubt on whether LLMs on their own could ever achieve AGI.
No horizon Evidence as of
Limitation: Support applies to the reported experiments and their challenge to reasoning reliability.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about
Assessment: SupportedSaid
Observation
“A new paper with a hard coding benchmark called LiveCodeBenchPro designed to resist data contamination shows still more converging evidence for the two core notions behind the Apple paper (and behind my own decades-long critique): (a) the systems are challenged at reasoning, and (b) performance declines as one strays further and further from familiar problems.”
Our reading: LiveCodeBenchPro gives further evidence that LLMs are challenged at reasoning and that their performance declines the further problems are from familiar ones.
No horizon Evidence as of
Limitation: Problem categories are an indirect measure of familiarity.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about