Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI

We recorded 47 claims from this post. 21 were checked against evidence. 26 can’t be checked. 9 more are held. Read the post at its source.

Claims from this post (47)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: UnresolvedSaid

    “Machine learning, the branch of AI concerned with tuning algorithms from data, is an amazing field that has changed the world — and will continue doing so.”

    What we tested: Machine learning has changed the world and will continue to do so.

  2. Assessment: Can’t be checkedSaid

    “In my view, neither neural networks nor classical AI can really stand on their own.”

    What we tested: In Marcus's view, neither neural networks nor classical symbolic AI can really stand on their own.

  3. Assessment: Can’t be checkedSaid

    “We must find ways to bring them together.”

    What we tested: The field must find ways to combine neural networks and symbolic AI.

  4. Assessment: Can’t be checkedSaid

    “After a thirty-year journey, I believe that neurosymbolic AI’s moment has finally arrived, in part from an unlikely place.”

    What we tested: Neurosymbolic AI's moment has arrived as of mid-2025.

  5. Assessment: SupportedSaid

    “For all the efforts that OpenAI and other leaders of deep learning, such as Geoffrey Hinton and Yann LeCun, have put into running neurosymbolic AI, and me personally, down over the last decade, the cutting edge is finally, if quietly and without public acknowledgement, tilting towards neurosymbolic AI.”

    What we tested: The cutting edge of AI is finally tilting towards neurosymbolic AI, quietly and without public acknowledgement.

    What happened: The surrounding technical description identifies Python execution and newly combined tool use within reasoning, and describes training the models to choose tools. It presents this as reinforcement learning and agentic tool use, without calling it neurosymbolic AI. (OpenAI, )

  6. Assessment: UnresolvedSaid

    “And it is about why, in 2025, neurosymbolic AI has emerged as the team to beat.”

    What we tested: In 2025 neurosymbolic AI has emerged as the leading approach in AI.

  7. Assessment: MixedSaid

    “Neural networks are good at learning but weak at generalization; symbolic systems are good at generalization, but not at learning.”

    What we tested: Neural networks learn well but generalize poorly, while symbolic systems generalize well but learn poorly.

    What held: The experiments support the neural-generalization weakness in the contextual sense. They concern tested recurrent networks and task splits, not every neural architecture. (ICML; author manuscript on arXiv, )

    What did not: The study reports systematic generalization from a standard neural architecture trained through meta-learning, including comparisons with symbolic models. It materially qualifies the broad neural-versus-symbolic division without proving unrestricted neural generalization. (Nature, )

  8. Assessment: SupportedSaid

    “A quarter century later, hallucinations remain ubiquitous. Within a neural network substrate, the problem still has not been solved.”

    What we tested: Hallucinations remain ubiquitous and unsolved in neural-network-based systems.

    What happened: The release evaluation reports SimpleQA hallucination rates of 51%, 79%, and 44% for o3, o4-mini, and o1, respectively; PersonQA rates are 33%, 48%, and 16%. These are substantial benchmark failure rates across several models, not estimates of everyday usage. (OpenAI, )

  9. Assessment: Can’t be checkedSaid

    “Hinton’s long-time hostility against any role at all for symbols has, in my judgement, cost the field dearly.”

    What we tested: Hinton's hostility to symbolic approaches has substantially harmed AI research.

  10. Assessment: Can’t be checkedSaid

    “Many other important ideas have likely also yet to be discovered, precisely because the Hinton path has distracted immense resources from other ideas, fostering an intellectual monoculture that, in the words of Emily Bender, has been “sucking the oxygen from the room.””

    What we tested: The dominance of the pure neural-network path has likely kept important ideas undiscovered by diverting resources into a monoculture.

  11. Assessment: Can’t be checkedSaid

    “AlphaFold, AlphaProof, and AlphaGeometry are all successful neurosymbolic models. We would likely have more of such innovation already, if Hinton had not so insistently shaped the modern landscape in such a narrow-minded way.”

    What we tested: AlphaFold, AlphaProof and AlphaGeometry are successful neurosymbolic models, and there would likely be more such innovation had Hinton not narrowed the field.

  12. Assessment: Can’t be checkedSaid

    “We are drowning in a sea of mediocre prose precisely because LLMs, the backbone of virtually all current sytems, lack the underlying representational wherewithal to do better.”

    What we tested: LLMs produce mediocre prose because they lack the representational capacity to do better.

  13. Assessment: UnresolvedSaid

    “Yet nearly all funding has been aimed simply at making them larger.”

    What we tested: Nearly all AI funding has gone to making LLMs larger.

  14. Assessment: SupportedSaid

    “As we will see in a moment, people are finally pursuing a tool-driven, neurosymbolic vision like this — and getting better results than they were able to get with pure neural networks.”

    What we tested: Tool-driven neurosymbolic approaches are now being pursued and yield better results than pure neural networks.

    What happened: The study compares tool-integrated reinforcement learning with a baseline without tools using Qwen2.5-Math base models. It reports 43.3% AIME24 accuracy for ToRL-7B and an advantage over the tool-free baseline, supporting gains from the combined approach on mathematical tasks. (Xuefeng Li, Haoyang Zou, and Pengfei Liu / arXiv, )

  15. Assessment: UnresolvedSaid

    “I argued that scaling as it was then defined (pretraining data and compute) alone was insufficient to solve challenges with reasoning, misalignment and hallucinations. (Spoiler alert: it still hasn’t.)”

    What we tested: Scaling pretraining data and compute alone has not solved reasoning, misalignment and hallucination problems.

  16. Assessment: SupportedSaid

    “Three years later, the pure scaling of pretraining data (and compute, which is to say more and more GPUs) simply hasn’t worked. The specific obstacles that I dwelled on, with respect to hallucinations, misalignment and reasoning errors, have not been overcome.”

    What we tested: Pure scaling of pretraining data and compute has not worked; hallucinations, misalignment and reasoning errors remain.

    What happened: Table 4 reports a PersonQA hallucination rate of 0.19 for GPT-4.5, versus 0.30 for GPT-4o. This documents improvement alongside a substantial remaining failure rate. (OpenAI, )

  17. Assessment: UnresolvedSaid

    “OpenAI quietly began doing this to some degree in 2023 with a system called “code interpreter”, in which LLMs (themselves neural networks) call (purely symbolic) Python interpreters. This is literally “putting programs inside”; exactly what Hinton said was a huge mistake.”

    What we tested: In 2023, OpenAI quietly began, to some degree, 'putting programs inside' its systems with Code Interpreter, in which LLMs call purely symbolic Python interpreters. Marcus says this is exactly what Hinton called a huge mistake.

  18. Assessment: SupportedSaid

    “Recent “reasoning” models borrow even more from classic symbolic approaches, such as techniques like search and conditionally iterating through multiple solutions, and aggregating the results, techniques that traditional networks had long eschewed, in favor of a single feed-forward pass through a neural net.”

    What we tested: Recent reasoning models borrow symbolic techniques such as search, iteration over multiple solutions and aggregation.

    What happened: OpenAI describes conditional strategy changes in o1 and reports AIME evaluations using consensus among 64 samples and reranking 1,000 samples. These document revision and aggregation, without establishing that aggregation was the default chat configuration. (OpenAI, )

  19. Assessment: Can’t be checkedSaid

    “GPT-4 probably wasn’t (I suspect it included some symbolic filters in its guardrails), and o3, when it invokes code interpreter and symbolically-executed control structures certainly isn’t.”

    What we tested: GPT-4 was probably not a pure LLM, and o3 using code interpreter is certainly not a pure LLM.

  20. Assessment: SupportedSaid

    “Both informal experimentation and a number of recent quantitative results make it abundantly clear that current generative AI systems perform better when they avail themselves of symbolic tools.”

    What we tested: Current generative AI systems perform better when they use symbolic tools.

    What happened: OpenAI reports AIME 2025 Python-enabled pass@1 scores of 99.5% for o4-mini and 98.4% for o3, explicitly describing tool-use improvements. These are developer-reported results, and the page cautions against treating tool-enabled scores as unaided-model performance. (OpenAI, )

  21. Assessment: UnresolvedSaid

    “In contrast, o3, when it draws on symbolic code (which it sometimes displays explicitly) can build crossword grids like this far faster than I could (short of my writing my own code).”

    What we tested: When o3 draws on symbolic code, it can build crossword grids like this one far faster than Marcus could, short of writing his own code.

  22. Assessment: SupportedSaid

    “Second, adding symbolic tools (orange dots) dramatically improves performance.”

    What we tested: Adding symbolic tools dramatically improves Grok 4's performance on Humanity's Last Exam.

    What happened: The paragraph reports ordinary Grok 4 scoring 25.4% without tools and 38.6% with tools, a 13.2 percentage-point increase. It separately identifies Grok 4 Heavy, so the comparison does not substitute the multi-agent model. (Scientific American, )

  23. Assessment: UnresolvedSaid

    “Between the strong success of DeepMind’s series of neurosymbolic hybrids (AlphaGo, AlphaFold, AlphaProof, AlphaGeometry, etc.) and the more recent results in which LLMs have been enhanced with python and other tools to dramatic effect, we can safely conclude that neurosymbolic AI is on track to be a major part of AI’s future.”

    What we tested: Neurosymbolic AI is on track to be a major part of AI's future.

  24. Assessment: Can’t be checkedSaid

    “None of which means we are close to AGI. There are many ways to put together symbols and neural nets. I am not at all convinced we have the right one.”

    What we tested: These results do not mean we are close to AGI, and Marcus is not at all convinced the field has found the right way to combine symbols and neural nets.

  25. Assessment: Can’t be checkedSaid

    “Including LLMs somewhere in the next evolution of AI makes sense to me, but leaving them at the core may be a mistake.”

    What we tested: LLMs may belong in future AI systems, but keeping them at the core may be a mistake.

  26. Assessment: Can’t be checkedSaid

    “A neurosymbolic substrate for AI by itself (as I argued in 2020) is likely to be necessary but not sufficient.”

    What we tested: A neurosymbolic substrate is likely necessary but not sufficient for AGI.

  27. Assessment: Can’t be checkedSaid

    “There is likely still a great deal of work left to be done.”

    What we tested: There is likely still a great deal of work left to be done.

  28. Assessment: SupportedSaid

    “Models like o3 are far better than non-code-interpreter systems on some tasks but not all (for now o3 is actually worse than o1 on hallucinations) and effective for some tasks, but not others.”

    What we tested: Models like o3 are far better than non-code-interpreter systems on some tasks but not all, and for now o3 is worse than o1 on hallucinations.

    What happened: Table 4 reports hallucination rates of 0.33 for o3 versus 0.16 for o1 on PersonQA, and 0.51 versus 0.44 on SimpleQA. It also reports higher o3 accuracy on both tests, qualifying any broader inference that o3 is simply worse at factual answering. (OpenAI, )

  29. Assessment: Can’t be checkedSaid

    “We still also (just as I argued in 2020) lack proper ways of building and inferring symbolic world/cognitive models on-the-fly.”

    What we tested: AI still lacks proper ways to build and infer symbolic world models on the fly.

  30. Assessment: Can’t be checkedSaid

    “Explicit cognitive models (as I discussed recently in this newsletter) are also likely to be essential, yet remain very much underexplored.”

    What we tested: Explicit cognitive models are also likely to be essential, yet they remain very much underexplored.

  31. Assessment: Can’t be checkedSaid

    “We have made some progress; we still likely have a long way to go.”

    What we tested: We have made some progress toward AGI, but we likely still have a long way to go.

  32. Assessment: SupportedSaid

    “Systems that draw on code interpreter, and hence by definition are neurosymbolic, often outperform those that lack symbolic tools, at least in a bunch of tasks. This is confirmation of the value of neurosymbolic AI.”

    What we tested: Systems using code interpreter often outperform those without symbolic tools on many tasks, confirming the value of neurosymbolic AI.

    What happened: The reported evaluation covers 144 tasks, including 37 held-out task types. On those held-out tasks, GPT-4o scores 58.6% with text-only reasoning and 70.9% with Code Interpreter; the trained R1-CI systems also improve across model sizes. (Chen et al., arXiv, )

  33. Assessment: Can’t be checkedSaid

    “Largely because of Hinton’s influence, the field still talks about neurosymbolic AI as if it were a dirty word. That’s likely holding back the science.”

    What we tested: The stigma against neurosymbolic AI, largely due to Hinton, is likely holding back AI science.

  34. Assessment: Can’t be checkedSaid

    “Hinton’s admonition against using symbolic AI was extraordinarily misguided, and probably delayed discovery of useful tools like Code Interpreter by years.”

    What we tested: Hinton's opposition to symbolic AI was misguided and probably delayed tools like Code Interpreter by years.

  35. Assessment: Can’t be checkedSaid

    “It’s likely that OpenAI, Anthropic and others are already doing more neurosymbolic AI than they actually make public. They still pretend it’s all “a model” but that model uses all kinds of tricks and mixtures with tools, code generation etc.”

    What we tested: OpenAI, Anthropic and others are likely using more neurosymbolic techniques than they disclose.

  36. Assessment: Can’t be checkedSaid

    “Massive infrastructure investments like Stargate are likely based on misleading impressions of what is actually driving progress.”

    What we tested: Massive AI infrastructure investments like Stargate are likely based on a mistaken view of what drives AI progress.

  37. Assessment: Can’t be checkedSaid

    “A lot of recent improvements are likely coming from improvements in using symbolic tools, rather than scaling.”

    What we tested: Much recent AI improvement likely comes from better use of symbolic tools rather than scaling.

  38. Assessment: Can’t be checkedSaid

    “It would appear that it is only recently, in desperation, as scaling started to reach diminishing returns, that mainstream machine learning began to broaden its vistas.”

    What we tested: It would appear that mainstream machine learning only recently began to broaden its vistas, in desperation, as scaling started to reach diminishing returns.

  39. Assessment: Can’t be checkedSaid

    “Where the field is getting stuck now is likely on how to scale these new, neurosymbolic tools and how to make models “understand” when to use what and how to logically combine the context.”

    What we tested: The field is likely stuck on scaling neurosymbolic tools and getting models to know when and how to use them.

  40. Assessment: ContradictedSaid

    “This is why so called “code agents” inevitably fall apart as soon as one gives them a more complex task which requires reasoning.”

    What we tested: Code agents inevitably fail on more complex tasks requiring reasoning.

    What happened: The documented agent edits files and executes commands to resolve real repository issues, with released methods and assets. The accompanying analysis reports multistep successful trajectories and nonzero success even on tasks rated hard, contradicting unavoidable failure under the stated reading. (Yang et al., arXiv, )

  41. Assessment: UnresolvedSaid

    “Very little of what has been tried has been discussed explicitly, and because the companies are often quite closed about what they are doing, the public science of neurosymbolic AI is greatly impoverished.”

    What we tested: Company secrecy has greatly impoverished public science on neurosymbolic AI.

  42. Assessment: Can’t be checkedSaid

    “Getting to AGI will likely take still more breakthroughs.”

    What we tested: Reaching AGI will likely require further breakthroughs.

  43. Assessment: Can’t be checkedSaid

    “But it may well be as simple as money. The message that we can simply scale our way to AGI is incredibly attractive to investors because it puts money as the central (and sufficient) force needed to advance.”

    What we tested: The industry's connectionist-only focus may be explained by money, because the scaling narrative appeals to investors.

  44. Assessment: Can’t be checkedSaid

    “Admitting that they need to rely on neurosymbolic tools would pierce the scaling narrative.”

    What we tested: If AI companies admitted that they need to rely on neurosymbolic tools, that would pierce the scaling narrative.

  45. Assessment: SupportedSaid

    “So here’s where we are: pure LLMs can’t reliably apply algorithms (vindicating the line of argument I first developed in the 1990s); if you enhance them with symbolic processes—yielding neurosymbolic systems—they often give better results.”

    What we tested: Pure LLMs cannot reliably apply algorithms, while LLMs enhanced with symbolic processes often perform better.

    What happened: Providing the Tower of Hanoi algorithm did not remove the tested models' execution failures. This supports a practical reliability limitation, although other parts of the puzzle study face methodological objections. (Apple researchers, arXiv, )

  46. Assessment: UnresolvedSaid

    “OpenAI, without any sort of of public acknowledgement whatsoever, has accidentally vindicated neurosymbolic AI.”

    What we tested: Without any public acknowledgement whatsoever, OpenAI has accidentally vindicated neurosymbolic AI.

  47. Assessment: Can’t be checkedSaid

    “Fostering its further development may be among the best things that companies, researchers, and governments can do. Investors, take note.”

    What we tested: Fostering the further development of neurosymbolic AI may be among the best things that companies, researchers, and governments can do, and investors should take note.

Held from this post (9)

These records did not pass every check, so nothing about them is published. Held is not a finding.