Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Claims checked against evidence by Gary Marcus, page 8 of 19

Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.

Claims checked against evidence by Gary Marcus (1870)

Each of these claims was checked against dated evidence.

Page 8 of 19: claims 701 to 800 of 1870, ordered by the date first said, newest first. To find a claim by its words, use search.

  1. Assessment: SupportedSaid

    “A new study on math, supporting what Davis and I wrote yesterday re LLMs struggling with mathematical reasoning from Mahdavi et al, converges on similar conclusions”

    What happened: The original abstract contains the conclusions Marcus attributes to the study. They align directly with concerns about invalid proofs and unreliable recognition of mathematical errors. (Mahdavi et al., arXiv, )

  2. Assessment: UnresolvedSaid

    “So Nestler’s experiment does not contradict the finding of the report; it corroborates it. o3, yet again, produced an invalid answer to this problem.”

  3. Assessment: SupportedSaid

    “The important problem is that they sometimes claim to they have solved problems that they have not properly solved; as with hallucinations, this reflects a deeply problematic failure to sanity check their own work.”

    What happened: The study pairs claims of completion with expert findings of flawed logic, unjustified steps and other proof defects. Its independent double-grading procedure supports the limited claim that models sometimes deliver incorrect work as solved. (arXiv, )

  4. Assessment: SupportedSaid

    “The results were dismal: None of the AIs scored higher than 5% overall.”

    What happened: Table 1 reports average totals of 2.0, 1.8, 1.5, 1.2, 1.2 and 0.9 points for the six named models, against a maximum of 42. The highest reported total therefore corresponds to approximately 4.76%, below 5%. (Petrov et al., ETH Zurich and INSAIT; arXiv, )

  5. Assessment: SupportedSaid

    “But that is because they rely in part on powerful, completely hand-written, symbolic reasoning systems. LLMs are not similarly immune.”

    What happened: The paper describes AlphaProof's use of Lean and attributes proof correctness to kernel verification. It also explains that informal LLM reasoning lacks guaranteed correctness; this supports the mechanism behind Marcus's comparison. (Nature, )

  6. Assessment: UnresolvedSaid

    “Importantly, the neurosymbolic method used by DeepMind’s AlphaProof and AlphaGeometry systems (which we discussed recently) which (more or less) achieved a silver-medal level performance on the 2024 International Math Olympiad, is immune to this problem.”

  7. Assessment: UnresolvedSaid

    “An LLM just spits back (sometimes in transmogrified form) what it has read.”

  8. Assessment: UnresolvedSaid

    “Humans are (usually, if well-motivated) smart enough to sort things like jokes and fiction from reality, but current AI doesn’t have a clue.”

  9. Assessment: SupportedSaid

    “In Taming Silicon Valley, I warned that LLMs would be used for dumb things that would affect lots of people.”

    What happened: This is primary textual evidence reproduced within a secondary publication. It establishes that the book raised concern about consequential, potentially biased uses of LLMs; it does not establish that those uses actually occurred. (Big Think, republishing an excerpt from The MIT Press book, )

  10. Assessment: UnresolvedSaid

    “I do genuinely believe today’s Turing Test takers are better AI systems than an earlier generation. I hardly think that means anything is “over”.”

  11. Assessment: UnresolvedSaid

    “But I am willing to wager that if someone put in that kind of effort and carefully built out the test, no current system would come close to human performance.”

  12. Assessment: SupportedSaid

    “To do well the system had to be coached to adopt the persona of “a young person who is introverted, knowledgeable about internet culture, and uses slang”; it’s still in no small part about the tricks.”

    What happened: The randomized comparison reports GPT-4.5 identification rates of 73% with the persona and 36% without it, with persona performance above chance in both populations. However, no-persona performance was not significantly below chance in the Prolific population, which limits any absolute claim of necessity. (arXiv; Cameron R. Jones and Benjamin K. Bergen, )

  13. Assessment: UnresolvedSaid

    “Unfortunately, nobody has ever fully implemented the comprehension challenge test that I proposed.”

  14. Assessment: SupportedSaid

    “But in an initial public offering yesterday CoreWeave valued themselves at just $23 billion - a major retreat.”

    What happened: Reuters directly reports both valuations on the fully diluted basis. The approximate reduction is 28.1%, which reasonably supports the description of a major retreat. (Reuters, syndicated by Yahoo Finance, )

  15. Assessment: UnresolvedSaid

    “Agents will be a disappointment, precisely because of all of the ongoing reliability and hallucination issues.”

  16. Assessment: SupportedSaid

    “Especially since DeepSeek there has been a massive price war.”

    What happened: Alibaba announced a large reduction within the target period. The maximum discount must not be read as a reduction across every model. (Reuters, syndicated by WHBL, )

  17. Assessment: UnresolvedSaid

    “For most GenAI model building companies, profits will be meager.”

  18. Assessment: UnresolvedSaid

    “This And the promised major jumps in economic productivity simply haven’t emerged.”

  19. Assessment: UnresolvedSaid

    “(Or nil; most are losing money; few have made any profits at all).”

  20. Assessment: UnresolvedSaid

    “The end game here is not that LLMs will vanish, but that LLMs will be virtually free.”

  21. Assessment: SupportedSaid

    “GenAI is an approach that is enormously popular, but one that is neither reliable nor particularly well-grounded in truth.”

    What happened: Company-reported adoption supports popularity. (TechCrunch, )

  22. Assessment: UnresolvedSaid

    “User disillusionment continues to grow.”

  23. Assessment: UnresolvedSaid

    “NVidia, the largest company actually making serious profits from AI, is down 17.75% for the year, and (after a meteoric rise) clearly facing headwinds.”

  24. Assessment: UnresolvedSaid

    “GenAI is never going to disappear.”

  25. Assessment: UnresolvedSaid

    “Thanks to a search tool provided by the Atlantic, I discovered tonight that in this one database alone Meta pirated over 100 of my books and scientific articles.”

  26. Assessment: SupportedSaid

    “Many other authors are discovering the same thing; social media is full of reports like mine.”

    What happened: Katsu explicitly places the start of repeated author discovery posts on Thursday, March 20. Her account also covers Friday; only its Thursday component bears on the target period. (Alma Katsu, )

  27. Assessment: SupportedSaid

    “He worked hard on every one of those books and articles and does not appreciate them being used without permission by AI systems with an established tendency towards regurgitation.”

    What happened: The experiments include LLaMA 7B and 65B and find repeated matching passages. For models with undisclosed training corpora, the researchers use an auxiliary corpus, which limits certainty about individual passages' training provenance. (Nasr and colleagues, arXiv, )

  28. Assessment: UnresolvedSaid

    “In every single case, they used them without my permission.”

  29. Assessment: UnresolvedSaid

    “For a man who allegedly believes human coders are basically done, Amodei sure has a lot of openings for coders.”

  30. Assessment: SupportedSaid

    “I don’t want spoil all the fun, but … things didn’t go well:”

    What happened: Supports. (The Guardian, )

  31. Assessment: SupportedSaid

    “To be sure, most or all trained coders will (if they are not already) soon use AI as a part of their workflow.”

    What happened: Daily use alone exceeds half of professional respondents. This avoids counting respondents who only plan to adopt AI. (Stack Overflow, )

  32. Assessment: UnresolvedSaid

    “AI will be a tool to help people write code, just as spell-checkers are a tool to help authors write articles and novel, but AI will not soon replace people who understand how to conceive of, write, and debug code.”

  33. Assessment: MixedSaid

    “As a fantastic, souped-up version of autocomplete, AI coding tools somewhat increase productivity.”

    What held: A randomized study of 96 Google engineers estimated about 21% less task time with completion, paste, and code-generation assistance. The task required passing tests, although the estimate had substantial uncertainty. (Google researchers / arXiv, )

    What did not: Sixteen experienced maintainers completed 246 randomized tasks more slowly when AI was allowed. The study covered February–June tools, primarily Cursor with Claude 3.5/3.7; it is adverse evidence for broad coding assistance, not an isolated test of autocomplete. (METR researchers / arXiv, )

  34. Assessment: UnresolvedSaid

    “And they don’t magically resolve the challenges of maintaining complex code bases over time.”

  35. Assessment: UnresolvedSaid

    “The idea that coders (and more generally, software architects) are on their way out is absurd.”

  36. Assessment: UnresolvedSaid

    “A few days ago he triumphantly claimed (without evidence) that nearly all code would be written by AI within the next year:”

  37. Assessment: SupportedSaid

    “Musk’s robotaxi promise hasn’t fared much better.”

    What happened: Tesla’s signed January disclosure still described launching the business and fully autonomous operation as future developments. This supports the inference that the million-robotaxi promise for 2020 remained substantially unfulfilled. (Tesla, Inc.; copy hosted by AnnualReports, )

  38. Assessment: UnresolvedSaid

    “That is not going to magically change next year, at a time when AI still can’t reliably solving river-crossing problems.”

  39. Assessment: UnresolvedSaid

    “That is not going to magically change next year, at a time when AI still can’t reliably solving river-crossing problems.”

  40. Assessment: SupportedSaid

    “Even in the best work, like Google’s new AI co-scientist, humans scientists are still choosing the research goals.”

    What happened: Google's launch description makes scientist-supplied objectives the system's input and describes experts selecting its evaluation goals. AI-generated hypotheses and research plans therefore operated within human-chosen objectives. (Google Research, )

  41. Assessment: UnresolvedSaid

    “Nobel-calibre science requires sound logic, reliable causal reasoning, and originality, and none of those are firmly in the grasp of current AI.”

  42. Assessment: SupportedSaid

    “Hallucinations haven’t solved, despite constant assurances.”

    What happened: Table 4 reports a PersonQA hallucination rate of 0.19 for GPT-4.5, compared with 0.30 for GPT-4o and 0.20 for o1. These results show improvement alongside substantial remaining failures; they do not measure all everyday usage. (OpenAI, )

  43. Assessment: UnresolvedSaid

    “• AGI should have a reliable ability to understand complex syntactic and semantic structure. (Failings here are also often reflected in quirky errors.”

  44. Assessment: UnresolvedSaid

    “Especially when we just saw GPT-5 (ultimately relabeled as GPT 4.5) fail despite more than two years of hard and expensive work.”

  45. Assessment: SupportedSaid

    “Hallucinations simply haven’t gone away, even in the latest models.”

    What happened: Table 4 reports PersonQA hallucination rates of 0.19 for GPT-4.5, 0.30 for GPT-4o, and 0.20 for o1. The release evaluation directly establishes persistence, while also showing improvement over the comparison models. (OpenAI, )

  46. Assessment: SupportedSaid

    “• AGI should have reliable ability to reason over abstractions. (Ongoing unpredictable errors in LLMs reflect this.)”

    What happened: MIR-Bench tests fifteen models on deriving outputs from examples of underlying functions. Its controlled evaluations find substantial remaining failures, including in o1-family models, supporting the limited reliability claim. (Kai Yan and colleagues, arXiv, )

  47. Assessment: UnresolvedSaid

    “But it is not just that I think Klein is wrong, or that he did a disservice to the other side of the argument, but that words like his are motivating a lot of dodgy bets being placed on the imminent AGI assumption — in ways that may not make sense.”

  48. Assessment: UnresolvedSaid

    “What that means is that a quarter century has passed without a principled solution to even one of those problems.”

  49. Assessment: UnresolvedSaid

    “(My view, as I explained in Taming Silicon Valley, is that industry has a vested interest in making people believe that AGI is imminent — how else to justify the lavish valuations - and that they have fooled a lot of people in government who aren’t technical experts into believing that AGI is closer than it really is).”

  50. Assessment: UnresolvedSaid

    “The fact that Grok 3 was 15x the size of Grok 2, and also generally viewed as disappointing.”

  51. Assessment: SupportedSaid

    “Most parts of the US still don’t have functioning driverless cars systems, for example, even after over a decade of lavish promises.”

    What happened: The article identifies existing Waymo service in Phoenix, San Francisco and Los Angeles, with Austin launching and other cities still planned. This supports geographically limited availability; national breadth is an inference from the deployment account. (Associated Press, syndicated by Tech Xplore, )

  52. Assessment: SupportedSaid

    “But it is absolutely absurd to pretend “AGI imminent” hypothesis isn’t getting play in high places.”

    What happened: OpenAI's leader publicly expresses confidence in a path to AGI, alongside discussion of near-term agents and further advances. This establishes high-level attention, without proving any particular arrival date. (Sam Altman, )

  53. Assessment: SupportedSaid

    “Every red cell in this graph (e.g, multiplying 20 digit numbers by 19 digit numbers2) is failure to generalize, even after immense amounts of training data:”

    What happened: The original experiment reports a clear limit to reliable multiplication as input length grows. The thread identifies o1-mini and reports similar accuracy for o1-preview, supporting the claim about the tested models. (Yuntian Deng, reproduced by Thread Reader, )

  54. Assessment: UnresolvedSaid

    “The fact that GPT 4.5 received very mixed reviews — despite years of labor and billions investment — and could in no reasonable way be considered a major (rather than minor, incremental) step forwards towards AGI.”

  55. Assessment: UnresolvedSaid

    “And these complex systems, built on the unreliable foundations of LLMs, are going to prove to be massively difficult to debug.”

  56. Assessment: UnresolvedSaid

    “But there’s also no reason to think that any of them are logically insolvable in principle. So we will get there someday.”

  57. Assessment: UnresolvedSaid

    “AI might replace average people in some tasks soon; replacing the best humans in a few years? Not going to happen.”

  58. Assessment: UnresolvedSaid

    “I can’t imagine that noncoders are really going to get those systems to work well enough for prime time (except perhaps in very very narrow use cases), let alone know what do when the back end internet calls break.”

  59. Assessment: UnresolvedSaid

    “They never let you know that a significant number of real world coders are deeply frustrated with automated coding tools like Cursor.AI (which is in part powered by Anthropic’s Claude):”

  60. Assessment: UnresolvedSaid

    “Nine years later and hundreds of startups later, my understanding is that not one radiologist has been replaced, and many parts of the world actually have a shortage of radiologists.”

  61. Assessment: UnresolvedSaid

    “People are increasingly going to discourage American children from learning to code based on naive enthusiasm from people like Roose.”

  62. Assessment: UnresolvedSaid

    “At least for another decade or two and probably more, big projects are always going to require software architects who know what they are doing.”

  63. Assessment: UnresolvedSaid

    “it’s often easy to get a solution that is 80% correct, and extremely difficult to get the last 20%. Driverless cars have been like that for decades.”

  64. Assessment: SupportedSaid

    “A new study of the latest system, GPT 4.5 showed that by on one standardized measure over 1/3 of the outputs are still hallucinations.”

    What happened: The report identifies a 37.1% hallucination rate for the February GPT-4.5 model on SimpleQA, exceeding one third. This is secondary reporting of OpenAI's result because the primary page's numeric chart values were absent from readable text. (Computerworld, )

  65. Assessment: SupportedSaid

    “Already, of course, DeepSeek has shown that our once strong lead is diminishing.”

    What happened: The historical comparison reports a US-China Chatbot Arena rating gap narrowing from 9.3% in January 2024 to 1.7% in February 2025. The provider chart identifies DeepSeek at 1,362, close to Google at 1,385 and OpenAI at 1,366, linking the narrowing to DeepSeek. (Stanford Institute for Human-Centered Artificial Intelligence, AI Index, )

  66. Assessment: SupportedSaid

    “Here Roose has displayed the cardinal sin of looking only at regurgitation rather than generalization. History has told us over and over that systems can be good at the former and still face enormous challenges at the latter.”

    What happened: StarCoderBase-15.5B achieved 72% accuracy on the most training-similar tenth of MBPP questions and 22% on the least similar tenth. The authors also investigated whether difficulty explained the association; these are familiarity comparisons, not proof of verbatim copying. (Martin Riddell, Ansong Ni and Arman Cohan, arXiv, )

  67. Assessment: SupportedSaid

    “Many many top people have left (Sutskever, Murati, Schulman, recently, Amodei and Karpathy before, etc and many, many more).”

    What happened: The individual accounts document eight senior departures during 2024, including Karpathy, Sutskever, Schulman, and Murati; the Leike section also identifies Dario Amodei as a former OpenAI employee. The article explicitly distinguishes Brockman's temporary leave, which is excluded. (Observer, )

  68. Assessment: SupportedSaid

    “Scaling hasn’t gotten them —or anyone else — to AGI.”

    What happened: The evaluator recognized a major advance but explicitly rejected treating the tested o3 as AGI. The discussion also identifies failures on tasks easy for humans. (ARC Prize, )

  69. Assessment: SupportedSaid

    “And of course, there is still no GPT-5, even after two years, of hype, bullshit, and cryptic hints.”

    What happened: OpenAI's CEO explicitly places GPT-5's public availability in the future shortly before the post. The roadmap distinguishes it from the impending GPT-4.5 release. (Sam Altman / OpenAI, )

  70. Assessment: SupportedSaid

    “Some have started serious competitors with similar IP.”

    What happened: The report identifies Dario and Daniela Amodei as Anthropic's co-founders and former OpenAI executives, and describes direct competition between the model families. (Reuters via Yahoo Finance, )

  71. Assessment: MixedSaid

    “Whatever lead OpenAI had two years ago has been squandered.”

    What held: The developer reports a competitor matching o1 on reasoning tasks, supporting erosion of an uncontested technical lead. This does not establish parity on every capability. (DeepSeek-AI, via arXiv, )

    What did not: The report identifies ChatGPT as the leading app in its mobile usage comparison, alongside monetization. Its usage evidence contradicts the broad claim that OpenAI had lost its product lead. (Andreessen Horowitz, using Sensor Tower data, )

  72. Assessment: UnresolvedSaid

    “DeepSeek led to a price war that cuts potential profits.”

  73. Assessment: UnresolvedSaid

    “(And Elon is right that they don’t have all or most of the Stargate funding in hand).”

  74. Assessment: UnresolvedSaid

    “Because OpenAI’s burn rate is so high (perhaps on the order of a couple billion dollars a month), they have limited runway, even after last year’s massive financing.”

  75. Assessment: SupportedSaid

    “GPT 4.5 is hugely expensive.”

    What happened: The same paragraph lists standard input at $75 and output at $150 per million tokens. Against the contemporary Anthropic prices in e2, those are premiums of 25 times for input and 10 times for output. (OpenAI, )

  76. Assessment: UnresolvedSaid

    “Even Altman himself was uncharacteristically tepid in his product announcement.”

  77. Assessment: SupportedSaid

    “So now GPT 4.5 has dropped, and — breaking news — I was right. Hallucinations didn’t disappear today, and nor did stupid errors.”

    What happened: The launch card reports a 0.19 PersonQA hallucination rate and 0.74 accuracy on unambiguous BBQ questions, where the answer is supplied in the prompt. These establish persistent hallucinations and straightforward answer-selection mistakes, although PersonQA improved over GPT-4o. (OpenAI, )

  78. Assessment: UnresolvedSaid

    “Gary Marcus was right all along that pure scaling would eventually hit a wall, or at least reach a point of massively diminishing returns.”

  79. Assessment: UnresolvedSaid

    “Right now, the math for profitability doesn’t work, influence on productivity of GenAI has been minimal, and prospects for pure scaling aren’t looking great.”

  80. Assessment: UnresolvedSaid

    “Right now, the math for profitability doesn’t work, influence on productivity of GenAI has been minimal, and prospects for pure scaling aren’t looking great.”

  81. Assessment: UnresolvedSaid

    “It’s not just OpenAI either; it’s the whole field, as I discussed in my own recent Fortune article:”

  82. Assessment: SupportedSaid

    “Actually, at any moment, we will finally get GPT 4.5, which most certainly wouldn’t live up to expectations around GPT-5 — hence its rather tentative name.”

    What happened: The release card reports a PersonQA hallucination rate of 0.19 for GPT-4.5. That is an improvement over the comparison models, but hallucinations clearly remain. (OpenAI, )

  83. Assessment: UnresolvedSaid

    “The truth is that scaling just isn’t producing the fruit that it once did.”

  84. Assessment: UnresolvedSaid

    “And for that matter people should be comparing GenAI revenue (on the order of low tens of billions per year, pre price wars) with GenAI expenditures (order half a trillion dollars to date), and doing the math, especially given the price wars induced by DeepSeek.”

  85. Assessment: SupportedSaid

    “In “Deep learning is hitting a wall”, I warned that problems with reasoning and hallucination wouldn’t go away simply as a function of scaling, and they haven’t.”

    What happened: The January launch system card, in its February file version, reports a 14.8% PersonQA hallucination rate for o3-mini. Its comparison models also have nonzero rates, despite improvements. (OpenAI, )

  86. Assessment: UnresolvedSaid

    “I personally noticed Musk’s shutting out of opposing views when he stopped answering my emails when I warned him in July – correctly - that throwing in with Trump would undermine the cause of AI safety (which for a decade, until a few weeks ago, seemed extremely important to Musk).”

  87. Assessment: SupportedSaid

    “Hallucinations haven’t been magically solved, etc.”

    What happened: Karpathy directly reports fabricated URLs during early-access testing of Grok 3's DeepSearch. Such fabrication establishes that hallucinations persisted in the launch product. (Andrej Karpathy, )

  88. Assessment: SupportedSaid

    “That wasn’t tested (presumably they don’t have access), and only a handful of benchmarks were actually reported.”

    What happened: This contemporaneous observation explicitly identifies the omission of full o3. It establishes the public chart's scope, not whether xAI privately tested o3 or had access. (Rex Asabor, OpenAI employee, )

  89. Assessment: SupportedSaid

    “That said, OpenAI’s moat keeps diminishing, so price wars will continue and profits will continue to be elusive for everyone except Nvidia.”

    What happened: The benchmark operator announced that early Grok 3 led its user-preference leaderboard. This supports the inference that another rival had become a credible capability alternative, although preference rankings do not measure every OpenAI advantage. (LMArena, reproduced by Thread Reader, )

  90. Assessment: SupportedSaid

    “That said, OpenAI’s moat keeps diminishing, so price wars will continue and profits will continue to be elusive for everyone except Nvidia.”

    What happened: Confirms a post-publication reduction on an existing OpenAI model. It does not independently establish the reason for the cut. (OpenAI, )

  91. Assessment: UnresolvedSaid

    “Spoiler alert: it wasn’t.”

  92. Assessment: SupportedSaid

    “Simply false, in view of the many basic aspects of geometry that AlphaGeometry2 does not cover, as discussed above.”

    What happened: The original report explicitly identifies classes of problems that cannot be expressed. Section 2 also lists three-dimensional geometry and nonlinear equations among its coverage gaps, supporting incomplete breadth rather than general mastery. (arXiv; Google DeepMind researchers, )

  93. Assessment: UnresolvedSaid

    “Taking this into account, the fraction of problems where a composite AI system could solve the problem starting with its English language statement is at most 60%.”

  94. Assessment: UnresolvedSaid

    “DeepMind didn’t make that claim, and it isn’t true.”

  95. Assessment: SupportedSaid

    “Unlike DeepMind’s AlphaZero system which learned to play multiple board games, or OpenAI’s ChatGPT, AlphaGeometry2 is nowhere close to being a single general-purpose learning architecture.”

    What happened: This retrospectively documents the system introduced in the February report: a symbolic geometry engine works with language-model proposals on formalized geometry inputs. That supports the ordinary reading of a specialized composite system. (Google DeepMind, )

  96. Assessment: UnresolvedSaid

    “Even if one were to glom together the best general AIs with AlphaGeometry2, the combination would fall far short of the geometric understanding and problem solving ability of high school math student with a talent for math.”

  97. Assessment: SupportedSaid

    “About year ago the City of New York tried this kind of thing, with less than spectacular results:”

    What happened: The reporters tested the operating city chatbot and documented incorrect guidance about housing vouchers, cash payments, and workers’ tips. Ten staff members received the same incorrect voucher answer, supporting repeated unreliability. (The Markup, )

  98. Assessment: UnresolvedSaid

    “Over the last couple days I have been asking version 2 to draw images identifying and tagging parts of images. The results are consistently awful:”

  99. Assessment: UnresolvedSaid

    “And it will be the citizens of the United States who bear the consequences, while the guy who stood to get a $400M armored T̶e̶s̶l̶a̶ EV government contract (until the media squawked) stands to collect a big slice of the profits, unemploying thousands of lifelong government employees.”

  100. Assessment: UnresolvedSaid

    “Yet Musk wants to use shove problematic AI, both biased and unreliable, down our throats, immediately, as reported by The New York Times, and noted in this Bluesky comment:”