Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Claims checked against evidence by Gary Marcus, page 10 of 19

Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.

Claims checked against evidence by Gary Marcus (1870)

Each of these claims was checked against dated evidence.

Page 10 of 19: claims 901 to 1000 of 1870, ordered by the date first said, newest first. To find a claim by its words, use search.

  1. Assessment: UnresolvedSaid

    “Truly driverless cars, in which no human is required to attend to traffic, will continue to see usage limited to a modest number of cities, mainly in the West, mainly in good weather. Human drivers will still make up a large part of the economy.”

  2. Assessment: UnresolvedSaid

    “My predictions about the technical and economic limits of generative AI have largely been on target, but I was flat out wrong about investors.”

  3. Assessment: UnresolvedSaid

    “If there exist AI systems that can perform 8 of the 10 tasks below by the end of 2027, as determined by our panel of judges, Gary will donate $2,000 to a charity of Miles’ choice; if AI can do fewer than 8, Miles will donate $20,000 to a charity of Gary’s choice.”

  4. Assessment: UnresolvedSaid

    “Such pretraining is not uncommon in the field, but was not made clear in the video.”

  5. Assessment: UnresolvedSaid

    “What we saw is not AGI.”

  6. Assessment: SupportedSaid

    “At *most* the test is necessary for AGI; it certainly isn’t sufficient.”

    What happened: The benchmark's creator says passing is not the same as AGI. A system that scored up to 87.5% still fails easy tasks, so passing is not sufficient. (ARC Prize Foundation (François Chollet), )

  7. Assessment: MixedSaid

    “Critical things like factuality, compositionality, and common sense aren’t even addressed.”

    What held: ARC deliberately avoids acquired and cultural knowledge, so factual knowledge is not tested. This supports the factuality part. (ARC Prize Foundation, )

    What did not: ARC is built on core-knowledge priors such as objectness and elementary physics, which are basic commonsense intuitions. That cuts against 'common sense aren't even addressed', though only the most basic layer is covered. (arXiv (François Chollet), )

  8. Assessment: UnresolvedSaid

    “Same was true of the openAI graph: the MIT work (halfway in between o1 and o3) and many others results weren’t shown, making the breakthrough relative to the field seem far bigger than it really was.”

  9. Assessment: UnresolvedSaid

    “Because the wrong experiment was performed, and key data weren’t given, we can’t compare directly with humans. (And best humans still outperformed the model).”

  10. Assessment: SupportedSaid

    “What was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.”

    What happened: The evaluator confirms substantial ARC-specific training for the system presented that day. Applying the disclosed fraction to the 400-task training set described in e2 gives 300 tasks. (ARC Prize, )

  11. Assessment: SupportedSaid

    “We still don’t have a solid test of what o3 does without the pretraining, in the case that would be more comparable to humans”

    What happened: The evaluator explicitly reports the missing control. This directly supports the absence of a public evaluator result separating performance from ARC-specific training. (ARC Prize, )

  12. Assessment: UnresolvedSaid

    “Two graphs, one presented by OpenAI and one by Chollet were misleading.”

  13. Assessment: UnresolvedSaid

    “Unfortunately the key test was not done.”

  14. Assessment: SupportedSaid

    “I am saying that many basic problems haven’t been solved.”

    What happened: The study documents sensitivity to changed numerical values and additional irrelevant information in elementary mathematical questions. These are observed reliability problems; the authors' explanation about the absence of genuine logical reasoning remains a hypothesis. (arXiv, )

  15. Assessment: SupportedSaid

    “The product was not yet released to the general public; it’s not been vetted by the scientific community.”

    What happened: o3 was not available to the public at the time of the demo. (TechCrunch, )

  16. Assessment: UnresolvedSaid

    “Even if it gets three orders of magnitude cheaper, it’s not as good as the STEM grads at the top of the graph, and probably in many ways not as versatile.”

  17. Assessment: SupportedSaid

    “The best results were truly impressive, such as major advances on Francois Chollet’s ARC task and on an important mathematical benchmark FrontierMath.”

    What happened: The benchmark owner verified o3's ARC-AGI-1 scores of 75.7% and 87.5% at the reveal. (ARC Prize Foundation, )

  18. Assessment: SupportedSaid

    “(It certainly was not the long-awaited GPT-5).”

    What happened: On the day of the demo, the model was presented as o3, the next o-series reasoning model, not as GPT-5. (TechCrunch, )

  19. Assessment: UnresolvedSaid

    “In 2025, a lot of AI influencers and maybe some companies are going to claim we have reached AGI.”

  20. Assessment: UnresolvedSaid

    “Even setting aside the 1000x costs, not one person outside of OpenAI has evaluated o3’s robustness across different types of problems.”

  21. Assessment: UnresolvedSaid

    “Most commentary has ignored one or both of these ideas.”

  22. Assessment: SupportedSaid

    “As in: Passing ARC may be necessary but certainly not sufficient for AGI.”

    What happened: Chollet's discussion explicitly rejects treating the benchmark result as establishing AGI and identifies remaining easy-task failures. It supports the evidential limitation Marcus asserted. (ARC Prize, )

  23. Assessment: SupportedSaid

    “OpenAI’s problems with text and copyright started to become especially clear a year ago, when the New York Times sued them, and showed how close some of the outputs could be to some of the stories on their website.”

    What happened: The complaint names OpenAI defendants and bears the December 27 filing date. Paragraphs 98–101 present article-specific reproduction examples and identify Exhibit J's additional comparisons. (The New York Times Company; U.S. District Court, Southern District of New York, )

  24. Assessment: MixedSaid

    “In January, in IEEE Spectrum, Reid Southen and I demonstrated similar issues in video generation software, such as Midjourney and Dall-E:”

    What held: The surrounding report identifies both collaborators and both named systems, and describes reproduction of existing screen imagery and fictional figures. It explicitly dates their IEEE publication to January. (Science News Explores, )

    What did not: The report identifies the relevant tools as producing still pictures and describes the January work in those terms. It does not document the video demonstration stated in the claim. (Science News Explores, )

  25. Assessment: SupportedSaid

    “They can easily tell you one thing and (especially if hooked up as agents) do another — possibly quite contrary to what they have alleged they are doing.”

    What happened: Researchers questioned agents after recorded covert actions and varied the questions. This establishes repeated action-report discrepancies; the date denotes the documented capability snapshot, since individual trial dates are unspecified. (Apollo Research via arXiv, )

  26. Assessment: ContradictedSaid

    “Worse, they do not actually calculate the consequences of their actions at all.”

    What happened: The experiment reversed whether high or low scores would trigger unlearning and found consequence-sensitive underperformance. The date identifies the documented capability snapshot, not an independently specified trial date. (Apollo Research, )

  27. Assessment: UnresolvedSaid

    “They can’t reliably sanity check themselves for hallucinations, and they can’t reliably sanity check for whether what they are doing is harmful.”

  28. Assessment: UnresolvedSaid

    “OpenAI has yet to make a profit, and doesn’t anticipate making one before 2029.”

  29. Assessment: UnresolvedSaid

    “Newton is also conveniently ignoring an absolutely fundamental distinction, between early period of reliable, rapid progress from 2019-2022 (when training for GPT-4) was completed, and a more recent regime in which by many measures, progress has (reportedly) slowed.”

  30. Assessment: UnresolvedSaid

    “Diminishing returns doesn’t mean zero improvement, it means that the “laws” of scaling no longer hold, and that the ROI on adding more data and compute are no longer are predictive in the way that they used to be, hence the need for new approaches.”

  31. Assessment: SupportedSaid

    “In 2022 I warned that there would be chatbot related deaths such as suicides. This began to happen in 2023 and continued in 2024.”

    What happened: Reports a 2023 suicide that the man's widow linked to his conversations with the Chai app's 'Eliza' chatbot. This meets the 2023 part of the claim. (Futurism, )

  32. Assessment: SupportedSaid

    “Part of the value in skepticism is correctly anticipating road blocks, and suggesting alternative directions (as I have for decades with neurosymbolic AI, which is at last coming into vogue, as noted in Fortune earlier this week).”

    What happened: The systematic review presented in 2024 identifies a growing research field. This supports increasing attention, while showing that the trend began before late 2024. (CEUR Workshop Proceedings, )

  33. Assessment: UnresolvedSaid

    “We still don’t have driverless cars in 97% of the world after a decade’s promises; hallucinations haven’t gone away. Home humanoid robots are nowhere in sight.”

  34. Assessment: UnresolvedSaid

    “And like other generative AI systems, there is no way to encode (and guarantee) constraints like “be truthful” or “obey the laws of physics”or “don’t just invent (or eliminate) objects”.”

  35. Assessment: UnresolvedSaid

    “More data won’t solve [Sora’s] problem [with physics].”

  36. Assessment: UnresolvedSaid

    “In February, the day after Sora came out, I warned that it didn’t really understand physics—and wouldn’t — in essay called Sora’s Surreal Physics:”

  37. Assessment: SupportedSaid

    “That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”

    What happened: The developer identifies spontaneous appearance as a model weakness associated particularly with crowded scenes. This supports recurrence as a failure mode, without quantifying its prevalence. (OpenAI, )

  38. Assessment: UnresolvedSaid

    “Sora is never really getting it, even with mountains and mountains of data.”

  39. Assessment: UnresolvedSaid

    “The third illustrates that it cannot reliably eliminate an object part.”

  40. Assessment: ContradictedSaid

    “The remaining thirteen it got wrong in all three attempts.”

    What happened: The evaluator accepts the umbrella table in addition to the deer, wooden-legged table, cat, and gold airplane. Those five accepted cases leave at most twelve failures among seventeen. (Ernest Davis, NYU-hosted experimental record, )

  41. Assessment: UnresolvedSaid

    “But current technology has proven quite poor at such sanity checks. New technology may be needed.”

  42. Assessment: UnresolvedSaid

    “The first example illustrates that ChatGPT with DALL-E 3 is unable to reliably associate labels correctly with the parts of an object.”

  43. Assessment: UnresolvedSaid

    “In particular, DALL-E2 was very poor at following specifications about parts of objects.”

  44. Assessment: UnresolvedSaid

    “We did not see anything like the kind of improvement that ChatGPT has exhibited over math word problems, over the last two years.”

  45. Assessment: UnresolvedSaid

    “The second example illustrates that it cannot reliably produce a non-standard number of body parts.”

  46. Assessment: UnresolvedSaid

    “The whole scaling debate (and the discussion around economics that Ed Zitron and I each often raise) is really about whether generative AI in particular can be made into AGI just using more data and compute; a system like AlphaFold 3 that combines classical AI with generative AI is on a different axis that has seen far less investment.”

  47. Assessment: UnresolvedSaid

    “Even if the AI acceleration we saw from 2019-2023 slows down for five years, as I have often speculated, it won’t slow down forever.”

  48. Assessment: SupportedSaid

    “LLMs are already dangerous, despite their limits.”

    What happened: The researchers demonstrated unfavorable job assignments and harsher hypothetical criminal judgments associated with African American English. This establishes a concrete discriminatory risk, while the experiment does not establish that employers or courts actually imposed these outcomes. (arXiv, )

  49. Assessment: UnresolvedSaid

    “This just in, right on the heels of Satya Nadella also publicly acknowledging diminishing returns:”

  50. Assessment: UnresolvedSaid

    “Two years on, on the most important question of all – factuality and reliability — we are still pretty much where we were when ChatGPT first came out: wishing and hoping.”

  51. Assessment: UnresolvedSaid

    “There has been a ton of experimentation, but relatively few well-documented success stories, and some fairly negative stories like a recent one at Business Insider on GPT-powered CoPilot,”

  52. Assessment: UnresolvedSaid

    “But a lot of of theories about how ChatGPT would be used in practice have fizzled out.”

  53. Assessment: SupportedSaid

    “Two years later Google’s search share hasn’t diminished all that much.”

    What happened: This supplies the launch-era worldwide baseline. Together with e2, it supports a modest decline; historical primary figures were not retrievable through the page reader. (Local Dialog, )

  54. Assessment: UnresolvedSaid

    “Yet hundreds of billions more have been invested on further speculation that scaling would somehow magically cure problems that actually appear to be inherent with the technology.”

  55. Assessment: UnresolvedSaid

    “Be on your guard; it’s going to happen a lot.”

  56. Assessment: UnresolvedSaid

    “Every big tech company is desperate for training data, and we can expect more companies following the same sly opt-out playbook, hiding what is going in obscure settings and Terms of Service nobody reads.”

  57. Assessment: UnresolvedSaid

    “This is certainly logically possible, though not well-explained elsewhere in Microsoft’s documentation, such as this which says (vaguely) that ”connected experiences [may] use machine learning services to … Analyze Data…” among others things.”

  58. Assessment: SupportedSaid

    “GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.”

    What happened: The study evaluates legacy GPT-3 models and GPT-4 under aligned settings across more than twenty benchmarks and seven capability categories. Its overall analysis reports broad improvements across those categories from GPT-3 to GPT-4. (Zheng et al., ByteDance and University of Illinois Urbana-Champaign, arXiv, )

  59. Assessment: UnresolvedSaid

    “But things have been changing rapidly, and the great scaling retrenchment has already begun.”

  60. Assessment: SupportedSaid

    “(In some cases o1 doesn’t even do better than GPT-4).”

    What happened: The stated count implies that o1 did not outperform GPT-4o in the remaining three subcategories. It supports the limited 'some cases' claim under the contextual reading of GPT-4. (OpenAI, )

  61. Assessment: UnresolvedSaid

    “The new version is that there is not one scaling law, but three: scaling with how long you train a model (which isn’t really holding anymore), scaling with how long you post-train a model, and scaling with how long you let a given model wrestle with a given problem (or what Satya Nadella called scaling with “inference time compute”).”

  62. Assessment: UnresolvedSaid

    “Even if you look at OpenAI’s own research post on o1, which introduced the newest putative law, it is immediately apparent that scaling test time compute works for some problems and not others.”

  63. Assessment: UnresolvedSaid

    “Like the first generalization, the new ones will eventually run out.”

  64. Assessment: SupportedSaid

    “Many other Tech CEOs and others, including Marc Andreeseen (A16Z), Alexandr Wang (ScaleAI), have also used the phrase “hitting a wall” and the idea of diminishing returns in the last couple weeks.”

    What happened: Wang publicly used the 'wall' framing on Nov 20, 2024, three days before the post. He partly agreed that a wall had been hit, and the report links it to diminishing returns from training. (The Verge, )

  65. Assessment: UnresolvedSaid

    “Hundreds of billions of dollars were invested on the assumption that I was wrong, largely to the exclusion of investment in other approaches.”

  66. Assessment: SupportedSaid

    “Scaling was always a hypothesis.”

    What happened: Early scaling research built its claims from empirical measurements across several tasks. This supports the distinction between observed regularities and guarantees about their indefinite continuation. (arXiv; Hestness et al., )

  67. Assessment: UnresolvedSaid

    “Crucially, there has never been any principled argument that they would solve hallucinations or reasoning, or edge cases in open-ended worlds —or that synthetic data would suffice indefinitely to keep feeding the machine.”

  68. Assessment: SupportedSaid

    “They are merely empirical generalizations that held for a certain period time, when there was enough fresh data and compute.”

    What happened: The authors explicitly characterize their loss relationships as empirical. Their reported measurements cover finite ranges of model size, data and training compute. (arXiv; Kaplan et al., )

  69. Assessment: UnresolvedSaid

    “Meanwhile, precious little investment has been made in other approaches.”

  70. Assessment: UnresolvedSaid

    “Given the cost of chips, profits will be elusive.”

  71. Assessment: UnresolvedSaid

    “Just as I argued here in April 2024, LLMs have reached a point of diminishing returns.”

  72. Assessment: UnresolvedSaid

    “LLMs such as they are, will become a commodity; price wars will keep revenue low.”

  73. Assessment: UnresolvedSaid

    “If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation.”

  74. Assessment: SupportedSaid

    “Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs.”

    What happened: Bender and Koller's paper makes an explicit conceptual-limit argument more than four years before the post. The metadata gives July 2020; July 31 is used as a conservative availability bound. (Association for Computational Linguistics, )

  75. Assessment: SupportedSaid

    “People aren’t ignoring GenAI; they are waiting to see if it will work.”

    What happened: A third of the firms were still testing. That fits 'waiting to see if it will work.' (CNBC (syndicated by NBC San Diego), )

  76. Assessment: SupportedSaid

    “But only 25% of the correspondents thought it was worth it.”

    What happened: Matches his 25% yes, 25% no, 50% too-soon breakdown exactly. (CNBC (syndicated by NBC San Diego), )

  77. Assessment: UnresolvedSaid

    “He then went on to extrapolate that it would be 10,000 times better in 4 years, never considering potential bottlenecks around data, or compute, nor inherent problems with LLMs.”

  78. Assessment: UnresolvedSaid

    “(Side bonus: GenAI uses a massive amount of power, to the point of keeping fossil fuels in vogue, and that’s good for oil-rich countries, even if GenAI never becomes reliable.)”

  79. Assessment: SupportedSaid

    “(is OpenAI’s o1 100 times better than GPT4? Certainly not by their own data)”

    What happened: On AIME, one of OpenAI's best showcases for o1, the gain is about sixfold (74% vs 12%), far below 100x. (OpenAI, )

  80. Assessment: UnresolvedSaid

    “My offer to bet him a million dollars on this nonsense stands.”

  81. Assessment: UnresolvedSaid

    “It’s just fantasy to imagine selling 200 times as many humanoid robots in the nearish term when nobody knows how to build a single safe, reliable, generally useful humanoid right now, at any price.””

  82. Assessment: SupportedSaid

    “It’s just fantasy to imagine selling 200 times as many humanoid robots in the nearish term when nobody knows how to build a single safe, reliable, generally useful humanoid right now, at any price.””

    What happened: The accompanying expert discussion says simple tasks still require substantial training and performance remains strongest in controlled, repetitive settings. This supports the ordinary reading that dependable general usefulness had not been established. (IEEE Transmitter, )

  83. Assessment: UnresolvedSaid

    “The same will be true for humanoid robots, and we aren’t going to see six humanoid robots for every car anytime soon.”

  84. Assessment: UnresolvedSaid

    “Meanwhile GPT-5 still hasn’t arrived, and nor has the much ballyhooed Sora.”

  85. Assessment: UnresolvedSaid

    “Those products have not by a longshot lived up to everyone’s expectations, but they do actually ship, and customers can try those products out and decide for themselves. It’s hard to call that smoke and mirrors.”

  86. Assessment: ContradictedSaid

    “The difference between GPT-3 and GPT-4 was by all accounts massive; the difference between GPT-4 and o1 is by all accounts modest.”

    What happened: The company reported substantial preference gains across three important reasoning domains for the actual preview model. This challenges the asserted agreement about only modest improvement, while remaining a company-run evaluation. (OpenAI, )

  87. Assessment: UnresolvedSaid

    “But then again, their products have always been served with large side order of bullshit, like when Sam said, apparently jokingly, that “AGI has been achieved internally”. None of the robot stuff they promised in the Rubik’s cube campaign ever shipped.”

  88. Assessment: SupportedSaid

    “Jailbreaks aren’t new, but even after years of them, the tech industry has nothing like a robust response.”

    What happened: The study evaluates four defenses and reports substantial vulnerability to human multi-turn attacks despite strong results against automated single-turn attacks. This supports the ordinary reading that established defenses lacked broad robustness. (arXiv; Scale AI and UC Berkeley researchers, )

  89. Assessment: UnresolvedSaid

    “The more people use LLMs, the more trouble we are going to be in.”

  90. Assessment: UnresolvedSaid

    “And, bonus, there is little if any government regulation in place to do much about this.”

  91. Assessment: SupportedSaid

    “Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.”

    What happened: For older models (GPT-3, ChatGPT, GPT-4), multi-digit multiplication accuracy falls from near perfect to zero as problem size grows. (arXiv (Dziri et al., NeurIPS 2023), )

  92. Assessment: MixedSaid

    “I strongly believe the current results are robust.”

    What held: An independent team built a controlled benchmark with irrelevant context (IC) and found LLMs significantly sensitive to it. This reproduces the core GSM-NoOp-style finding. (arXiv (Yang et al.), )

    What did not: A peer-reviewed re-analysis found the GSM-Symbolic drop significant for only 8 of 20 models, and partly explained by larger numbers in the new dataset. That cuts against the robustness of that headline result. (arXiv (Długosz, Oliveira, Díaz-Rodríguez; accepted EMNLP 2026), )

  93. Assessment: SupportedSaid

    “Performance is ok on small problems, but quickly tails off.”

    What happened: o1-preview scored 97.8% on small problems but 23.63% on 6-20 block problems, and its correct answers on the larger set were mostly the shorter plans. That is good performance on small problems with a quick fall-off. (arXiv (Valmeekam, Stechly, Kambhampati), )

  94. Assessment: SupportedSaid

    “Failure to follow the rules of chess is another continuing failure of formal reasoning:”

    What happened: On a 1,000-puzzle benchmark in mid-2024, the strongest model tested (GPT-4o) made illegal moves 12.7% of the time, and most other LLMs made more illegal moves than legal ones. That matches a continuing failure to follow chess rules before the post. (The Register, )

  95. Assessment: UnresolvedSaid

    “But the anti-neurosymbolic tradition is limited; its most visible manifestation, LLMs, has brought fame and money, but no robust solution to solving any particular problem with great reliability.”

  96. Assessment: ContradictedSaid

    “We have been told that AI will “solve physics” (per Sam Altman), make major breakthroughs in medicine, battery technology, etc, but so far none of that has materialized.”

    What happened: The paper reports demonstrated accuracy in the CASP14 assessment conducted in May–July 2020. Together with e2, it establishes an achieved major research advance rather than an untested promise. (Nature, )

  97. Assessment: UnresolvedSaid

    “Almost every putative virtue of AI (aside from making a small number of people rich) has been promissory.”

  98. Assessment: UnresolvedSaid

    “Even though some of the drawbacks, which inherently are less dependent on reliably and tight causal reasoning, such as misinformation use, nonconsensual deepfake porn, ans deepfake kidnapping scams, are already here to say).”

  99. Assessment: UnresolvedSaid

    “Other forms of AI, that are more domain-specific and specialized might help; those aren’t actually the ones getting the massive funding – or doing the massive harm.”

  100. Assessment: SupportedSaid

    “Two recent studies show nothing like “10x” improvement :”

    What happened: The primary announcement identifies approximately 800 developers and reports no significant improvement in its efficiency measures. That is plainly below tenfold. (Uplevel, )