Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Claims checked against evidence by Gary Marcus, page 17 of 19

Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.

Claims checked against evidence by Gary Marcus (1870)

Each of these claims was checked against dated evidence.

Page 17 of 19: claims 1601 to 1700 of 1870, ordered by the date first said, newest first. To find a claim by its words, use search.

  1. Assessment: UnresolvedSaid

    “This would be very bad news, not just in terms of human and economic costs (which would mean more underpaid layers doing awful work) but also in terms of trustworthiness, because it would be mean that we have zero guarantee that any new iteration of a large language model is going to safe.”

  2. Assessment: UnresolvedSaid

    “Anyone who watched the last week unfold will realize that the new Bing has (or had1) a tendency to get really wild, from declaring a love that it didn’t really have to encouraging people to get divorced to blackmailing them to teaching people how to commit crimes, and so on.”

  3. Assessment: SupportedSaid

    “Seven years of AI has taught us that deep learning is unpredictable, and not always human like.”

    What happened: Humans and networks classified the same images differently. The experiments also identified differing reliance on shape and texture. (Geirhos et al., ICLR paper hosted on arXiv, )

  4. Assessment: SupportedSaid

    ““Adversarial attacks” like these have shown remarkable weakness, time and again, repeatedly establishing that what deep learning systems do just isn’t the same as what people do:”

    What happened: Distracting sentences sharply reduced model performance. Direct human testing showed substantially smaller losses, supporting a behavioral robustness difference rather than perfect human immunity. (Jia and Liang, EMNLP paper hosted on arXiv, )

  5. Assessment: SupportedSaid

    “And the match wasn’t even close; Pelrine beat a top AI system 14 games to 1, in a 15 match series.”

    What happened: The report identifies Pelrine as an amateur and confirms the score against a leading Go system. The researchers' paper corroborates the result. (Financial Times, republished by Ars Technica, )

  6. Assessment: SupportedSaid

    “People had been testing these Ais abilities at simple arithmetic for years before it occurred to anyone to check whether they can count. (They can’t, reliably).”

    What happened: Object-counting accuracy was 44.0%, 45.2%, and 51.2% with answer-only prompts for InstructGPT, Codex, and PaLM; step-by-step prompts raised these to 77.2%, 93.2%, and 83.2%. The Codex result exceeded the average human-rater score of 86.1%, materially limiting any claim of general counting incapacity. (Suzgun et al., arXiv, )

  7. Assessment: UnresolvedSaid

    “The problems in physical and psychological reasoning that we pointed in the Fall of 2020 very much remain.”

  8. Assessment: UnresolvedSaid

    “For one thing, what we publish often gets incorporated into the training; perhaps in company benchmarks as well.”

  9. Assessment: UnresolvedSaid

    “But In practice, that isn’t going to happen.”

  10. Assessment: SupportedSaid

    “The truth, though, is that GPT often fails in problems involving theory of mind.”

    What happened: Ullman reports failures across four unexpected-transfer variations and four unexpected-contents variations in the contemporary GPT-3.5 system. These establish recurring failures, although the selected examples do not estimate their prevalence across all possible tasks. (arXiv; Tomer D. Ullman, )

  11. Assessment: UnresolvedSaid

    “However, just in terms of percentage right vs. wrong, the improvement was slight; It now got 50% right, 40% wrong, and 10% borderline (marginally better than the 45% right before).”

  12. Assessment: SupportedSaid

    “AI and AI test sets have a huge English-language bias.”

    What happened: The developer reports that roughly 96% of its instruction dataset was classified as English, possibly more because of classifier errors. This directly supports English predominance in instruction tuning, without treating that percentage as the pretraining distribution. (Ouyang et al., OpenAI, arXiv, )

  13. Assessment: UnresolvedSaid

    “each reflect the fact that you cannot build AI in the real world from Big Data and deep learning alone.”

  14. Assessment: UnresolvedSaid

    “There was a broken image link, and I went so fast I neglected to mention the personally salient fact that Bing had lied about me, claiming that I believe that “Bing’s AI was more accurate and reliable than Bard”, when I never said any such thing.”

  15. Assessment: SupportedSaid

    “What I learned this morning is that retraining GPT-3 in some new political direction isn’t a half-million dollar proposition.”

    What happened: Rozado identifies the base model as GPT-3 Davinci and reports right-of-center responses on held-out political tests and open-ended questions. The completed experiment announced that day had reported computational expense far below Marcus's comparison figure. (David Rozado, )

  16. Assessment: UnresolvedSaid

    “Microsoft can (at least to a first approximation) do whatever the hell it wants with Bing.”

  17. Assessment: UnresolvedSaid

    “There are almost no regulations here; we are living in the Wild AI West.”

  18. Assessment: SupportedSaid

    “Oh, and for good measure the tooling for spreading automatically generated misinformation is getting more efficient, too:”

    What happened: The announcement describes a new integration passing generated responses directly into subsequent actions, including a WhatsApp round trip and conversion of form prompts into posts. This supports reduced manual work; applicability to misinformation is an inference about the same publishing capability. (Uncanny Owl / Uncanny Automator, )

  19. Assessment: SupportedSaid

    “As I have discussed in earlier essays, the jury is still out on how far Chat-style search can get, given its obvious current problems with truthiness.”

    What happened: The report reproduces Bard's erroneous attribution of the first exoplanet images to the James Webb Space Telescope and describes the correction. Reputable reporting is used because the original demonstration's answer is not present in the readable text of Google's announcement. (The Verge, )

  20. Assessment: ContradictedSaid

    “The only thing keeping it from being even more toxic and deceitful than it already is, is a system called Reinforcement Learning by Human Feedback, and “OpenAI” has been very closed about exactly how that really works.”

    What happened: OpenAI's confirmation describes a toxicity detector incorporated into ChatGPT and efforts to filter training material. These are protections beyond reinforcement learning; the report does not quantify their separate effects. (TIME, )

  21. Assessment: SupportedSaid

    “What ChatGPT delivered, in spades, that its predecessors like Microsoft Tay (released March 23, 2016, withdrawn March 24 for toxic behavior) and Meta’s Galactica (released November 16, 2022, withdrawn November 18) could not, was an illusion—a sense that the problem of toxic spew was finally coming under control.”

    What happened: The report describes a currently available method for bypassing restrictions on violent and illegal content. It also says the method does not always succeed, limiting any inference that safeguards were wholly ineffective. (Dark Reading, )

  22. Assessment: UnresolvedSaid

    “And, bonus, there is little if any government regulation in place to do much about this.”

  23. Assessment: SupportedSaid

    “All the theatre around its political correctness is masking a deeper reality: it (or other language models) can and will be used for dangerous things, including the production of misinformation at massive scale.”

    What happened: The investigation links this high-volume deceptive publishing network to LLM rewriting through exposed prompts and output artifacts. The count covers network articles, not 19,000 individually verified AI-generated falsehoods. (Recorded Future, Insikt Group, )

  24. Assessment: SupportedSaid

    “I first pointed out this risk in 2001, in the fifth chapter of my book The Algebraic Mind, and the problem has persisted ever since. To blithely assume that the problem will soon go away is to ignore 20 years of history.”

    What happened: Independent authors say chapter 5 of the 2001 book argued that distributed neural encodings lose track of particular individuals. That is a risk of blurring facts about individuals, consistent with Marcus's account. (arXiv, Chuma, Otsuka and Sato, )

  25. Assessment: UnresolvedSaid

    “I do actually think that these problems will eventually get ironed out, possibly after some important fresh discoveries are made, but whether they get ironed out soon is another matter entirely.”

  26. Assessment: UnresolvedSaid

    “Our new systems work better, but they are still superficial, fine for lab demos but often at a loss in the real world.”

  27. Assessment: UnresolvedSaid

    “Deep learning has undeniably made amazing progress, on other dimensions, but the fact is it hasn’t solved the core problems that many of us thought were essential 30 years ago.”

  28. Assessment: UnresolvedSaid

    “Melanie Mitchell’s book, Bender et al’s “Stochastic Parrots”; all of these more recent criticisms of deep learning help establish that some of the basic issues that were known long ago, with predecessors to current models, still remain even today.”

  29. Assessment: SupportedSaid

    “But a lot of things that a bunch of us worried about 30 years ago remain true.”

    What happened: Controlled changes to COGS exposed failures to transfer familiar linguistic components into new contexts in T5. This directly supports persistence of compositional generalization problems, rather than merely unfamiliar-word recognition problems. (arXiv; Najoung Kim, Tal Linzen and Paul Smolensky, )

  30. Assessment: SupportedSaid

    “The first one collapsed; this one has more staying power.”

    What happened: The account dates the retreat from the mid-1990s through 2007. It also records continued technical successes and institutional support, so it supports declining prominence rather than literal extinction. (MIT Press; Ian Goodfellow, Yoshua Bengio and Aaron Courville, )

  31. Assessment: SupportedSaid

    “Silly errors certainly do, and problems of abstraction are widespread in large language models.”

    What happened: Experiments on GPT-J and two GPT-Neo sizes find errors in arithmetic and strong dependence on training-term frequency. That supports elementary errors and fragile transfer, although frequency dependence alone does not prove an absence of reasoning. (arXiv; Yasaman Razeghi and colleagues, )

  32. Assessment: UnresolvedSaid

    “Not sure how much has changed in that regard, either, except that the hacks that were once written by graduate students are now farmed out to sweatshops, for a new form of handcrafting:”

  33. Assessment: UnresolvedSaid

    “Turns out you don’t need to solve them to make beautiful pictures, or decent if error-filled conversation, but anyone who thinks we can get to AGI without solving abstraction, compositionality, and the type-token distinction is kidding themselves.”

  34. Assessment: UnresolvedSaid

    “The measures are new, the model is new, but the stumbling block remains the same.”

  35. Assessment: SupportedSaid

    “Same story; only the architectures and training regimes have changed. The troubles with abstraction persist.”

    What happened: The study finds substantial frequency-associated gaps on addition and multiplication in the tested GPT-based models. This supports continuing difficulty with reliable arithmetic generalization, without establishing that all performance is memorization. (arXiv, )

  36. Assessment: UnresolvedSaid

    “And I still attribute those problems to a lack of world models that represent the ways in which things unfold over time. I still see this as the central flaw in large language models.”

  37. Assessment: UnresolvedSaid

    “But more than three years on, it’s still—even after a new update a few days ago focused around math—having lot of problems with easy word problems like this:”

  38. Assessment: UnresolvedSaid

    “I said the problem wasn’t math per se, but that large language models didn’t represent robust world models of how events fold over time; I stand by this 100% even today.”

  39. Assessment: SupportedSaid

    “Its output is more fluent, and lots of individual mistakes have been addressed.”

    What happened: Under matched prompting, evaluators distinguished GPT-2 from human text more accurately than GPT-3 from human text across stories, news, and recipes. This supports improved naturalness, although humanlikeness is an indirect fluency measure. (arXiv; Elizabeth Clark and colleagues, )

  40. Assessment: MixedSaid

    “Automation is a double-edged sword, and there is a kind of uncanny valley. We know perfectly well not to trust lousy systems; and wouldn’t need to pay attention to truly reliable systems, but the closer they get to perfect, the easier it is for mere mortals to space out.”

    What held: The experiment compared 50% and 90% reliability during simulated cockpit multitasking and found poorer monitoring with greater reliability. The availability date conservatively uses the university record's stated repository date. (Human Factors and Ergonomics Society; record hosted by University of Central Florida, )

    What did not: Sixteen participants supervised an autopilot under two reliability conditions lasting 45 minutes each. Reliability changed trust and perceived demand, but the study did not find the predicted increase in task-unrelated mind wandering, limiting the broad causal claim. (Frontiers in Human Neuroscience, )

  41. Assessment: SupportedSaid

    “Fact is, “Mack” Mackworth was right: humans can’t pay attention 100% of the time.”

    What happened: The reported experiment demonstrates a vigilance decrement during sustained monitoring. The same abstract reports that intermittent task switches prevented the decrement, limiting any claim that decline is unavoidable under every task design. (Cognition / Elsevier; author abstract indexed by PubMed, )

  42. Assessment: UnresolvedSaid

    “Anyone who thinks about how AI is used should know about it—especially as we are about to head into an era of ubiquitous yet fallible AI.”

  43. Assessment: UnresolvedSaid

    “Machines will just get more and more reliable and more general.”

  44. Assessment: UnresolvedSaid

    “So I don’t think we will waste more than 5 years before a substantial fraction of AI effort is looking outside the limited streetlight that currently are sucking up so much mental energy.”

  45. Assessment: SupportedSaid

    “ChatGPT is (at least in many ways) a far inferior intelligence, that, despite having some undeniable talents, cannot at all be trusted.”

    What happened: The developer itself said, at launch, that ChatGPT gives plausible but wrong answers. This supports the claim that its output could not be trusted without checking. (OpenAI, )

  46. Assessment: UnresolvedSaid

    “Neither one of us believes for a minute that large language models are remotely close to artificial general intelligence, and what’s more, we mostly agree about what’s missing: semantics, reasoning, common sense, theory of mind and so on.”

  47. Assessment: UnresolvedSaid

    “I think that in achieving AGI we will have no choice but to find ways of integrating whatever we can get out of deep learning with explicit knowledge and explicit cognitive models.”

  48. Assessment: SupportedSaid

    “we can also expect more and circles of fake web sites in order to sell advertisements”

    What happened: Three months after the post, an independent monitor documented AI-written content-farm sites run for ad revenue. This is the kind of site Marcus predicted. (NewsGuard, )

  49. Assessment: UnresolvedSaid

    “A decent fraction of the AI community has already recognized that the primary forte of large language models is in generating bullshit; many also see that the current path to autonomous vehicles is not making the progress that some expected.”

  50. Assessment: UnresolvedSaid

    “In a recent Wired article, I predicted that 2023 see the first death attributable to large language models, and I stand by that prediction.”

  51. Assessment: SupportedSaid

    “These characters (especially perhaps the latter two) might or might not be argued to be sentient, but it seems to be that they are flexible, resourceful, reliably (in a way that LLMs certainly aren’t), and adaptive.”

    What happened: The developer of the leading LLM admitted that it produces incorrect or nonsensical answers. (OpenAI, )

  52. Assessment: SupportedSaid

    “Already we have seen minor abuses; plagiarism, for example, will never be the same.”

    What happened: The professor reported a student's admission to using ChatGPT for the assignment in December 2022. This establishes a pre-post misuse example without treating the detector score as proof. (New York Post, )

  53. Assessment: UnresolvedSaid

    “The neurosymbolic AI community is growing fast; conferences that used to be dozens are now thousands.”

  54. Assessment: SupportedSaid

    “one of the things that popped out to me is that the system still quickly lands in the land of hallucination that has so haunted ChatGPT.”

    What happened: Goodside's test shows Claude confabulating plot details on an ordinary factual request, much as ChatGPT did. (Scale AI, )

  55. Assessment: SupportedSaid

    “A bunch of that work was done poorly paid labor in Kenya, paid less than $2/hour to evaluate (e.g.) graphic descriptions of sexual situations involving children and animals that I prefer not to describe it in detail.”

    What happened: Based on payslips, Kenyan labellers took home about $1.32–$2 per hour, which matches 'less than $2/hour'. (TIME, )

  56. Assessment: MixedSaid

    “CNET became the first casualty of the recently fashionable tendency to put too much faith in ChatGPT.”

    What held: Confirms that CNET's AI-written stories had many errors: more than half needed corrections. (The Verge, )

    What did not: CNET says the stories came from an in-house AI engine, not ChatGPT. This contradicts the claim that CNET over-trusted ChatGPT and posted ChatGPT-written stories. (CNET, )

  57. Assessment: UnresolvedSaid

    “ChatGPT simply cannot reason well enough to to do this; as I have documented in previous essays, it has demonstrably misunderstood physics, biology, psychology, medicine, and so forth.”

  58. Assessment: SupportedSaid

    “Statistical word prediction is no substitute for real math, but more than a few people have thought that it was.”

    What happened: Wolfram presents direct comparisons in which ChatGPT’s plausible mathematical answers fail checks against computational results. These support the narrow claim that unaided contemporary output could not be treated as dependable calculation. (Stephen Wolfram, )

  59. Assessment: UnresolvedSaid

    “It is not contributing genuine ideas, nor coming up with carefully controlled experimental designs, nor summing up prior literature in thoughtful ways.”

  60. Assessment: SupportedSaid

    “People are starting to treat ChatGPT as if it were a bona fide, well-credentialed scientific collaborator.”

    What happened: The paper lists ChatGPT and Zhavoronkov as authors and explicitly explains the decision to credit ChatGPT. The contribution statement confirms that the human author accepted its substantive arguments. (Oncoscience / Impact Journals, )

  61. Assessment: SupportedSaid

    “ChatGPT hasn’t done the conversion correctly (answer below), but the will to believe in this great new tool, full of precision (down to the second!), but lacking in reliability, is so strong that the author didn’t bother to check.”

    What happened: This contemporary transcription identifies the erroneous answer. Applying the NIST definitions gives 420.038784 seconds per mile, which rounds to 7:00 rather than 6:52. (Az, )

  62. Assessment: UnresolvedSaid

    “We are going to see a lot of this – misuses of ChatGPT in which people trust it, and even trumpet it—when it’s flat-out wrong.”

  63. Assessment: SupportedSaid

    “ChatGPT has proven itself to be both unreliable and untruthful. It makes boneheaded arithmetical errors, invents fake biographical details, bungles word problems, defines non-existent scientific phenomenon, stumble over arithmetic conversion, and on and on.”

    What happened: The dated transcript records a false elementary arithmetic assertion and inconsistent calculations. It also contains correct answers, so it establishes fallibility rather than universal failure. (Vipul Naik, LessWrong; GreaterWrong mirror, )

  64. Assessment: SupportedSaid

    “But there’s another side to Altman, too, the part that was recently forced to acknowledge that ChatGPT, for all its fame, really isn’t anything like AGI yet:”

    What happened: The paper identifies the December 15, 2022 ChatGPT version and reports 63.41% average accuracy across ten reasoning categories, inconsistent reasoning, and hallucinations. These results support the ordinary capability reading; the event date identifies the historical version rather than the later publication. (arXiv, Yejin Bang and colleagues, )

  65. Assessment: UnresolvedSaid

    “That’s a lot, and not the sort of thing that a rocketship startup would typically agree to.”

  66. Assessment: SupportedSaid

    “It could sometimes produce outputs that were correct and acceptable in these regards but not reliably.”

    What happened: OpenAI confirms that at launch the same question could get correct or failed answers depending on phrasing and retries. (OpenAI, )

  67. Assessment: SupportedSaid

    “It couldn’t reason about the physical world (figure 4).”

    What happened: Repeated questions about a thrown object's acceleration produced incorrect and inconsistent explanations alongside correct statements. December timing follows the manuscript's December receipt and the authors' retrospective description of experiments a few weeks after launch; the exact test day is unspecified. (IOP Publishing, Physics Education, )

  68. Assessment: UnresolvedSaid

    “For another, although OpenAI may be tracking errors like these for internal purposes, the scientific community has no access to whatever they have collected, which leaves it in the lurch.”

  69. Assessment: UnresolvedSaid

    “For one thing, although many people seem to have heard of a few of these errors (e.g. from reports in the media), few realize how pervasive or broad in scope they are; indeed a recent Twitter thread that claimed falsely that these problems had been resolved got a surprising amount of traction.”

  70. Assessment: UnresolvedSaid

    “ChatGPT made all the same kinds of mistakes that its predecessors did.”

  71. Assessment: SupportedSaid

    “It made things up (figure 6). Its output, when released, exhibited sexist and racist biases (figure 7).”

    What happened: OpenAI itself acknowledged at launch that ChatGPT produced plausible but false answers. (OpenAI, )

  72. Assessment: UnresolvedSaid

    “It is therefore impossible for external scientists to determine to what extent any specific correct answer merely reflects some very similar examples in the training data.”

  73. Assessment: UnresolvedSaid

    “ChatGPT couldn’t, for example, reliably count to four (figure 1) or do one-digit arithmetic in the context of simple word problem (figure 2).”

  74. Assessment: UnresolvedSaid

    “In our opinion, this was inevitable, in a system that has been trained to optimize the goal of producing one plausible-sounding word after another rather than actually engage with the meaning of language.”

  75. Assessment: UnresolvedSaid

    “(Spoiler alert: they’re not, at least not yet.)”

  76. Assessment: SupportedSaid

    “That’s a huge commitment, 3.5 hours, but one thing that I think all of our speakers could agree on is that artificial intelligence, in whatever form it currently is, is about to have a huge impact on society.”

    What happened: The representative survey reports generative-AI adoption among 39.4% of US adults aged 18–64, with 28.1% of employed respondents reporting work use. This establishes mass uptake within approximately twenty months of the post; the measure does not imply that all users used AI daily. (Federal Reserve Bank of St. Louis; Alexander Bick, Adam Blandin, and David Deming, )

  77. Assessment: UnresolvedSaid

    “But as they do so, as they glom stuff together, something often gets lost in translation: which bits of text do and do not truly belong together.”

  78. Assessment: SupportedSaid

    “Driverless cars have taken many years longer than we were originally told; outliers (AKA edge cases) have, at least so far, prevented them from making the transition from demo to widely-available reality.”

    What happened: The report documents missed rollout expectations, including Tesla's 2020 robotaxi promise, and explains how unexpected situations create stops requiring human help. This supports a material operational barrier to expansion, though it does not isolate edge cases from other causes. (Reuters, republished by Business Recorder, )

  79. Assessment: UnresolvedSaid

    “Once again, the result was fluent prose, and once again, hallucination ruled:”

  80. Assessment: UnresolvedSaid

    “As it happens, large language models are also wicked hard to update, typically requiring full retraining, sometimes taking weeks or months.”

  81. Assessment: UnresolvedSaid

    “Which means for example that ChatGPT, released in November, is so old it doesn’t know who owns Twitter:”

  82. Assessment: SupportedSaid

    “Large language models are not databases. They are glommers-together-of-bits-that don’t always belong together.”

    What happened: The technical description identifies generation as successive probabilistic predictions. This supports the distinction from retrieving explicit records; the database comparison is the evaluator's inference from the described mechanism. (BigScience collaboration / arXiv, )

  83. Assessment: SupportedSaid

    “Guardrails (a la ChatGPT) may be in place, but the guardrails will teeter between being too weak (beaten by “jailbreaks”) and too strong (rejecting some perfectly reasonable requests).”

    What happened: The launch disclosure describes GPT-4's safety training and interventions, establishing that safeguards were present. It also reports safety improvements, which limit any reading that the safeguards were wholly ineffective. (OpenAI, )

  84. Assessment: SupportedSaid

    “I guarantee that minds will be blown.”

    What happened: The announcement establishes that the release condition occurred. (OpenAI, )

  85. Assessment: UnresolvedSaid

    “Without external aids it won’t be able beat Meta’s Cicero in Diplomacy; it won’t be able to drive a car reliably; it won’t be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.”

  86. Assessment: SupportedSaid

    “It will often do what you want, sometimes not—and it will remain difficult to anticipate which in advance..”

    What happened: Using approximately 19,000 human generalization responses across 79 tasks, the researchers estimated advance beliefs and compared them with model correctness: GPT-4 performed best under equal error and success weights but worst under their highest-stakes weighting. This supports meaningful mismatches between expected and actual capability, while also demonstrating substantial success. (arXiv, )

  87. Assessment: UnresolvedSaid

    “GPT-4 will be trained on considerably more, a significant fraction of the internet as a whole.”

  88. Assessment: UnresolvedSaid

    “Its natural language output still won’t be something that one can reliably hook up to downstream programs; it won’t be something, for example, that you can simply and directly hook up to a database or virtual assistant, with predictable results.”

  89. Assessment: UnresolvedSaid

    “GPT-4 may be better at faking term papers, but if it follows the same playbook as it predecessors, it still won’t really understand the world, and seams will eventually show.”

  90. Assessment: UnresolvedSaid

    “GPT-4 will not have reliable models of the things that it talks about that are accessible to external programmers in a way that reliably feeds downstream processes.”

  91. Assessment: SupportedSaid

    “The system will still not be able to restrict its output to reliably following a shared set of human values around helpfulness, harmlessness, and truthfulness.”

    What happened: The report describes continuing factual and reasoning failures and recommends safeguards in consequential applications. It also reports a 19-percentage-point improvement over GPT-3.5 on internal adversarial factuality evaluations, which qualifies the finding without establishing dependable truthfulness. (OpenAI, via arXiv, )

  92. Assessment: UnresolvedSaid

    “As I told NYT columnist Farhad Manjoo, ChatGPT, like earlier, related systems is “still not reliable, still doesn’t understand the physical world, still doesn’t understand the psychological world and still hallucinates.””

  93. Assessment: UnresolvedSaid

    “In retrospectives written in 2043, intellectual historians will conclude that there was an initial overemphasis on large language models, and a gradual but critical shift of the pendulum back to more structured systems with deeper comprehension.”

  94. Assessment: UnresolvedSaid

    “Although GPT-4 will definitely seem smarter than its predecessors, its internal architecture remains problematic.”

  95. Assessment: SupportedSaid

    “GPT-4 will solve many of the individual specific items used in prior benchmarks, but still get tripped up, particularly in longer and more complex scenarios.”

    What happened: The report gives GPT-4 scores of 86.4% on MMLU and 95.3% on HellaSwag, establishing that it solved many established benchmark items. These particular results concern the pretrained base model and do not independently establish deployed-model reliability. (OpenAI via arXiv, )

  96. Assessment: SupportedSaid

    “As OpenAI has learned, bigger in many ways means better, with outputs more and more humanlike with each iteration. GPT-4 is going to be a monster.”

    What happened: The report records preference for GPT-4 on 70.2% of 5,214 prompts and broad benchmark gains over predecessors. These are developer-run evaluations supporting a substantial capability and response-quality advance, not proof of human understanding. (OpenAI / arXiv, )

  97. Assessment: SupportedSaid

    “It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”

    What happened: GPT-4 achieved 76% accuracy joining first letters but 3% joining second letters, and 80% sorting words alphabetically but 32% in reverse order. These controlled differences support recurring elementary mistakes and unexpectedly uneven performance, although the authors also successfully predicted broad patterns from task frequency. (arXiv, )

  98. Assessment: UnresolvedSaid

    “Which is to say it makes stuff up, regularly; a fair bit of what it is says simply isn’t true.”

  99. Assessment: SupportedSaid

    ““Alignment” between what humans want and what machines do will continue to be a critical, unsolved problem.”

    What happened: OpenAI acknowledges remaining harmful behavior and jailbreaks despite improved safeguards. Its report of substantial safety improvements qualifies the claim but does not show alignment was solved. (OpenAI, )

  100. Assessment: UnresolvedSaid

    ““Scaling” alone—building bigger and models until they absorb the entire internet — will prove useful, but only to a point.”