Claims checked against evidence, page 27 of 31
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (3100)
Each of these claims was checked against dated evidence.
Page 27 of 31: claims 2601 to 2700 of 3100, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
“Breaking news: the draft EU AI Act—the closest thing we have to AI regulation with real teeth—is in serious jeopardy, potentially a direct victim of regulatory capture, governments doing the bidding of those they are supposed govern.”
What happened: The report describes Parliament representatives ending the preceding Friday's meeting early over foundation-model rules; adviser Kai Zenner retrospectively put the prospects of passage at only 0–10% when they left. His improved assessment and the rapporteur's optimism on 15 November concern subsequent developments. (IAPP, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Update: A few hours after I posted this Cruise CEO Kyle Vogt essentially confirmed that their “driverless” cars need very regular human intervention:”
What happened: CNBC attributes this acknowledgment to Vogt’s Sunday statement and reports verifying it with Cruise. It supports recurring remote assistance, although the percentage is not a count of human interventions. (CNBC, )
Assessment: SupportedSaid
“Shareholders will certainly sue, and if it’s bad as it looks, I doubt that GM will continue the project, which was recently suspended.”
What happened: Edward Shamoon filed this securities class action against GM and executives after the post. The filing stamp establishes the date; the complaint and subsequent order establish the Cruise-related subject matter. (U.S. District Court, Eastern District of Michigan; complaint copy hosted by The D&O Diary, )
Assessment: ContradictedSaid
“I realize there’s been almost zero public disclosure of how any of the driverless car companies are using remote operators. It’s not just Cruise.”
What happened: Waymo publicly described human assistance while distinguishing it from vehicle control. The paper also explains requests for contextual information and directions to stop, so this was substantive disclosure of operators’ roles. (Waymo, published through arXiv, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Nobody knows for SURE that there are no serious possible consequences of open source AI, e.g in terms of bioweapons, misinformation generation etc, as Séb Krier at DeepMind nicely highlights above.”
What happened: The researchers explicitly describe unresolved protection against misuse and report experiments undoing safety training in publicly available Llama 2 models. This supports the absence of an established safety guarantee, without proving that serious real-world harm occurred. (arXiv; Simon Lermen, Charlie Rogers-Smith and Jeffrey Ladish, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“I—truly a techno-optimist, popular perceptions to the contrary—do think driverless will come, eventually.”
What happened: The disclosure reports more than 250,000 paid weekly trips across Phoenix, San Francisco, Los Angeles, and Austin, with over 1,500 commercial vehicles. This establishes recurring practical deployment well beyond a demonstration. (Waymo, )
Assessment: SupportedSaid
“you don’t have to be superintelligent to create serious problems.The real issue is control.”
What happened: Establishes actual fatal harm involving a specialized automated-driving system. Classifying this system as below superintelligence is an evaluator inference. (National Transportation Safety Board, )
Assessment: SupportedSaid
“We have no way of guaranteeing that large language models will reliably stick to facts, a point so thoroughly established by now I don’t think it needs a reference.”
What happened: The contemporary survey considers factuality evaluation and enhancement for standalone and retrieval-augmented models, and describes fully factual LLMs as an outstanding research goal. This supports the ordinary research-state reading, although it does not prove a universal impossibility. (arXiv; Cunxiang Wang and colleagues, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“We have no way of guaranteeing that large language models won’t be jailbroken into saying unethical things. To the contrary, a just-published technique shows that current LLMs can be jailbroken quickly, and automatically.”
What happened: The published demonstration reports roughly five-minute successful jailbreak generation and success rates of 60% for GPT-3.5, 62% for GPT-4, and 72% for PaLM-2. Success was substantially lower for Llama-2 and Claude, so the evidence establishes capability across multiple models rather than uniform vulnerability. (arXiv; Patrick Chao and colleagues, )
Assessment: SupportedSaid
“But, crucially, it often gets these wrong, particularly when they involve spatial reasoning beyond a cookie-cutter formula or reasoning about sequences of disparate events.”
What happened: Both systems failed the pendulum and chemical-reaction problems. The spatial eclipse subset contained six wrong answers out of eight for GPT4+WA and five out of eight for GPT4+CI. These are repeated failures in the described collection. (New York University; Ernest Davis and Scott Aaronson, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Second, GPT-4 doesn’t reliably understand how to use the tools.”
What happened: By completion of testing, the report documented repeated unsuccessful Wolfram queries, failure to request a usable numerical representation, and Code Interpreter bypasses that produced calculation errors. These distinguish tool-use failures from merely misunderstanding a science problem. (Ernest Davis and Scott Aaronson, hosted by New York University, )
Assessment: MixedSaid
“In many cases, a naive user might take the results, invariably presented as fact, as correct, when they are not.”
What held: The answer key for problem 3 is F, while the output confidently selects H. Problems 1 and 4–7 also contain incorrect declarative answers, supporting the recurring confident-error component. (New York University; Ernest Davis and Scott Aaronson, )
What did not: The life-in-the-universe calculation explicitly warns about uncertainty in its estimates and repeats that qualification after reporting percentages. This contradicts the absolute assertion that results invariably appear as unqualified facts. (New York University; Scott Aaronson and Ernest Davis, )
Assessment: SupportedSaid
“Finding a way to get LLMs to reliably use external tools is far from trivial.”
What happened: The July benchmark reports low completion rates for agents operating realistic websites with supporting tools, despite incorporating reasoning before acting. This supports the inference that connecting models to an actionable environment did not itself deliver dependable performance. (arXiv; Shuyan Zhou and colleagues, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Certainly, adding these plug-ins does improve things.”
What happened: On the common 120-problem subset, Figure 4a reports 48.33% success for ChatGPT with Code Interpreter, 45.0% with Wolfram Alpha, and 30.0% for vanilla GPT-4. The commercial products were not evaluated on the entire MATH test set. (arXiv; Qingyun Wu and colleagues, )
Assessment: SupportedSaid
“There is no question that the two plug-ins significantly extend GPT-4’s ability to solve math and science problems.”
What happened: On the matched 120-problem subset, figure 4(a) reports 48.33% success for ChatGPT with Code Interpreter, 45.0% with Wolfram Alpha, and 30.0% for vanilla GPT-4. These are substantial practical gains for both plug-ins, although the comparison does not fully isolate differences between the web products and baseline configuration. (Qingyun Wu and colleagues, arXiv, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“GPT-4 and similar current language-based AI programs notoriously struggle with mathematical calculations.”
What happened: Table 3 reports exact-answer accuracy of 18.84% for GPT-4 and 10.00% for ChatGPT on 9,592 arithmetic cases. These substantial error rates support the ordinary reading of calculation difficulty. (arXiv; Zhen Yang and colleagues, )
Assessment: SupportedSaid
“On some problems that capable students might well find hard, the AI did perfectly; on other problems that lot of mathematically-inclined college students people would find easy, the AI failed miserably.”
What happened: The report identifies correct high-dimensional probability and finite-field linear algebra answers alongside elementary failures. Their mathematical prerequisites support the qualitative difficulty contrast, but the authors explicitly report no human-subject testing. (New York University; Ernest Davis and Scott Aaronson, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“As is by now widely known, they are a hallmark of systems that generalize based on word distribution.”
What happened: The survey identifies hallucination as a substantial reliability problem and organizes existing detection, explanation and mitigation research. This supports broad recognition of a recurring failure, without supplying a universal frequency. (arXiv; Yue Zhang and colleagues, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“To be sure, large language models are, in a lots of ways, an incredible advance over the AI of the 1970s. They can engage in a much broader range of tasks, and generally they need less hand-tinkering (Though they still need some – that’s what reinforcement learning with human feedback is really about.)”
What happened: The report describes instruction-based generalization and evaluations across diverse held-out tasks. Compared with the bounded historical setting, this supports broader reuse with less task-specific programming, while its training-method discussion still documents substantial human design work. (Google Research, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“As we can see, multimodal training is not magically making hallucinations go away.”
What happened: The release assessment reports hallucinations in GPT-4V itself. Its Be My Eyes discussion also reports reductions in hallucination frequency and severity during testing, supporting improvement without elimination. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Transformers and generative AI are getting a lot of attention, and have considerable utility, but they are nothing like the all purpose tools some people seems to imagine them to be.”
What happened: The cumulative usage disclosed on this date demonstrates substantial engagement. The same paragraph documents uses including illustrated stories, social-media thumbnails and design inspiration, supporting practical utility without establishing universal reliability. (Microsoft Bing, )
Assessment: SupportedSaid
“No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”
What happened: The first incremental successor reports fewer false statements than the October-generation Claude 2.0 on the company's factual-question evaluation. This directly supports mitigation of an existing weakness, within that evaluation's scope. (Anthropic, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: MixedSaid
“It makes current AI hard to debug (nobody knows for example how to ground them in facts), and it makes them difficult predict, which means, unlike calculators or spreadsheets, we don’t really know what’s going to happen when we ask them a question.”
What held: The opening paragraph explicitly connects poor understanding with difficulty diagnosing failures, fixing them, and certifying safety. The research also demonstrates limited progress in interpreting and steering a small model, so it supports difficulty rather than total ignorance. (Anthropic, )
What did not: The original paper describes retrieval and individual verification of proposed facts against Wikipedia, with improved factual accuracy. This contradicts the absence of grounding methods, but does not establish error-free generation. (arXiv; Sina J. Semnani and colleagues, )
Assessment: SupportedSaid
“Just like we can’t guarantee that they won’t make stuff up, we don’t know how to guarantee that they won’t write flawed code.”
What happened: The developer explicitly acknowledges factual fabrication in its contemporary model. The surrounding discussion recommends review and grounding appropriate to the application rather than claiming assured factual correctness. (OpenAI, via arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“When it comes to AI in particular, we are getting way ahead of our skis, rushing forward a technology we don’t fully understand.”
What happened: This documents actual expansion of access to GPT-4. It establishes deployment growth, although not that the chosen pace was excessive. (OpenAI, )
Assessment: SupportedSaid
“There’s some information in there, but whatever’s there is often both patchy and problematic.”
What happened: Human evaluation of generated biographies found substantial mixtures of supported and unsupported atomic facts. This supports both useful factual content and recurring reliability problems within that task. (arXiv, )
Assessment: SupportedSaid
“The connections in chatbots are very different from the connections in the brain; it’s a mistake to compare them directly in this way.”
What happened: The review documents materially different learning mechanisms. The evaluator's inference is that counting connections alone does not control for those differences and therefore cannot establish superior learning per connection. (arXiv, )
Assessment: SupportedSaid
“Actually … a lot of us still think you are declaring victory prematurely.”
What happened: Mitchell and Krakauer describe competing research camps and unresolved questions about understanding. This supports the existence of substantive disagreement, rather than unanimous acceptance of a completed victory. (arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It’s also not even remotely new.”
What happened: Louwerse and Zwaan recovered geographic relationships among 50 US cities using newspaper text, latent semantic analysis and multidimensional scaling. This directly establishes the earlier finding Marcus describes. (Cognitive Science; abstract reproduced by PubMed, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Rumors that LLMs have solved then hallucination problem are greatly exaggerated.”
What happened: The revised paper identifies hallucination as an unresolved contemporary problem and reports experiments reducing it through verification. The event date identifies this current-state assessment, not the undisclosed dates of individual experimental runs. (arXiv; Shehzaad Dhuliawala and colleagues, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Explicit verbal instruction is key in training human employees, and LLMs aren’t very good at following those instructions.”
What happened: PaLM 2 S satisfied every checked instruction on 43.07% of prompts under strict scoring and 46.95% under relaxed scoring; individual-instruction accuracy was 55.76% and 59.11%, respectively. These August results show frequent explicit-constraint failures, but this smaller model alone cannot establish performance across all leading LLMs. (arXiv; Jeffrey Zhou and colleagues, )
Assessment: SupportedSaid
“Hallucinations are a huge, unsolved issue.”
What happened: OpenAI identifies continuing factual unreliability and recommends safeguards or avoiding some high-stakes uses. The same section reports substantial improvement over preceding models, which limits any reading that hallucination mitigation had made no progress. (OpenAI, )
Assessment: SupportedSaid
“The competitors may not (yet) be as good, but they are free, which will suit some customers fine.”
What happened: The case study documents a deployed customer application and reports positive reception and further integration. Together with the release terms, this establishes that a free competing model suited a customer. (Meta, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“On made-up facts, in a first experiment, the model was at zero percent correct, on celebrities, in a second experiment, performance was still dismal.”
What happened: Table 1 gives GPT-3-175B synthetic reverse accuracies of 0.0% and 0.1% for the two directions. Celebrity reversal succeeds in 33% of cases, counting a correct answer in any of ten attempts, versus 79% for identifying parents. (arXiv; Berglund et al., )
Assessment: SupportedSaid
“Forty one years later, neural networks (at least of the popular variety) still struggle with this. They still remain pointillistic masses of blurry memory, never as systematic as reasoning machines ought to be.”
What happened: The study evaluates GPT-3, ChatGPT and GPT-4 on multiplication, relational logic puzzles and dynamic programming. Performance deteriorates sharply as compositional complexity increases, supporting the claim that popular models struggle with systematic reasoning. (arXiv; Dziri et al., )
Assessment: SupportedSaid
“The shocking (yet, in historical context, see below, unsurprising) answer is no:”
What happened: Table 1 reports strong directional asymmetry, and section 2.1.2 finds no detectable advantage for the correct reversed name over random names. Section 1 explicitly excludes in-context learning from this failure. (arXiv; Berglund et al., )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Voluntary guidelines aren’t laws, and they aren’t enforcement mechanisms, either; companies can still do as they please.”
What happened: The instrument expressly distinguishes voluntary commitments from future regulations. Reading the complete document reveals no new statutory authority, compulsory enforcement process or penalties for failing to implement its eight safeguards. (White House, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“At least in the United States, policy around AI is still largely notional, not actual.”
What happened: The administration described major executive and legislative safeguards as work still to come. This directly supports the contextual reading of the claim. (White House, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Large language models still make stuff up (“hallucinate”), still can’t be trusted in their reasoning, and remain unreliable.”
What happened: Human fact-checking found substantial unsupported content in generated biographies. These are factual-precision measurements against the study's knowledge source, not general error rates for every ChatGPT answer. (Sewon Min and colleagues, via arXiv, )
Assessment: SupportedSaid
“whose open-air beta test of self-driving vehicles has harmed and killed people”
What happened: The Mountain View crash killed the Tesla driver and injured the Mazda driver; the NTSB identified Autopilot steering and ineffective driver monitoring as causal factors alongside driver distraction and overreliance. The report also documents the beta designation, while identifying damaged roadside safety equipment as contributing to injury severity. (National Transportation Safety Board, )
Assessment: SupportedSaid
“Your private data are going to feed the insatiable beast, too — like it or not.”
What happened: Google describes reviewers assessing users' nonpublic conversations and supplying improvements used by its models. The same FAQ explicitly includes generative-model training among uses of saved activity when Keep Activity is enabled; e2 establishes the default-participation policy. (Google, )
Assessment: SupportedSaid
“And even if this specific example is patched up, there will inevitably be others of a similar flavor.”
What happened: Google identifies AI Overviews as powered by a customized language model and acknowledges advice about putting glue on pizza, alongside mishandling a question about eating rocks. These are subsequent commonsense failures matching the surrounding description of plausible answers containing nonsense. (Google, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Indeed, what we have seen on the AI front so far in the US, for the most part, is voluntary guidelines – stuff that the companies are comfortable with.”
What happened: The announcement directly establishes that the participating companies accepted voluntary commitments. Acceptance supports the ordinary reading of company comfort, without proving their private preferences. (The American Presidency Project, reproducing White House remarks, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“When I think about how poor his company (Meta) is even at making LLMs that don’t hallucinate, I get worried.”
What happened: The 70B chat model achieved 64.14% generations judged both truthful and informative on TruthfulQA, showing a substantial gap from reliable factual answering. This result is also present in the dated July technical report; unsuccessful answers cannot all be counted as hallucinations. (Meta, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“this too seems like a rather large assumption that is far from proven, a possibility but not demonstrably a default,”
What happened: The authors explicitly limit the inference from their formal results to hypothetical superintelligent agents, noting differences between optimal and learned policies. This supports an evidential gap in a key step toward default extinction, although it does not by itself evaluate every possible extinction argument. (arXiv; Alexander Matt Turner and coauthors, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“And of course not every enterprise-scale experiment will be successful.”
What happened: Reporting from Tuesday's bankruptcy hearing documents inability to fulfill the district contract and preparations to close the supplier. Together with the deployed chatbot's shutdown, this establishes failure of the original implementation, beyond a routine temporary interruption. (The 74, )
Assessment: SupportedSaid
“The backlash as begun.”
What happened: Gioia's essay is direct evidence that prominent critical commentary existed before Marcus's post. Its claims about demand are not accepted as evidence of actual demand. (The Honest Broker / Ted Gioia, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“What would make artificial general intelligence general is generality: the ability to cope with new things. And that’s precisely where current AI is still lacking.”
What happened: ConceptARC tested generalization across variations of concepts. Humans substantially exceeded specialized solvers; GPT-4 scored below 30% on 15 of 16 concepts, supporting a substantial limitation under these test conditions. (arXiv; Arseny Moskvichev, Victor Vikram Odouard and Melanie Mitchell, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“But it is also well-known throughout the industry that benchmarks are often quite gameable.”
What happened: The review documents benchmark shortcuts across vision, language, reinforcement learning and algorithmic decision-making. This supports a recurring, publicly recognized problem rather than an isolated vulnerability. (arXiv, )