Claims checked against evidence, page 24 of 31
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (3100)
Each of these claims was checked against dated evidence.
Page 24 of 31: claims 2301 to 2400 of 3100, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“I think it’s fair to say that they have not. GPT-Turbo released this week still produces its share of groaners, like this interchange Phil Libin just sent:”
What happened: The tests used the newly released snapshot on 133 Python exercises, allowing two attempts. Unsuccessful tasks demonstrate that coding errors persisted; this is not a measurement of factual hallucinations. (Aider, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Already in recent weeks, Inflection AI largely closed shop, Stability AI is struggling, and LLM-based autonomous vehicle company called Ghost closed shop, and a software engineer on YouTube raised quite serious questions about the wildly-hyped AI coding system Devin.”
What happened: The regulator found that almost all of the team transferred, including its core expertise. This supports substantial dismantling of the original organization, under the stated interpretation. (UK Competition and Markets Authority, )
Assessment: SupportedSaid
“Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns.”
What happened: Table 1 reports 32.4% for fine-tuned GPT-2 and 43.9% for GPT-3 X-Large evaluated few-shot, a reported increase of 11.5 percentage points. The different evaluation methods prevent treating this as a controlled comparison of unchanged model preparation. (Dan Hendrycks and coauthors, arXiv, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Skeptical of this, I offered Musk a $1 million bet that AGI superior to the smartest individual humans would not arrive by the end of 2025, reprinted here for those who missed it.”
What happened: The retrospective report says every ARC-AGI-2 task was solved by at least two independent non-expert human testers. Its analysis identifies continuing generalization limitations in 2025 AI systems. (ARC Prize Foundation, via arXiv, )
Assessment: SupportedSaid
“Tesla FSD is getting better. But is getting enough better?”
What happened: Crowdsourced tracker data as of early April 2024 showed v12.3 with far more miles between critical disengagements than v11 or v10, a measurable improvement just before the post. (NextBigFuture, )
Assessment: SupportedSaid
“current models aren’t literally look up tables; they can generalize to some degree, but not enough.”
What happened: The study evaluates 25 LLMs and four prompting techniques. It reports resilience to some numerical and wording changes alongside substantial failures under other mathematical perturbations, supporting the contextual claim of partial but unreliable generalization. (arXiv; Li and colleagues, )
Assessment: ContradictedSaid
“This is why driverless cars are still just demos, and why LLMs will never be reliable.”
What happened: The filing describes daily public driverless rides for work, school, errands and airport travel, and identifies Waymo One as a deployed commercial service. Those operations contradict the demonstration-only premise. (Waymo LLC; filed with the California Public Utilities Commission, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Neural networks (at least in the configurations that have been dominant over the last three decades) have trouble generalizing beyond the multidimensional space that surrounds their training examples.”
What happened: WILDS reports performance losses across ten datasets involving natural distribution shifts. This supports recurring generalization difficulty across applications, without establishing a universal inability. (arXiv; Koh and colleagues, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The good news is that word is finally getting out more broadly.”
What happened: A major mainstream body said AI-driven misinformation dominated its 2024 risk outlook, three months before the post. That shows the concern had reached the mainstream. (World Economic Forum, )
Assessment: SupportedSaid
“Meanwhile X itself is promoting nonsense.”
What happened: Mashable directly observed the false story and documented its placement in Explore and the homepage sidebar. This establishes amplification by X's own features during the target period. (Mashable, )
Assessment: SupportedSaid
“Ironic that his own chatbot, Grok, is making the problem worse.”
What happened: Grok wrote a false headline that X promoted to users, a direct addition to information pollution. (Mashable, )
Assessment: SupportedSaid
“The bad news, which outweighs the good, is that the problem is quickly getting worse.”
What happened: A more than tenfold rise in AI content-farm sites in about seven months is fast growth in AI information pollution. (The Guardian (Nigeria), reporting NewsGuard data, )
Assessment: ContradictedSaid
“Artificial intelligence isn't new, these models aren't new, and after just over a year of hype, generative AI is yet to prove it can do anything substantially different other than helping tech company CEOs raise money and create headlines.”
What happened: The public research demonstration reported practical customer-support benefits beyond fundraising or publicity. The event date records when that demonstration became public, not when the underlying workplace observations occurred. (arXiv; Erik Brynjolfsson, Danielle Li, and Lindsey Raymond, )
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“In essence, the AI boom requires more high-quality data than currently exists to progress past the point we're currently at, which is one where the outputs of generative AI are deeply unreliable.”
What happened: SCoRe uses existing MATH and MBPP problems, generated correction traces, and correctness rewards. Its Gemini 1.0 Pro coding experiments improve self-correction on held-out HumanEval, demonstrating further error-correction progress without requiring an expanded stock of human-written material. (Google DeepMind researchers via arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: MixedSaid
“As I've discussed before, today's models are rife with hallucinations that make integrating them actively dangerous for many businesses, and the only way to have fewer hallucinations is to give the models more compute power to process and analyze data, a dwindling resource that AI is actively helping to destroy.”
What held: The January version reports frequent hallucinations on verifiable legal questions, supporting a material risk in consequential business workflows. It supports the risk clause within legal tasks, not a measured count of affected businesses or the compute-depletion claim. (arXiv; Matthew Dahl and colleagues, )
What did not: The paper reports improved Alpaca truthfulness on TruthfulQA through activation intervention and explains how to incorporate it into existing parameters. This counters the necessity of greater deployed processing capacity, although learning the intervention requires offline computation and the result is benchmark-specific. (arXiv; Kenneth Li and colleagues, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Models like ChatGPT and Claude are deeply dependent on training data to improve their outputs, and their very existence is actively impeding the creation of the very thing they need to survive.”
What happened: The study estimates a 25% relative reduction in Stack Overflow activity within six months of ChatGPT's release, using less-exposed comparison platforms. It examines content creation and reports no significant deterioration in post quality that would explain the loss as merely low-quality material. (PNAS Nexus; institutional record at UCL Discovery, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The trouble is that the bigger the model, the higher the internal costs. And current-sized models, already expensive, aren’t reliable enough for many jobs.”
What happened: The contemporaneous engineering assessment supports the general size-resource relationship and describes substantial deployment demands. It also explains that sparse architectures can weaken the relationship between total parameters and computation. (arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“As noted here before, one study indicates that code quality is going down.”
What happened: The study reports declining maintainability indicators in code authored through December 2023. This directly supports the limited assertion that one study indicated declining quality. (GitClear, )
Assessment: SupportedSaid
“On the other hand, the use cases on the negative side of the ledger are proliferating fast, literally faster than I can report them here, from fake books (undermining authors) to rings of fake websites designed to sell ads”
What happened: The report records 713 identified AI-generated sites, compared with fewer than 50 in May and around 600 in December 2023, and describes advertising incentives. These are discovery counts, not a controlled census of site creation. (The Guardian Nigeria, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Some core problems have lingered, particularly hallucinations, instability and occasional nonsensical errors.”
What happened: The experiments demonstrate false supporting assertions that GPT-4 can recognize as wrong when asked separately. They establish hallucination and elementary-error failure modes, not a population-wide error rate. (Zhang and colleagues, arXiv, )
Assessment: SupportedSaid
“We can expect price wars.”
What happened: Reuters documents Alibaba's deep cuts followed hours later by Baidu making previously paid Ernie Speed and Ernie Lite services free. It explicitly identifies the competitive episode as extending into LLM pricing. (Reuters, republished by Yahoo Finance, )
Assessment: SupportedSaid
“There has however been lots of progress both in discovering potential applications and in putting GPT-4 class models into practice.”
What happened: The announcement identifies GPT-4-powered conversation practice and answer explanations, available immediately through Duolingo Max in selected iOS markets. This documents distinct educational applications implemented in a released product. (Duolingo, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Translation: Using a well-known algorithm that we didn’t invent to decide which faces of the cube to turn, we taught a robot to do motor control on a specially instrumented Rubik’s cube with bluetooth sensors inside.”
What happened: The announcement explicitly separates Kociemba's move sequence from the neural networks' learned manipulation. Its approach section states that training occurred in simulation. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Translation: But (revealed a few weeks later) behind the scenes we will try hard to water down your regulations.”
What happened: The surrounding exchange supports public cooperation on regulation. Elsewhere in the same testimony, Altman advocates licensing and safety requirements. (U.S. Government Publishing Office, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“GenAI is messing with science.”
What happened: The original paper expressly identifies Midjourney as the source of its images. It is evidence of AI provenance, not evidence that the paper's scientific assertions are reliable. (Frontiers in Cell and Developmental Biology; Xinyu Guo, Liang Dong and Dingjun Hao, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Generative AI is a perfect tool for soullessly churning out content to match a particular set of instructions — such as those that an algorithm follows — and while an algorithm can theoretically be tuned to evaluate content as "human," so can scaled content be tweaked to make it seem more human.”
What happened: The researchers generated tens of thousands of instruction-following examples, establishing repeatable production at scale. Their accompanying evaluation and limitations do not establish flawless adherence or universal content quality. (Stanford Center for Research on Foundation Models, )
Assessment: SupportedSaid
“The algorithms are easily-tricked, and the tools used to trick them are becoming easier to use and scale.”
What happened: The report describes using Byword to generate 1,800 articles in hours and obtain search traffic, supporting accessible production at scale. It also reports that much of the traffic subsequently disappeared; figures are estimates and campaign claims, not audited platform records. (The Decoder, )
Assessment: SupportedSaid
“Yet at some point these portals will generate an answer based off of the data they have (or have access to, in the case of Tumblr and Wordpress) rather than linking you to a place where you can find an answer by reading something created by another person.”
What happened: Google announced public US rollout of AI Mode, describing retrieval across web subtopics and an in-search response experience. The same announcement retains source links, which limits support to direct answers substituting for necessary source visits. (Google, )
Assessment: SupportedSaid
““Current deep-learning systems frequently succumb to stupid errors.” Still true.”
What happened: Experts evaluated 26 ordinary election questions across five leading models; errors included incorrect contact details and fabricated information. GPT-4 performed substantially better, with 19% of its responses rated inaccurate, and the authors explicitly warn that this small, nonrandom sample is not representative of everyday usage. (Proof News and the Institute for Advanced Study, AI Democracy Projects, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
““new effort by OpenAI to solve these problems wound [still] fabricat[ing] authoritative nonsense ” Still true.”
What happened: GPT-4 confidently supplied permission and an accompanying false description of Texas polling-place rules, which the evaluators checked against the restriction on campaign apparel. The same report finds GPT-4 substantially more accurate than its peers and describes successful referrals to official polling information. (Proof News and the Institute for Advanced Study, AI Democracy Projects, )
Assessment: SupportedSaid
“And the apparent copyright infringements that Reid Southen and I discussed in early January continue apace two months later, as I found in a quick experiment this morning:”
What happened: The article reports its own reproduction of the described Elsa images with Copilot Designer and identifies DALL-E 3 as the underlying model. This is firsthand corroboration published on March 6; the article does not give a separate test timestamp. (The Verge, )
Assessment: SupportedSaid
“Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has trained on.”
What happened: The same paragraph expressly withholds detailed architecture, size, hardware, compute, dataset construction, and training methods. It nevertheless identifies a Transformer-style model, next-token pretraining, public and licensed data, and reinforcement learning from human feedback. (OpenAI, )
Assessment: SupportedSaid
“They knew they were misleading people and failed to take an opportunity to correct.”
What happened: Amodei described a belief that OpenAI would release its AI source code publicly. Altman acknowledged that universal release was not the plan and then asked that the misconception not be corrected. (The New Yorker, )
Assessment: SupportedSaid
“On overt measures, the systems were fine. On covert measures, they were a disaster:”
What happened: The abstract contrasts strongly negative dialect-based stereotypes with more positive overt stereotypes. This retrospectively corroborates the findings already reported in the March 1 preprint; the event date identifies that original reporting, not a new August experiment. (Nature, )
Assessment: MixedSaid
“The agony starts with the fact that Elon Musk has a point; OpenAI, Altman and Brockman *have* changed their mission since he gave them his money, his time and his reputation.”
What held: OpenAI introduced investor and employee returns and moved most staff into the new entity, materially changing its original operating model. The same announcement retained nonprofit control and priority for the Charter, limiting what the restructuring proves about its ultimate purpose. (OpenAI, )
What did not: The Charter preserves a humanity-wide beneficiary and expressly prioritizes public benefit. It contradicts change in that component of the mission, although a stated commitment alone cannot establish compliance in practice. (OpenAI, )
Assessment: MixedSaid
“The entity strayed from that mission.”
What held: OpenAI explicitly made commercial funding a reason to offer controlled API access instead of releasing the models. This supports departure from its original operating approach, while the same explanation also invokes accessibility and misuse prevention. (OpenAI, )
What did not: The testimony records continuing nonprofit control and capped commercial returns; adjacent provisions require directors to further the public-benefit mission. This contradicts departure in the governance and beneficiary components, without proving that all conduct complied with them. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Dropped is any mention of (a) the nonprofit (even though legally the company remains a nonprofit), (b) open source, once central both to their mission and name, and (c) the original phrasing about being "unconstrained by a need to generate financial return."”
What happened: Inspection of the complete extracted page finds none of the three elements. The quotation illustrates the page's research presentation; the absence finding rests on the full page, including its navigation and footer. (OpenAI, archived by Internet Archive, )
Assessment: UnresolvedSaid
Assessment: MixedSaid
“The company Sam and Greg built has little to do with what was originally promised.”
What held: OpenAI introduced investor and employee financial returns through a commercial subsidiary. This materially changed the original model, although the same announcement retained nonprofit control and mission-first obligations. (OpenAI, )
What did not: The announcement identifies the nonprofit as the governing body and retains mission-based oversight. This is meaningful continuity with the original purpose, although it does not prove that commercial pressures had no practical influence. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Shortly after I sent yesterday’s note on OpenAI’s latest woes, it was revealed that they are also under SEC investigation.”
What happened: The contemporaneous report describes an ongoing SEC investigation involving OpenAI. This supports the claim that OpenAI was under investigation at the time, while retaining Reuters' attribution to the Wall Street Journal. (Reuters, republished by Yahoo Finance, )
Assessment: MixedSaid
“Amazon already plans to heavily integrate generative AI into Alexa, and Apple is planning the same with Siri, all to show the world that they're "innovative" and "pushing boundaries" without being able to explain how.”
What held: Amazon publicly described a substantial LLM-based redesign before the post. This supports the Alexa plan component, not the claimed motive. (Amazon, )
What did not: Amazon explained a concrete intended improvement: creating routines through speech without manual app programming. This contradicts a literal inability to explain the proposed benefit, but does not prove reliable delivery or disprove promotional motives. (Amazon, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“While Google has yet to fully make the move, the writing is on the wall, regardless of whether the actual assistant is fit for the task and not, say, hallucinating things that don't exist.”
What happened: The reporter directly confirmed that Assistant could no longer be selected on the tested Pixel. The article places this observation within Google's mobile migration and reports the removal of the previous switching controls. (Droid Life, )
Assessment: SupportedSaid
“That’s exactly what classical AI has been trying to do for decades, with only limited success. And what systems like Gemini still struggle with.”
What happened: The historical synthesis describes domain restrictions and adaptation problems, including the 1984 SAVVY system. This supports limited general coverage, rather than an absence of successful applications. (arXiv; Kamath and Das, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The trouble is that so far RAG has been hit and miss.”
What happened: The study evaluated Bing Chat, NeevaAI, perplexity.ai, and YouChat using responses collected in February–March 2023; its methods identify their use of retrieved content. Average sentence citation support was 51.5% and citation precision was 74.5%, documenting supported outputs alongside citation failures. (Liu, Zhang and Liang; Findings of EMNLP 2023, hosted on arXiv, )