Claims checked against evidence, page 25 of 31
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (3100)
Each of these claims was checked against dated evidence.
Page 25 of 31: claims 2401 to 2500 of 3100, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
“Imagining perfectly reliable, hallucination-free chatbots by this time next year is bonkers.”
What happened: The release documents continuing factual errors and evaluates several OpenAI models. Its tests exclude browsing, so they establish limitations of those model configurations rather than every retrieval-enabled chatbot. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Imagining this as a six-month project is naive.”
What happened: CRAG tests diverse factual questions across domains and question types. Its reported industry-system performance falls well short of broadly reliable query answering; the remaining percentage should not automatically be treated as a hallucination rate. (Yang et al., arXiv, )
Assessment: SupportedSaid
“But we are kidding ourselves if it we think the solution is near at hand.”
What happened: The BBC tested ChatGPT, Copilot, Gemini and Perplexity with access to BBC sources. Beyond the broader issue measure quoted here, it reported factual errors in 19% of answers citing BBC content, supporting continued factual unreliability. (BBC, )
Assessment: SupportedSaid
“The road from research idea to reliable production remains steep.”
What happened: The January report describes seven failure modes across research, education, and biomedical implementations. Its findings on testing, retrieval, answer extraction, and incomplete responses support substantial remaining production work. (arXiv; Barnett and colleagues, Deakin University, )
Assessment: SupportedSaid
“Reality is that RAG is still in its infancy, a work in progress much as LLMs themselves are, an no more guaranteed to be correct. Like LLMs, RAG can appear spectacular at times, but there is no evidence that it is a cure-all.”
What happened: Three engineering case studies identify unresolved retrieval and generation failures, including failure despite the answer being supplied in context. The January report documents the then-existing reliability limitations; individual test days are not specified. (Barnett and colleagues, Deakin University; arXiv, )
Assessment: UnresolvedSaid
Assessment: MixedSaid
“So let me get this straight? Sora’s can’t reliably handle basic physics, ChatGPT had an unexplained meltdown, Gemini can’t even remember who was on Apollo 11, there are no formal guarantees that any of this will ever work, you yourself admit you don’t know how it works, or how much better it will get next time around, and you want another $7 trillion dollars?”
What held: Documents the conspicuous malfunction described as a meltdown. It supports that component, not continued absence of an explanation on February 22. (ITPro, )
What did not: The event being dated here is publication of the explanation. OpenAI's February 21 update attributed the malfunction to an optimization bug and described incorrect inference-kernel results, preceding Marcus's February 22 post. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Over the last few hours, people reporting having been report a variety of problems with ChatGPT:”
What happened: OpenAI acknowledged receiving reports during the incident window. This corroborates their existence without establishing every reported symptom. (OpenAI, )
Assessment: SupportedSaid
“And one wants the system to have a nuanced understanding of what is even been asked. (That’s the clearly lacking now).”
What happened: The reporter's tests produced women as nineteenth-century US senators, despite the article identifying 1922 as the first woman's Senate service. The same report found a substantially appropriate Founding Fathers response, showing inconsistent contextual handling rather than universal failure. (The Verge, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It’s a general issue.”
What happened: The reporters reproduced inappropriate demographic depictions for German soldiers and nineteenth-century US senators, including female senators. These are separate historical subjects beyond popes; the report also records a more appropriate Founding Fathers result. (The Verge, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“You might have thought that the overactive guardrail problem would have been solved by now, what with billions more invested and ever-bigger models, not to mention the cultural backlash from the right. You’d be wrong.”
What happened: The reporter directly tested ordinary portrait requests and observed refusals. This establishes that excessive restrictions persisted in a major assistant during the target period; a news test supplies the dated example missing from the provider's disclosure. (The Daily Dot, )
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“ChatGPT can spin up piles of anodyne business copy, yet its outputs always require enough editing that it's questionable how much time you've actually saved.”
What happened: The randomized experiment involved 453 professionals completing occupation-specific writing assignments. Substantial savings on completed work, accompanied by higher assessed quality, contradict the blanket claim under the stated reading of business copy. (Science; abstract reproduced by Stanford SCALE, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“There are cool little app integrations, interesting things like live translation in Samsung devices, but these are features, not applications.”
What happened: Samsung's launch documentation places Live Translate inside its existing Phone app. That supports the feature-versus-standalone-product distinction for the named example. (Samsung Electronics, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“And there are so many stories about how unreliable this technology is.”
What happened: The court found that ChatGPT fabricated cases used in the March 1 filing. This is a primary judicial finding of incorrect generated material, rather than an allegation repeated by the speaker. (U.S. District Court for the Southern District of New York, reproduced by FindLaw, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“In short order, with Sora or some opensource counterparts that will emerge, scammers will start to make photorealistic fake videos simply to sell ads.”
What happened: The official release record establishes an actual video-generation release after the prediction. The later dated paper supplies fuller evidence of functionality and access. (HPC-AI Tech, )
Assessment: SupportedSaid
“… but, um, ants have six legs.”
What happened: An independent source says the ants in the Sora ant-tunnel video have four legs, not six. That matches Marcus's point. (HTC VIVE (VIVE POST-WAVE blog), )
Assessment: SupportedSaid
“I guarantee he won’t be the last:”
What happened: A wire service reports that Sora videos have fooled many people since September 2025. That directly matches the forecast. (The Canadian Press (via Castanet), )
Assessment: SupportedSaid
“In brief, an Air Canada chatbot confabulated (“hallucinated”) an answer – telling a customer that bereavement trips would be refunded.”
What happened: The decision transcribes advice allowing a reduced bereavement rate through a refund application after travel, contrary to the policy it records. It finds negligent misrepresentation; the promised benefit concerned a partial fare reduction. (British Columbia Civil Resolution Tribunal, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Aside from showing that (not surprisingly) Sora is pretty clueless about chess, it’s revealing about how the mysterious Sora even works.”
What happened: The report attributes the same elementary depiction errors to Rahul Agarwalla, rather than quoting Marcus. The event date refers to the original monkey demonstration, whose timestamp was checked on Altman's post; March 6 is the reporting date. (The Times of India, )
Assessment: SupportedSaid
“Nowadays, we have “multimodal” systems, trained on both text and images, and the aforementioned “hallucinations” (perhaps better called failed approximations, as Gerben Wierda has pointed out) and other errors persist and even crop up in new ways.”
What happened: The launch material documents invented entities in generated video. This supports the ordinary reading that multimodal generation retains fabrication errors and exhibits visual consistency failures. (OpenAI, )
Assessment: SupportedSaid
“Turns out Sora makes stuff up, too,”
What happened: OpenAI identifies an unintended physical error in a launch demonstration. This supports the existence of realistic-looking errors without depending on Marcus's chessboard analysis. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Gary Marcus is astonished that some people still haven’t recognized that generative AI has struggled mightily when it comes to world models, high level reasoning and factuality, even as evidence for that conclusion continues to pour in, even with data of ever-greater magnitude and sophistication..”
What happened: The evaluation reports substantial failures to generate requested object relationships despite strong visual quality. This supports difficulties with spatial world consistency, without proving the absence of internal world models. (arXiv; Gokhale and colleagues, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“We will, I am certain, see more systemic glitches as more people have access.”
What happened: The release expanded access beyond the February preview group, satisfying the antecedent. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Sora is not a solution to AI’s longstanding woes with space, time, and causality.”
What happened: The motion weakness directly establishes an unresolved physical-consistency problem. The surrounding limitations text also describes errors in spatial instructions, event trajectories, and consequences of interactions. (OpenAI, )
Assessment: SupportedSaid
“And god save us from the deluge of deepfakery that is to come.”
What happened: IWF assessed 3,443 realistic AI-generated abuse videos in 2025, versus 13 in 2024. Its accompanying observations include fabricated videos of known victims, establishing substantial harmful impersonation rather than merely benign synthetic production. (Internet Watch Foundation, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”
What happened: OpenAI describes spontaneous appearances as a model weakness associated with a class of scenes, rather than as one isolated video defect. This directly supports recurrence without establishing how often it happens. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“But the truth is it doesn’t work that well yet.”
What happened: On CodeContests validation problems, direct prompting solved 19% with five attempts; a structured testing and revision process reached 44%. This supports substantial reliability limitations while also documenting useful capability and improvement. (arXiv; Tal Ridnik, Dedy Kredo and Itamar Friedman, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“In some ways we have made progress, but in the ones I keep harping on—factuality and compositionality—we have not.”
What happened: The report compares factual accuracy on TruthfulQA and reports gains over earlier models. This contradicts a categorical absence of factuality progress, subject to the company's evaluation methods and contamination caveat. (OpenAI via arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“OpenAI has never disclosed those numbers for GPT-4; I can scarcely imagine what they will be for GPT-5.”
What happened: The assessment includes OpenAI's GPT-4. The following sentence identifies broader environmental impact among the universally undisclosed indicators, and the preceding discussion explicitly includes water use. (Stanford CRFM and collaborating researchers, via arXiv, )
Assessment: SupportedSaid
“And, let’s be honest folks, the “alignment problem” isn’t remotely solved.”
What happened: The experiments show a substantial robustness gap: standard safety training did not reliably remove deliberately implanted deceptive behavior. This supports the ordinary reading that general alignment remained unresolved, without proving that deployed models spontaneously developed such behavior. (arXiv; Evan Hubinger and coauthors, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“In a system that can write emails, make appointments, sign checks, etc, the consequences of unreliability, discomprehension, and hallucination all escalate.”
What happened: The researchers reproduced destructive execution in a real sandbox, rather than merely generating harmful text. The paper traces agent failures involving misinterpretation and unsafe assumptions into consequential tool actions; the exact experiment date is not reported. (arXiv, )
Assessment: SupportedSaid
“This list is probably not exhaustive and will continue to grow.”
What happened: Tenable publicly disclosed a ChatGPT Search attack that bypassed isolation to inject and extract memories. The disclosure adds a distinct product-specific issue to the earlier list; the event dated here is public disclosure, following private reporting in December 2024. (Tenable, )
Assessment: SupportedSaid
“More evidence against his claims just keeps piling up.”
What happened: The public study documents sensitivity to meaning-preserving formatting changes. This supplies evidence of fragile generalization, although it does not establish an absence of understanding. (arXiv; Sclar and colleagues, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Reliability remain a problem.”
What happened: Table 3 reports GPT-4 prompt-level accuracy of 76.89% under strict checking and 79.30% under loose checking. The November tests therefore show frequent instruction failures, while also showing that most tested prompts succeeded. (arXiv; Jeffrey Zhou and colleagues, )
Assessment: SupportedSaid
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
What happened: OpenAI describes GPT-4 as a milestone in scaling deep learning while acknowledging factual and reasoning errors. The same disclosure reports reduced hallucinations and improved factuality, showing progress without complete resolution. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Compositionality remains a problem. So does negation.”
What happened: The researchers tested multiplication, logic puzzles and dynamic programming. GPT-4's multiplication accuracy declined sharply with problem complexity; step-by-step prompting improved simpler cases without eliminating the harder failures. (Allen Institute for Artificial Intelligence, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: MixedSaid
“Special-purpose AI has been great for static, easily-simulated games with rules that have been stable for centuries, like Chess, Go, and even Diplomacy, but never been great at dynamic environments in which the environment itself is unpredictable.”
What held: The report describes task-trained AlphaZero systems achieving superhuman chess and Go performance and defeating champion programs. This supports the positive game-performance component. (arXiv; David Silver and colleagues, )
What did not: The study describes real-world balloon control despite forecast errors and sparse wind measurements, including a controlled field experiment. Strong performance under those external uncertainties contradicts the unrestricted never assertion. (Nature; abstract indexed by PubMed, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It’s notorious for making illegal moves in chess.”
What happened: The researchers tested gpt-3.5-turbo-0301 across 3,200 games. Their procedure terminated games after ten consecutive illegal attempts, and their results document persistent rule-following failures. (arXiv; Mu-Tien Kuo, Chih-Chung Hsueh and Richard Tzong-Han Tsai, )
Assessment: MixedSaid
“The New York Times versus OpenAI lawsuit made that abundantly clear, with a hundred examples like this one, in which ChatGPT reconstructed entire stretches of text (red) from brief prompts2 (black text in top line):”
What held: Exhibit J identifies 100 GPT-4 examples and presents matching article passages. Its actual prompts vary: Examples 1 and 2 contain several paragraphs, while Examples 3 and 100 have short lead-ins, so the exhibit does not uniformly substantiate 'brief prompts.' (The New York Times Company; exhibit filed in the Southern District of New York, )
What did not: This later filing describes the December exhibit as API-generated GPT-4 outputs and separately categorizes ChatGPT outputs. It contradicts the precise product attribution, although it is a defendant's submission rather than a court finding. (OpenAI defendants; filing in the Southern District of New York, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“But the variety of the examples along with the technical literature on memorization and data leakage in large language models make clear that this kind of reconstruction certainly can happen with some regularity.”
What happened: Thousands of distinct extracted examples establish recurrence under a targeted attack. Because ChatGPT's training corpus was unavailable, the researchers checked matches against a large auxiliary corpus; this is less direct than verification against a disclosed training set. (arXiv; Nasr and colleagues, )
Assessment: SupportedSaid
“LLMs don’t store things in simple databases, but when presented with enough data, their regeneration of distributed bits has demonstrably duplicated lengthy training materials multiple times, a phenomenon that is called memorization in the field.”
What happened: The GPT-2 study documents an entire code file and complete license texts among its extended extractions. These are multiple substantial reproductions, well beyond ordinary short-phrase overlap. (Carlini et al., arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Neural networks do effectively memorize some of what they are trained on, and indeed there is a whole subfield now about the topic.”
What happened: The paper reports substantial extraction from neural language models and directly checks known training corpora for open models. Its methods and review of preceding studies also support the existence of an established research line. (Nasr and colleagues, arXiv, )
Assessment: SupportedSaid
“An entire cast of deepfaked people include a deep faked CFO scammed a company out $25 million.”
What happened: CNN reports the transfers and their US-dollar equivalent, alongside police descriptions of the fabricated CFO and colleagues. It also reports Arup's confirmation that it was the victim; the amount supports the approximate reading. (CNN, )
Assessment: SupportedSaid
“Even scarier is that the employee who was tricked was suspicious, not naive, and still fell for it:”
What happened: CNN attributes initial suspicion to police superintendent Baron Chan. The following passages explain that the employee suspected phishing, set aside those doubts after seeing apparent colleagues, and agreed to transfer the money. (CNN, )
Assessment: SupportedSaid
“I am here to tell you that OpenAI’s research, analyzed carefully, shows no such thing, and that there is plenty of reason to be concerned.”
What happened: Limited sensitivity prevents treating nonsignificance as a tight risk bound. The study also distinguishes information access from physical implementation and leaves the translation from scores to risk unsettled. (OpenAI, )