Claims checked against evidence, page 30 of 31
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (3100)
Each of these claims was checked against dated evidence.
Page 30 of 31: claims 2901 to 3000 of 3100, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
“It made things up (figure 6). Its output, when released, exhibited sexist and racist biases (figure 7).”
What happened: OpenAI itself acknowledged at launch that ChatGPT produced plausible but false answers. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“That’s a huge commitment, 3.5 hours, but one thing that I think all of our speakers could agree on is that artificial intelligence, in whatever form it currently is, is about to have a huge impact on society.”
What happened: The representative survey reports generative-AI adoption among 39.4% of US adults aged 18–64, with 28.1% of employed respondents reporting work use. This establishes mass uptake within approximately twenty months of the post; the measure does not imply that all users used AI daily. (Federal Reserve Bank of St. Louis; Alexander Bick, Adam Blandin, and David Deming, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Driverless cars have taken many years longer than we were originally told; outliers (AKA edge cases) have, at least so far, prevented them from making the transition from demo to widely-available reality.”
What happened: The report documents missed rollout expectations, including Tesla's 2020 robotaxi promise, and explains how unexpected situations create stops requiring human help. This supports a material operational barrier to expansion, though it does not isolate edge cases from other causes. (Reuters, republished by Business Recorder, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Large language models are not databases. They are glommers-together-of-bits-that don’t always belong together.”
What happened: The technical description identifies generation as successive probabilistic predictions. This supports the distinction from retrieving explicit records; the database comparison is the evaluator's inference from the described mechanism. (BigScience collaboration / arXiv, )
Assessment: SupportedSaid
“Guardrails (a la ChatGPT) may be in place, but the guardrails will teeter between being too weak (beaten by “jailbreaks”) and too strong (rejecting some perfectly reasonable requests).”
What happened: The launch disclosure describes GPT-4's safety training and interventions, establishing that safeguards were present. It also reports safety improvements, which limit any reading that the safeguards were wholly ineffective. (OpenAI, )
Assessment: SupportedSaid
“I guarantee that minds will be blown.”
What happened: The announcement establishes that the release condition occurred. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It will often do what you want, sometimes not—and it will remain difficult to anticipate which in advance..”
What happened: Using approximately 19,000 human generalization responses across 79 tasks, the researchers estimated advance beliefs and compared them with model correctness: GPT-4 performed best under equal error and success weights but worst under their highest-stakes weighting. This supports meaningful mismatches between expected and actual capability, while also demonstrating substantial success. (arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The system will still not be able to restrict its output to reliably following a shared set of human values around helpfulness, harmlessness, and truthfulness.”
What happened: The report describes continuing factual and reasoning failures and recommends safeguards in consequential applications. It also reports a 19-percentage-point improvement over GPT-3.5 on internal adversarial factuality evaluations, which qualifies the finding without establishing dependable truthfulness. (OpenAI, via arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“GPT-4 will solve many of the individual specific items used in prior benchmarks, but still get tripped up, particularly in longer and more complex scenarios.”
What happened: The report gives GPT-4 scores of 86.4% on MMLU and 95.3% on HellaSwag, establishing that it solved many established benchmark items. These particular results concern the pretrained base model and do not independently establish deployed-model reliability. (OpenAI via arXiv, )
Assessment: SupportedSaid
“As OpenAI has learned, bigger in many ways means better, with outputs more and more humanlike with each iteration. GPT-4 is going to be a monster.”
What happened: The report records preference for GPT-4 on 70.2% of 5,214 prompts and broad benchmark gains over predecessors. These are developer-run evaluations supporting a substantial capability and response-quality advance, not proof of human understanding. (OpenAI / arXiv, )
Assessment: SupportedSaid
“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”
What happened: GPT-4 achieved 76% accuracy joining first letters but 3% joining second letters, and 80% sorting words alphabetically but 32% in reverse order. These controlled differences support recurring elementary mistakes and unexpectedly uneven performance, although the authors also successfully predicted broad patterns from task frequency. (arXiv, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
““Alignment” between what humans want and what machines do will continue to be a critical, unsolved problem.”
What happened: OpenAI acknowledges remaining harmful behavior and jailbreaks despite improved safeguards. Its report of substantial safety improvements qualifies the claim but does not show alignment was solved. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“In short, ChatGPT might sound like the Star Trek computer, but for now, you still can’t trust it.”
What happened: The developer acknowledges that convincing presentation can accompany incorrect content. The need to verify substantive answers before relying on them is an evaluator inference from this disclosed limitation. (OpenAI, )
Assessment: SupportedSaid
“Some of its advice will be head-scratchingly bad.”
What happened: Two infectious-disease physicians assessed GPT-4 management recommendations against clinical literature and guidelines and identified harmful antibiotic advice. The same evaluation found optimal antibiotic recommendations in other cases, so it supports some bad advice without establishing universal failure. (Clinical Infectious Diseases; abstract hosted by PubMed, )
Assessment: SupportedSaid
“When queried on medicine it will either resist answering (if there are aggressive guardrails) or occasionally spout plausible-sounding but dangerous nonsense.”
What happened: Researchers submitted 44 bloodstream-infection cases for medical management advice. This establishes the medical-query condition and the apparent fluency of the responses. The availability date is the indexed print publication date. (Clinical Infectious Diseases, indexed by PubMed, )
Assessment: SupportedSaid
“Although ChatGPT can write about anything, it is also easily confused.”
What happened: The dated transcript records a false answer to a simple arithmetic question, followed by reasoning inconsistent with that answer. Other exchanges show repeated mistakes, although some prompts deliberately test false premises and some answers are correct. (Vipul Naik / LessWrong, )
Assessment: SupportedSaid
“It will remain turbocharged pastiche generator, and a fine tool for brainstorming, and for first drafts, but not trustworthy general intelligence.”
What happened: The institution reports improved performance with GPT-4 on consulting work covering creativity and writing. The underlying paper identifies product brainstorming, press-release drafts and persuasive memos among the tasks. (Harvard Business School AI Institute, )
Assessment: SupportedSaid
“GPT-4 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
What happened: The systematic evaluation documents repeated harmful outputs under jailbreaks; its main GPT data collection ran from April 26 through May 1, with a subsequent correction rerun. Material qualification within the same study is that unmodified requests rarely elicited harmful answers and some attacks were less effective against GPT-4 than GPT-3.5. (Alexander Wei, Nika Haghtalab, and Jacob Steinhardt / arXiv, )
Assessment: SupportedSaid
“Alas, the genie can no longer be stuffed back in the bottle.”
What happened: Meta pulled only the demo. It said the models stayed available so others could reproduce the results, which fits his point that withdrawal did not contain the technology. (MIT Technology Review, )
Assessment: SupportedSaid
“Large language models lack mechanisms for verifying truth; we need to find new ways to integrate them with the tools of classical AI, such as databases, webs of knowledge, and reasoning.”
What happened: The paper explicitly describes default generation as ungrounded and introduces additional training and evidence provision to address it. This supports the contextual reading about standalone models, rather than an assertion that no augmented system existed. (DeepMind researchers, via arXiv, )
Assessment: SupportedSaid
“For another, chatGPT, just released by OpenAI, is more or less just as capable of producing similar nonsense, such as instant essays on adding wood chips to breakfast cereal.”
What happened: In its first week ChatGPT was reported to confidently produce plausible but false content, the same failure Galactica was pulled for. (The Verge, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“GPT doesn’t talk randomly, because it’s pastiching things actual people said.”
What happened: The study documents training on web text, books and Wikipedia, alongside controlled assessments of generated articles. Together these support learned language patterns as the basis of structured output, without establishing literal cut-and-paste generation. (OpenAI researchers / arXiv, )
Assessment: SupportedSaid
“The impressive thing about GPT is that it spits out hundreds of perfectly fluent, often plausible prose at a regular clip, with no human filtering required.”
What happened: The first four conversations contain repeated readable, generally grammatical responses across recipes, social media, mood, and AI. The transcript exceeds hundreds of words and is explicitly described as unedited; this is evidence of output quality, not validation of its advice. (Jeff Turner, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Chance is definitely part of what’s going on. But it’s not quite the right way to understand the juxtaposition of brilliance and stupidity that we see within GPT.”
What happened: Human detection accuracy was about 86% for the degraded control and 52% for GPT-3, with outputs selected programmatically. This supports systematic competence beyond random text, although the control changes several factors together. (Brown et al., arXiv, )
Assessment: SupportedSaid
“It could not be immediately applied to the challenge of, say, customer service or guiding the actions of a domestic robot, or well, almost anything else, really.”
What happened: Meta explicitly limits the released agent's capabilities to Diplomacy while describing other applications as prospects for its technology. This supports Marcus's distinction between immediate deployment and later adaptation. (Meta, )
Assessment: SupportedSaid
“Strikingly, and in opposition to much of the Zeitgeist, Cicero relies quite heavily on hand-crafting, both in the data sets, and in the architecture; in this sense it is in many ways more reminiscent of classical “Good Old Fashioned AI” than deep learning systems that tend to be less structured, and less customized to particular problems.”
What happened: The supplement specifies rules for corrupting game entities, targeted examples for counting and negation errors, and a separately configured classifier ensemble. Together with its explicit filtering and strategic-planning procedures, these document substantial task-specific engineering in both data and architecture. (Science / AAAS; supplementary PDF hosted by Gwern, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Cicero makes extensive use of machine learning, but is hardly a poster child for simply making ever bigger models (so-called “scaling maximalism”), nor for the currently popular view of “end-to-end” machine learning of in which some single general learning algorithm applies across the board, with little internal structure and zero innate knowledge.”
What happened: The announcement describes a pretrained and fine-tuned language model combined with a specialized planning engine and message filters. This supports the factual claim that the system's design goes beyond enlarging a single general model. (Meta AI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Bottom line: From the outset Large Language Models like GPT-3 have great at generating surrealist prose, and they can beat a lot of benchmarks, but they are not (and may never be) great tech for reliably inferring user intent from what users say.”
What happened: The study tests implied meaning rather than fluent text production and finds substantial shortcomings even in adapted models. It supports the ordinary reading that dependable contextual understanding remained a weakness, while acknowledging meaningful improvements. (Ruis et al., arXiv, )
Assessment: SupportedSaid
“LLMs are unruly beasts; nobody knows how to make them refrain 100% of time from insulting users, giving bad advice, or just plain making stuff up.”
What happened: OpenAI described reliable refusal of unsafe instructions as unsolved despite improvements from human-feedback training. Its limitations section also reports remaining toxic outputs and fabricated facts. (OpenAI, )
Assessment: ContradictedSaid
“It's worth asking since (a) Amazon is apparently greatly reducing Alexa's staff, (b) hasn't added significant features in years, and (c) seems clumsy and limited compared to GPT-3.”
What happened: The dated launch account describes multiparty interaction and speech directed at Alexa without repeatedly saying the wake word. This is a substantive interaction feature within the claimed period, although it does not establish GPT-3-like open conversation. (Amazon Science, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Pitch perfect and utterly bogus imitations of science and math, presented as the real thing.”
What happened: The preserved response presents a purported paradox with formal sections and a solution, treating the sum of two 0.5 terms as 0.4 to reach this conclusion. Direct arithmetic gives 1, so the mathematical exposition is invalid despite its factual presentation. (Ernest Davis and Andrew Sundstrom, NYU-hosted experiment record, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
What happened: The release compares optimized and baseline implementations of open text generators, including BLOOM and OPT, using dollars per million generated tokens. Its cost-sensitive benchmarks report substantial reductions, directly supporting cheaper attainable inference before the post. (Microsoft DeepSpeed, )
Assessment: SupportedSaid
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
What happened: The experiments establish that alternatives can produce convincing election-disinformation content. For ten articles daily over a month, the cost section estimates $0.30 using Gemini or $9 using hosted Llama 3 70B, compared with $4,500 for the cited human-writing service; these are generation-stage estimates, not total campaign budgets. (PLOS ONE, )
Assessment: SupportedSaid
“Scaling here was largely AWOL; even the best model was only at 80.6%, and for most models, scaling had at best a neglible effect.”
What happened: The best overall mean is 80.6% with 30 examples; Section 4.1 reports near-zero or decreasing slopes for many families, with only Cohere, T0 and InstructGPT-3 performing best at their largest tested size. The same source qualifies the headline: structured prompts average 81.7%, and Table 58 includes an individual-template score of 82.8%. (arXiv; Laura Ruis and colleagues, )
Assessment: SupportedSaid
“Scaling is an empirical observation, not a guaranteed, continued law of nature.”
What happened: The foundational paper presents empirical relationships and explicitly qualifies their theoretical status. Appendix C and section 6.3 discuss limits to trusting or extending those relationships, supporting the absence of an indefinite guarantee. (Jared Kaplan and colleagues, arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“having a machine look at a radiology scan is very different from replacing the entire job of a radiologist”
What happened: The society's contemporary statement describes patient advocacy, safety, treatment and communication responsibilities alongside diagnosis. These substantive duties support the distinction between interpreting images and replacing the whole clinical role. (European Society of Radiology; Insights into Imaging, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It’s way too much charity to deep learning to claim that there is any kind of direct link.”
What happened: The author's account describes mapping, correlations and effects of changing model depth and attention. Inferring that these do not establish identical biological mechanisms is the evaluator's assessment, not an explicit concession in this quotation. (Charlotte Caucheteux, reproduced by Thread Reader, )
Assessment: SupportedSaid
“So if you have a correlation of R = .5, you actually “explain” (really, just “predict”) only 25% of the variance—which means fully three-quarters (not half) of the variability remains unexplained.”
What happened: The paragraph ties this statement to R = 0.50. Squaring that coefficient yields 25% accounted for and 75% remaining. (Scientific Reports, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Of course we see the same thing over and over in language systems, too.”
What happened: The surrounding results describe recurrent false-premise acceptance, excessive hedging, and constraint failures despite strong language-task performance. These directly match the behavioral pattern Marcus describes. (arXiv; OpenAI research authors, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“There was a lot of bluster (we will 100x the world’s productivity) but no road map.”
What happened: Tesla supplied tentative delivery timing. The same transcript outlines prototype improvement followed by factory deployment and progressively broader uses. (Tesla, transcribed by Stock Analysis, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Sooner or later he will recognize that humanoid robots, too, pose a whole set of challenges that lie well beyond our current grasp.”
What happened: Google's robotics research head describes unresolved real-world adaptation and understanding, with helpful robots still limited to restricted kitchen tasks and explicit safety controls. Applying these shared robotics limitations to general household humanoids is an inference, rather than a direct humanoid safety test. (Google, )
Assessment: SupportedSaid
“The real problem w robots has never been demos, anyway (there have been many, for decades); it’s making robots in robust in the real world.”
What happened: SRI dates Shakey's development to 1966–1972 and describes integrated perception, reasoning, navigation, and task execution. The endpoint records demonstrated robotic capabilities roughly five decades before the post. (SRI International, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Over and over, the field of AI has discovered otherwise. Guy van den Broeck’s study is a perfect example of this.”
What happened: Zhang and colleagues, including van den Broeck, found near-perfect performance within a training distribution but substantial failures under different sampling of the same logical problem space. Their analysis attributes this to statistical features rather than acquisition of the intended reasoning procedure. (arXiv, )
Assessment: UnresolvedSaid