Supported claims, page 9 of 9
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Supported claims (832)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 9 of 9: claims 801 to 832 of 832, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
“Sooner or later he will recognize that humanoid robots, too, pose a whole set of challenges that lie well beyond our current grasp.”
What happened: Google's robotics research head describes unresolved real-world adaptation and understanding, with helpful robots still limited to restricted kitchen tasks and explicit safety controls. Applying these shared robotics limitations to general household humanoids is an inference, rather than a direct humanoid safety test. (Google, )
Assessment: SupportedSaid
“The real problem w robots has never been demos, anyway (there have been many, for decades); it’s making robots in robust in the real world.”
What happened: SRI dates Shakey's development to 1966–1972 and describes integrated perception, reasoning, navigation, and task execution. The endpoint records demonstrated robotic capabilities roughly five decades before the post. (SRI International, )
Assessment: SupportedSaid
“Over and over, the field of AI has discovered otherwise. Guy van den Broeck’s study is a perfect example of this.”
What happened: Zhang and colleagues, including van den Broeck, found near-perfect performance within a training distribution but substantial failures under different sampling of the same logical problem space. Their analysis attributes this to statistical features rather than acquisition of the intended reasoning procedure. (arXiv, )
Assessment: SupportedSaid
“Over and over I have stressed that you could have a good model of engineering without being a good model of humans, or vice versa.”
What happened: Faggella's interview report describes Marcus contrasting evolved human memory with more reliable computer memory. It records an earlier example of engineering desirability differing from human fidelity; the quotation preserves the source's apparent typo. (Emerj; Dan Faggella, )
Assessment: SupportedSaid
“But the smart money is still on being cautious about this, and in recognizing that what the Facebook researchers said in 2019 is still the case.”
What happened: Reducing misleading correlations improved independent-test performance for BERT and stronger RoBERTa and ALBERT models, supporting continued susceptibility in 2022. The same experiments found better generalization from larger models and debiasing, which limits any claim that no genuine progress occurred. (arXiv; ACL 2022 author manuscript, )
Assessment: SupportedSaid
“On #2, the answer is mixed. They do well on some inputs (like astronaut rides horse) but more poorly on others (like horse rides astronaut, which I discussed in an earlier post).”
What happened: The study's human judgments show some successful matching alongside widespread relational failures. Its results were reported by July; the precise image-generation dates are not supplied. (arXiv; Colin Conwell and Tomer Ullman, )
Assessment: SupportedSaid
“The dirty secret in current AI is that for all the bloviating about how current systems have mastered human language, they are still really weak on compositionality, which most linguists (since Frege, over a century ago) would agree is at the very core of how language works.”
What happened: Winoground found large human-model gaps across several vision-language model families on image-caption matching that depends on composition. The finding supports a broad weakness within the tested contemporary systems. (arXiv; Thrush and colleagues, )
Assessment: SupportedSaid
“The progress that Alexander is vaunting shows that the system did better on a handful of items than DALL-E2 (which was not even state of the art at compositionality when it appeared), but not that the newer system (Imagen) systematically groks compositionality”
What happened: The page specifies five prompts, reports DALL-E 2 at 0/5 and Imagen at 3/5 under Alexander's scoring, and contains no systematic retest of earlier successes. Alexander also expressly declines to claim full mastery, so the methodological limitation does not establish that he claimed otherwise. (Astral Codex Ten, )
Assessment: SupportedSaid
“This should be a piece of cake, but for currently popular systems, it’s not.”
What happened: The group score requires all relevant caption-image comparisons to be correct; every tested model scored below its random baseline. Some models exceeded chance on the less demanding text score, but remained far behind humans and did not reliably solve the complete matching task. (arXiv; Thrush and colleagues, )
Assessment: SupportedSaid
“Driverless car demos have been around for decades, but making them work reliably has proven far harder than most people realized.”
What happened: The report records a direct industry admission and describes executives narrowing expectations because of technical complexity. A readable primary transcript was not obtained, so contemporary reporting is used. (WIRED, )
Assessment: SupportedSaid
“Performance hovered at around 75% in the best case, when the robots had a limited number of options to consider.”
What happened: The reported overall execution rates are 74% in the mock kitchen and 60% in the real kitchen. Thus the better overall setting is approximately 75%; planning is a separate measure. (arXiv; Michael Ahn and colleagues, )
Assessment: SupportedSaid
“First, the language technology that the new system relies on is well-known to be problematic and second it is likely to be even more problematic in the context of robots.”
What happened: PaLM's own announcement acknowledges an existing research literature on risks and describes analyses of bias and model outputs. This directly supports recognition of problems in the relevant technology. (Google Research, )
Assessment: SupportedSaid
“it’s even less clear how it would work with words like no and not (a major sore spot for current neural networks).”
What happened: The study reports substantial RoBERTa performance losses on subsets where negation matters: QNLI macro F1 is 0.67 for important negations versus 0.92 for unimportant negations. These are dataset-subset comparisons, not matched sentence pairs; the findings were already reported in the March preprint. (Association for Computational Linguistics, )
Assessment: SupportedSaid
“Since then, the results on benchmarks have gotten better, but there’s still something missing.”
What happened: PaLM's contemporary report documents continued benchmark improvements. This supports the improvement component, without establishing that benchmark success implies general understanding. (arXiv; Chowdhery et al., )
Assessment: SupportedSaid
“noncanonical reorderings such as these remain problematic for current AI approaches.)”
What happened: Winoground requires matching two images to captions containing the same words in different orders. Its poor group scores across contemporary vision-language models support substantial difficulty preserving the changed relationships. (Thrush et al., CVPR 2022 / arXiv, )
Assessment: SupportedSaid
“Because it shows that the people who really claim to be working on full self-driving (which Tesla waffles on1) are having plenty of problems of their own.”
What happened: The record confirms repeated ADS crash involvement in the initial reporting period. Its abstract defines reportability by ADS use within 30 seconds of a crash, not by a finding of ADS fault. (Transportation Research Board, summarizing NHTSA, )
Assessment: SupportedSaid
“Level 5 has to handle it all; everyday stopped vehicles, weird outliers and so forth. We’re just not there.”
What happened: The contemporaneous report directly addresses the claimed industry-wide state. Reputable reporting is used because no readable primary source establishing that complete industry-wide negative was obtained. (The Verge, )
Assessment: SupportedSaid
“Indeed, the NHTSA report data show that even some pretty ordinary circumstances, like avoiding stopped vehicles, are continuing to prove problematic.”
What happened: The surrounding summary identifies these as additional Autopilot collisions with stopped first-responder vehicles. Its incident list dates Orlando to August 2021, Petaluma to September 2021, and Desert Center to January 2022; the event date records the latest of these in-window incidents. (National Highway Traffic Safety Administration, )
Assessment: SupportedSaid
“The basic finding was: loads of things scale, but not at all ( as foreseen both by Kurenkov in his essay last year and in my much lampooned but basically accurate essay, Deep learning is hitting a wall).”
What happened: The original report directly states the coexistence Marcus describes. Sections 3.4.2 and 6.1 also explain that metric choice can conceal gradual progress and that some limitations may change with larger models or improved data. (Srivastava et al.; arXiv, )
Assessment: SupportedSaid
“The massive paper looks at scaling on many measures, and sees progress on some– but not others. The stuff that isn’t scaling is the “wall”.”
What happened: The report directly confirms improving and non-improving task scores across evaluated scales. Its discussion of smoother metrics limits what those flat scores imply about underlying capabilities. (Srivastava et al.; arXiv, )
Assessment: SupportedSaid
“so many of the unfortunate consequences of a webscraping/big data approach to AI are disproportionately borne by women and minorities.”
What happened: The audit reports removal rates of 42% for African American English and 32% for Hispanic-aligned English, against 6.2% for White-aligned English. This documents unequal exclusion from a web-scraped training corpus, using dialect as a proxy rather than identifying authors. (Dodge et al.; EMNLP 2021, arXiv version, )
Assessment: SupportedSaid
“I would be very surprised if that fundamental lack of reliability did not also hold true for GPT-4. Maybe it will take three paragraphs in a prompt rather than one to break it, but a big Transformer alone does not a a theory of the world create.”
What happened: OpenAI's launch-era report explicitly acknowledges both predicted failure classes and advises safeguards where reliability matters. It also reports substantial factuality improvement, which is compatible with Marcus's allowance for fewer errors. (OpenAI / arXiv, )
Assessment: SupportedSaid
“You said, for instance in 2015, that (truly) self-driving cars were two years away; you’ve pretty much said the same thing every year since. It still hasn’t happened.”
What happened: The original interview establishes the attributed prediction. Musk separately allowed additional time for regulatory approval, so the prediction concerned technical capability first. (Fortune, )
Assessment: SupportedSaid
“Even within perception 3D perception remains a challenge, and scene understanding is by no means solved.”
What happened: Testing fifteen common corruptions exposed substantial robustness weaknesses in leading 3D recognition models. The authors also report mitigation methods, so the result establishes remaining challenges rather than an absence of progress. (arXiv; Jiachen Sun and colleagues, )
Assessment: SupportedSaid
“But for the kinds of AI techniques that are currently popular, and which rely more on big data than on reasoning, it’s a very serious issue.”
What happened: The contemporary synthesis documents poor learning of rare classes and substantial limitations in conventional deep-learning methods. It also reviews mitigation methods, supporting a serious practical problem without implying that every method fails. (International Journal of Computer Vision; author manuscript rendered by ar5iv, )
Assessment: SupportedSaid
“And it’s why I absolutely guarantee you won’t be shipping level 5 self-driving cars this year or next, no matter what you might have told Chris Anderson at TED.”
What happened: The annual filing describes Tesla's offerings as driver assistance and says the driver remains responsible for controlling the vehicle. That capability does not meet Level 5. (Tesla, Inc., mirrored by AnnualReports.com, )
Assessment: SupportedSaid
“But you and I both know that they continue to be a major problem that as yet has no known robust solution.”
What happened: The survey identifies unresolved obstacles involving graph structure, differing tasks, and different kinds of distribution shift. It also describes progress, supporting the distinction between useful remedies and a general solution. (arXiv; Haoyang Li and colleagues, )
Assessment: SupportedSaid
“That’s where the deep understanding of language should live—but, despite the PR, it isn’t there.”
What happened: The original authors identify a category-level alignment weakness in the figure containing the horse-riding-an-astronaut prompt. This supports the contextual reading that reliable compositional instruction understanding was missing. (Google Research, Brain Team; copy hosted by 3DVAR, )
Assessment: SupportedSaid
“Success on a few problems with a particular strategy in no way guarantees that we can solve all problems in a similar way. It is sheer inductive folly to imagine otherwise, particular when the failure modes (unreliability, bizarre errors, failures in compositionality and discomprehension) have not changed since Fodor and Pinker pointed them out (in separate articles) in 1988.”
What happened: Tests involving SCAN and RobustFill found continuing difficulty combining learned components in new ways, even with modified Transformers. This supports persistence of compositional generalization problems while explicitly acknowledging progress. (arXiv; Kensen Shi and colleagues, )
Assessment: SupportedSaid
“DeepMind’s newest star, just unveiled, Gato, is capable of cross-modal feats never seen before in AI, but still, when you look in the fine print, remains stuck in the same land of unreliability, moments of brilliance coupled with absolute discomprehension:”
What happened: The launch paper reports one model spanning dialogue, captions, games and real robot control, including human-level or better performance on 23 Atari games, while acknowledging frequent poor dialogue responses. Its prior-work comparison supports novelty in that combined coverage, rather than novelty of multimodality itself. (arXiv; DeepMind, )
Assessment: SupportedSaid
“But it’s also pretty prone to surprising stumbles, like this uncomprehending reply to “a red cube on top of a blue cube”.”
What happened: The authors discuss the exact red-cube/blue-cube prompt in Figure 14 and report recurring object–attribute mix-ups in reconstructions in Figure 15. This supports a recurring compositional weakness in the original model, without establishing a population-wide error rate. (arXiv; OpenAI, )
Assessment: SupportedSaid
“The history missed is the the fact that many neural net enthusiasts, have argued against symbols for decades, and the reality that is missed is the above-documented fact that symbolic descriptions like “red cube on blue cube” still elude the state-of-the-art in 2022.)”
What happened: This dated scholarly account documents an explicit anti-symbolic strand well before the post. It is historical synthesis rather than a primary statement by an advocate. (Stanford Encyclopedia of Philosophy, )