Supported claims by Gary Marcus, page 6 of 6
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Supported claims by Gary Marcus (558)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 6 of 6: claims 501 to 558 of 558, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Observation
“Already we have seen minor abuses; plagiarism, for example, will never be the same.”
Our reading: Minor abuses of ChatGPT have already been seen, and plagiarism, for example, will never be the same.
No horizon Evidence as of
Limitation: The incident is documented through reporting and the professor's testimony, not a published disciplinary file.
Assessment: SupportedSaid
Observation
“one of the things that popped out to me is that the system still quickly lands in the land of hallucination that has so haunted ChatGPT.”
Our reading: Anthropic's Claude, despite its alignment focus, still quickly hallucinates like ChatGPT.
No horizon Evidence as of
Limitation: The comparison is informal and based on few prompts.
Assessment: SupportedSaid
Observation
“ChatGPT has proven itself to be both unreliable and untruthful. It makes boneheaded arithmetical errors, invents fake biographical details, bungles word problems, defines non-existent scientific phenomenon, stumble over arithmetic conversion, and on and on.”
Our reading: ChatGPT is unreliable and untruthful: it makes arithmetic errors, fabricates biographical details, bungles word problems, and invents scientific phenomena.
No horizon Evidence as of
Limitation: Selected demonstrations do not establish an overall error rate or intentional deception [e1, e2, e3, e4].
Source: Scientists, please don’t let your chatbots grow up to be co-authors
Assessment: SupportedSaid
Observation
“It could sometimes produce outputs that were correct and acceptable in these regards but not reliably.”
Our reading: ChatGPT could sometimes produce outputs that were correct and acceptable in the regards listed (counting, arithmetic, event order, physical reasoning, human psychology, fabrication, bias), but not reliably.
No horizon Evidence as of
Limitation: The evidence covers the seven failure types unevenly.
Source: Large Language Models like ChatGPT say The Darnedest Things
Assessment: SupportedSaid
Observation
“It made things up (figure 6). Its output, when released, exhibited sexist and racist biases (figure 7).”
Our reading: ChatGPT made things up and, at release, exhibited sexist and racist biases.
No horizon Evidence as of
Limitation: The bias examples came from adversarial prompts reported in the press, not from a systematic audit.
Source: Large Language Models like ChatGPT say The Darnedest Things
Assessment: SupportedSaid
Forecast
“That’s a huge commitment, 3.5 hours, but one thing that I think all of our speakers could agree on is that artificial intelligence, in whatever form it currently is, is about to have a huge impact on society.”
Our reading: AI, even in its current form, is about to have a huge impact on society.
Settled before its deadline Evidence as of
Limitation: Marcus supplied no fixed deadline or numerical definition of huge.
Source: An epic AI Debate—and why everyone should be at least a little bit worried about AI going into 2023
Assessment: SupportedSaid
Causal claim
“Driverless cars have taken many years longer than we were originally told; outliers (AKA edge cases) have, at least so far, prevented them from making the transition from demo to widely-available reality.”
Our reading: Driverless cars have taken years longer than promised, because edge cases have so far kept them from becoming widely available.
No horizon Evidence as of
Limitation: The result depends on reading widely available as broad access beyond restricted local services.
Assessment: SupportedSaid
Observation
“Large language models are not databases. They are glommers-together-of-bits-that don’t always belong together.”
Our reading: LLMs are not databases; they combine bits of text that don't always belong together.
No horizon Evidence as of
Limitation: The cited passages do not establish a general error rate across models or tasks.
Assessment: SupportedSaid
Forecast
“I guarantee that minds will be blown.”
Our reading: GPT-4 will impress people enormously when released.
Settled before its deadline Evidence as of
Limitation: The accounts are illustrative, not a representative opinion survey.
Assessment: SupportedSaid
Forecast
“It will often do what you want, sometimes not—and it will remain difficult to anticipate which in advance..”
Our reading: It will remain difficult to anticipate in advance when GPT-4 will or won't do what users want.
Settled before its deadline Evidence as of
Limitation: The finding is strongest for task correctness and settings where mistakes carry high costs.
Assessment: SupportedSaid
Forecast
“The system will still not be able to restrict its output to reliably following a shared set of human values around helpfulness, harmlessness, and truthfulness.”
Our reading: GPT-4 will not reliably follow human values of helpfulness, harmlessness and truthfulness.
Settled before its deadline Evidence as of
Limitation: The judgment depends on an ordinary, nonnumerical reading of reliability.
Assessment: SupportedSaid
Forecast
“GPT-4 will solve many of the individual specific items used in prior benchmarks, but still get tripped up, particularly in longer and more complex scenarios.”
Our reading: GPT-4 will solve many prior benchmark items but still fail, especially on longer, complex scenarios.
Settled before its deadline Evidence as of
Limitation: Benchmark success does not establish whether individual items were learned or memorized.
Assessment: SupportedSaid
Forecast
“As OpenAI has learned, bigger in many ways means better, with outputs more and more humanlike with each iteration. GPT-4 is going to be a monster.”
Our reading: Because scale improves output in many ways, GPT-4 will be very powerful.
Settled before its deadline Evidence as of
Limitation: The decisive evaluation was conducted by the model developer.
Assessment: SupportedSaid
Forecast
“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”
Our reading: GPT-4 will make a significant number of hard-to-predict stupid errors.
Settled before its deadline Evidence as of
Limitation: The judgment depends on reading hard to predict as unexpectedly uneven performance, rather than requiring a direct prediction experiment.
Assessment: SupportedSaid
Forecast
““Alignment” between what humans want and what machines do will continue to be a critical, unsolved problem.”
Our reading: Alignment will remain a critical unsolved problem with GPT-4.
Settled before its deadline Evidence as of
Limitation: This assesses behavioral alignment in GPT-4, not every meaning of alignment.
Assessment: SupportedSaid
Observation
“In short, ChatGPT might sound like the Star Trek computer, but for now, you still can’t trust it.”
Our reading: ChatGPT cannot currently be trusted.
No horizon Evidence as of
Limitation: The conclusion concerns unverified factual reliance, not every possible use.
Assessment: SupportedSaid
Forecast
“Some of its advice will be head-scratchingly bad.”
Our reading: Some of GPT-4's advice will be very bad.
Settled before its deadline Evidence as of
Limitation: The study concerns a particular infectious-disease setting and does not measure all GPT-4 advice.
Assessment: SupportedSaid
Forecast
“When queried on medicine it will either resist answering (if there are aggressive guardrails) or occasionally spout plausible-sounding but dangerous nonsense.”
Our reading: On medical queries GPT-4 will either refuse (with aggressive guardrails) or occasionally produce plausible but dangerous nonsense.
Settled before its deadline Evidence as of
Limitation: The decisive evaluation concerns bloodstream-infection management and the tested GPT-4 version.
Assessment: SupportedSaid
Observation
“Although ChatGPT can write about anything, it is also easily confused.”
Our reading: ChatGPT is easily confused.
No horizon Evidence as of
Limitation: The arithmetic demonstrations were selected to probe weaknesses.
Assessment: SupportedSaid
Forecast
“It will remain turbocharged pastiche generator, and a fine tool for brainstorming, and for first drafts, but not trustworthy general intelligence.”
Our reading: GPT-4 will be a pastiche generator useful for brainstorming and drafts but not trustworthy general intelligence.
Settled before its deadline Evidence as of
Limitation: The experiment evaluates people working with GPT-4 on consulting tasks, not dependable performance across all general tasks.
Assessment: SupportedSaid
Forecast
“GPT-4 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
Our reading: GPT-4 will be reckless and hard to control like its predecessors.
Settled before its deadline Evidence as of
Limitation: Adversarial test results do not establish an ordinary-use failure rate.
Assessment: SupportedSaid
Observation
“Alas, the genie can no longer be stuffed back in the bottle.”
Our reading: Misinformation-capable LLM technology can no longer be contained or withdrawn.
No horizon Evidence as of
Limitation: Shows availability during the period, not long-term containment.
Source: AI's Jurassic Park moment
Assessment: SupportedSaid
Observation
“Large language models lack mechanisms for verifying truth; we need to find new ways to integrate them with the tools of classical AI, such as databases, webs of knowledge, and reasoning.”
Our reading: LLMs lack mechanisms for verifying truth.
No horizon Evidence as of
Limitation: The result depends on distinguishing standalone models from augmented systems.
Source: AI's Jurassic Park moment
Assessment: SupportedSaid
Observation
“For another, chatGPT, just released by OpenAI, is more or less just as capable of producing similar nonsense, such as instant essays on adding wood chips to breakfast cereal.”
Our reading: ChatGPT is about as capable as Galactica of producing nonsense misinformation.
No horizon Evidence as of
Limitation: The packet found no head-to-head test of ChatGPT and Galactica, so 'more or less just as capable' is judged qualitatively, not measured.
Source: AI's Jurassic Park moment
Assessment: SupportedSaid
Causal claim
“GPT doesn’t talk randomly, because it’s pastiching things actual people said.”
Our reading: GPT's output is not random, because it recombines things real people have said.
No horizon Evidence as of
Limitation: The result depends on reading pastiching as learned recombination rather than literal copying.
Source: How come GPT can seem so brilliant one minute and so breathtakingly dumb the next?
Assessment: SupportedSaid
Observation
“It could not be immediately applied to the challenge of, say, customer service or guiding the actions of a domestic robot, or well, almost anything else, really.”
Our reading: Cicero cannot be immediately applied to tasks such as customer service or domestic robotics, or almost anything besides blitz Diplomacy.
No horizon Evidence as of
Limitation: The evidence is a developer capability statement, not an independent transfer benchmark.
Source: What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Assessment: SupportedSaid
Observation
“Strikingly, and in opposition to much of the Zeitgeist, Cicero relies quite heavily on hand-crafting, both in the data sets, and in the architecture; in this sense it is in many ways more reminiscent of classical “Good Old Fashioned AI” than deep learning systems that tend to be less structured, and less customized to particular problems.”
Our reading: Against much of the Zeitgeist, Cicero relies quite heavily on hand-crafting in both its data sets and its architecture, and in this sense it is in many ways more reminiscent of classical Good Old Fashioned AI than of deep learning systems, which tend to be less structured and less customized to particular problems.
No horizon Evidence as of
Limitation: Quite heavily is assessed through substantive design choices, not a measured percentage of effort or performance.
Source: What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Assessment: SupportedSaid
Observation
“Cicero makes extensive use of machine learning, but is hardly a poster child for simply making ever bigger models (so-called “scaling maximalism”), nor for the currently popular view of “end-to-end” machine learning of in which some single general learning algorithm applies across the board, with little internal structure and zero innate knowledge.”
Our reading: Cicero makes extensive use of machine learning but is hardly a poster child for scaling maximalism or for the popular end-to-end view of machine learning with little internal structure and zero innate knowledge.
No horizon Evidence as of
Limitation: The architecture alone does not establish how much each component contributed to performance.
Source: What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Assessment: SupportedSaid
Observation
“Bottom line: From the outset Large Language Models like GPT-3 have great at generating surrealist prose, and they can beat a lot of benchmarks, but they are not (and may never be) great tech for reliably inferring user intent from what users say.”
Our reading: LLMs like GPT-3 are not great technology for reliably inferring user intent from what users say.
No horizon Evidence as of
Limitation: The principal evidence is a research benchmark, not a representative household deployment study.
Assessment: SupportedSaid
Observation
“Pitch perfect and utterly bogus imitations of science and math, presented as the real thing.”
Our reading: The Galactica examples Andrew Sundstrom shared are pitch-perfect but utterly bogus imitations of science and math, presented as the real thing.
No horizon Evidence as of
Limitation: Support depends on including the companion examples expressly linked in the post.
Source: A Few Words About Bullshit
Assessment: SupportedSaid
Observation
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
Our reading: Knockoffs of GPT-3 are getting cheaper and cheaper.
No horizon Evidence as of
Limitation: The result depends on including inference operating costs in cheaper.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: SupportedSaid
Causal claim
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
Our reading: Cheaper GPT-3 knockoffs will drive the cost of generating misinformation toward zero.
No horizon Evidence as of
Limitation: Support depends on interpreting zero as negligible marginal cost.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: SupportedSaid
Observation
“Scaling here was largely AWOL; even the best model was only at 80.6%, and for most models, scaling had at best a neglible effect.”
Our reading: On the Ruis et al. pragmatics task, the best model reached only 80.6%, and for most models scaling had at best a negligible effect.
No horizon Evidence as of
Limitation: Support depends on reading 80.6% as the overall mean, not the maximum across every prompt configuration.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: SupportedSaid
Observation
“Scaling is an empirical observation, not a guaranteed, continued law of nature.”
Our reading: AI scaling is an empirical observation, not a law of nature that is guaranteed to keep holding.
No horizon Evidence as of
Limitation: An empirical relationship can remain useful and predictive without being guaranteed.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: SupportedSaid
Observation
“Of course we see the same thing over and over in language systems, too.”
Our reading: Language systems repeatedly show the same mix of fluent output and lack of understanding.
No horizon Evidence as of
Limitation: Support concerns the recurring behavioral pattern, not the exact anecdotes.
Assessment: SupportedSaid
Observation
“Sooner or later he will recognize that humanoid robots, too, pose a whole set of challenges that lie well beyond our current grasp.”
Our reading: Humanoid robots pose a set of challenges well beyond current technology's grasp.
No horizon Evidence as of
Limitation: The decisive account addresses shared robotics limitations rather than directly testing a humanoid in a home.
Source: Sub-Optimal
Assessment: SupportedSaid
Observation
“Over and over, the field of AI has discovered otherwise. Guy van den Broeck’s study is a perfect example of this.”
Our reading: The AI field has repeatedly found that doing well on a benchmark designed for X does not mean a system has actually solved X.
No horizon Evidence as of
Limitation: The studies cover selected logical-reasoning and language-inference tasks, not all AI benchmarks or models [e4, e5].
Assessment: SupportedSaid
Observation
“But the smart money is still on being cautious about this, and in recognizing that what the Facebook researchers said in 2019 is still the case.”
Our reading: State-of-the-art models still learn to exploit spurious statistical patterns rather than learning meaning flexibly as humans do, so benchmark gains warrant caution.
No horizon Evidence as of
Limitation: The evidence covers selected NLP models and tasks, not every leading model available by September 23, 2022 [e6, e7].
Assessment: SupportedSaid
Observation
“On #2, the answer is mixed. They do well on some inputs (like astronaut rides horse) but more poorly on others (like horse rides astronaut, which I discussed in an earlier post).”
Our reading: Image synthesis systems only inconsistently match their linguistic input to the images produced, doing well on some prompts and poorly on others.
No horizon Evidence as of
Limitation: The strongest experimental evidence covers DALL-E 2.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: SupportedSaid
Observation
“The dirty secret in current AI is that for all the bloviating about how current systems have mastered human language, they are still really weak on compositionality, which most linguists (since Frege, over a century ago) would agree is at the very core of how language works.”
Our reading: Current AI systems are still very weak at compositionality, which is central to how language works.
No horizon Evidence as of
Limitation: The result depends on reading current AI as prominent contemporary neural systems rather than every AI architecture.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: SupportedSaid
Observation
“The progress that Alexander is vaunting shows that the system did better on a handful of items than DALL-E2 (which was not even state of the art at compositionality when it appeared), but not that the newer system (Imagen) systematically groks compositionality”
Our reading: Alexander's test shows only that Imagen beat DALL-E 2 on a handful of items, not that Imagen systematically understands compositionality.
No horizon Evidence as of
Limitation: The scores reflect Alexander's judgments, not independent verification [e1].
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: SupportedSaid
Observation
“This should be a piece of cake, but for currently popular systems, it’s not.”
Our reading: Currently popular vision-language systems cannot reliably tell apart captions that use the same words in a different order.
No horizon Evidence as of
Limitation: The result applies to the evaluated models and task.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: SupportedSaid
Observation
“Performance hovered at around 75% in the best case, when the robots had a limited number of options to consider.”
Our reading: PaLM-SayCan's performance was around 75% at best, when the robot had a limited number of options.
No horizon Evidence as of
Limitation: Best case is interpreted as the best overall setting.
Assessment: SupportedSaid
Observation
“Since then, the results on benchmarks have gotten better, but there’s still something missing.”
Our reading: Since 2019 large language models have done better on benchmarks, but something is still missing from them.
No horizon Evidence as of
Limitation: TruthfulQA deliberately tests susceptibility to misconceptions; its percentages are not general error rates for all language use.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: SupportedSaid
Observation
“noncanonical reorderings such as these remain problematic for current AI approaches.)”
Our reading: Current AI approaches still struggle with noncanonical word orders such as 'the mat is on the cat'.
No horizon Evidence as of
Limitation: The evidence concerns tested vision-language models, not every contemporary AI approach [e1].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: SupportedSaid
Observation
“Because it shows that the people who really claim to be working on full self-driving (which Tesla waffles on1) are having plenty of problems of their own.”
Our reading: Companies working on full (Level 5) self-driving are having plenty of problems, per NHTSA ADS data.
No horizon Evidence as of
Limitation: Support is for repeated crash involvement, not technical fault or comparative danger.
Assessment: SupportedSaid
Observation
“Level 5 has to handle it all; everyday stopped vehicles, weird outliers and so forth. We’re just not there.”
Our reading: Level 5 self-driving that handles everything has not been achieved.
No horizon Evidence as of
Limitation: The industry-wide finding relies on contemporaneous reporting; the primary permit corroborates a particular deployment.
Assessment: SupportedSaid
Observation
“Indeed, the NHTSA report data show that even some pretty ordinary circumstances, like avoiding stopped vehicles, are continuing to prove problematic.”
Our reading: NHTSA data show even ordinary circumstances like avoiding stopped vehicles remain problematic for automated driving systems.
No horizon Evidence as of
Limitation: The decisive source is NHTSA's contemporaneous investigation, not a verified extraction of the specific June 15 summary tables.
Assessment: SupportedSaid
Observation
“The basic finding was: loads of things scale, but not at all ( as foreseen both by Kurenkov in his essay last year and in my much lampooned but basically accurate essay, Deep learning is hitting a wall).”
Our reading: BIG-bench shows many capabilities improve with scale but not all, which bears out Marcus's 'Deep learning is hitting a wall' essay.
No horizon Evidence as of
Limitation: The finding does not establish a permanent scaling barrier or what happens beyond the tested model sizes [e1].
Assessment: SupportedSaid
Observation
“The massive paper looks at scaling on many measures, and sees progress on some– but not others. The stuff that isn’t scaling is the “wall”.”
Our reading: The BIG-bench paper looked at scaling on many measures and found progress on some but not others, and the measures that did not scale in it are what Marcus calls the 'wall'.
No horizon Evidence as of
Limitation: The finding applies to the evaluated models, scales and metrics.
Assessment: SupportedSaid
Observation
“so many of the unfortunate consequences of a webscraping/big data approach to AI are disproportionately borne by women and minorities.”
Our reading: Many of the harmful consequences of the webscraping/big-data approach to AI fall disproportionately on women and minorities.
No horizon Evidence as of
Limitation: Support depends on reading many qualitatively and including representational and data-exclusion harms.
Assessment: SupportedSaid
Forecast
“I would be very surprised if that fundamental lack of reliability did not also hold true for GPT-4. Maybe it will take three paragraphs in a prompt rather than one to break it, but a big Transformer alone does not a a theory of the world create.”
Our reading: GPT-4 will very likely still have a fundamental lack of reliability, though it may take longer prompts to break.
Settled before its deadline Evidence as of
Limitation: Support applies to recurring factual and reasoning unreliability, not a claim that GPT-4 has no useful capabilities.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: SupportedSaid
Observation
“You said, for instance in 2015, that (truly) self-driving cars were two years away; you’ve pretty much said the same thing every year since. It still hasn’t happened.”
Our reading: Musk said in 2015 that truly self-driving cars were two years away and has pretty much said the same thing every year since, and truly self-driving cars still had not arrived.
No horizon Evidence as of
Limitation: Several predictions are documented through contemporary reporting because readable original transcripts were unavailable.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Observation
“Even within perception 3D perception remains a challenge, and scene understanding is by no means solved.”
Our reading: 3D perception and scene understanding remain unsolved for current AI.
No horizon Evidence as of
Limitation: The studies cover particular tasks rather than every form of perception or scene understanding.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Observation
“But for the kinds of AI techniques that are currently popular, and which rely more on big data than on reasoning, it’s a very serious issue.”
Our reading: The long-tail problem of rare events is a very serious issue for the currently popular AI techniques, which rely more on big data than on reasoning.
No horizon Evidence as of
Limitation: Evidence is strongest for visual recognition.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Forecast
“And it’s why I absolutely guarantee you won’t be shipping level 5 self-driving cars this year or next, no matter what you might have told Chris Anderson at TED.”
Our reading: Tesla will not ship level 5 self-driving cars in 2022 or 2023.
Deadline passed Evidence as of
Limitation: The assessment concerns delivered capability, not whether Tesla sold an option named Full Self-Driving.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Observation
“But you and I both know that they continue to be a major problem that as yet has no known robust solution.”
Our reading: Outliers remain a major AI problem with no known robust solution.
No horizon Evidence as of
Limitation: The absence of a broadly validated solution is an inference from the inspected public research record, not proof that every possible solution was absent.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Observation
“That’s where the deep understanding of language should live—but, despite the PR, it isn’t there.”
Our reading: Imagen lacks deep understanding of language despite Google's PR.
No horizon Evidence as of
Limitation: The result depends on interpreting deep understanding through the role-and-relationship behavior discussed in the post.
Source: Horse rides astronaut