Claims checked against evidence, page 14 of 15
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (1432)
Each of these claims was checked against dated evidence.
Page 14 of 15: claims 1301 to 1400 of 1432, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Gary Marcus Observation
“As I told NYT columnist Farhad Manjoo, ChatGPT, like earlier, related systems is “still not reliable, still doesn’t understand the physical world, still doesn’t understand the psychological world and still hallucinates.””
Our reading: ChatGPT is not reliable, does not understand the physical or psychological world, and hallucinates.
No horizon Evidence as of
Limitation: The result depends on treating understanding as practical reasoning competence.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“In retrospectives written in 2043, intellectual historians will conclude that there was an initial overemphasis on large language models, and a gradual but critical shift of the pendulum back to more structured systems with deeper comprehension.”
Our reading: In retrospectives written in 2043, intellectual historians will conclude that there was an initial overemphasis on large language models and a gradual but critical shift of the pendulum back to more structured systems with deeper comprehension.
Not yet due Evidence as of
Limitation: The forecast leaves the author population and degree of agreement unspecified.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Although GPT-4 will definitely seem smarter than its predecessors, its internal architecture remains problematic.”
Our reading: GPT-4 will seem smarter than predecessors but its architecture remains problematic.
No deadline stated Evidence as of
Limitation: The comparative evaluation was conducted by the model's developer.
Assessment: SupportedSaid
Gary Marcus Forecast
“GPT-4 will solve many of the individual specific items used in prior benchmarks, but still get tripped up, particularly in longer and more complex scenarios.”
Our reading: GPT-4 will solve many prior benchmark items but still fail, especially on longer, complex scenarios.
Settled before its deadline Evidence as of
Limitation: Benchmark success does not establish whether individual items were learned or memorized.
Assessment: SupportedSaid
Gary Marcus Forecast
“As OpenAI has learned, bigger in many ways means better, with outputs more and more humanlike with each iteration. GPT-4 is going to be a monster.”
Our reading: Because scale improves output in many ways, GPT-4 will be very powerful.
Settled before its deadline Evidence as of
Limitation: The decisive evaluation was conducted by the model developer.
Assessment: SupportedSaid
Gary Marcus Forecast
“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”
Our reading: GPT-4 will make a significant number of hard-to-predict stupid errors.
Settled before its deadline Evidence as of
Limitation: The judgment depends on reading hard to predict as unexpectedly uneven performance, rather than requiring a direct prediction experiment.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Which is to say it makes stuff up, regularly; a fair bit of what it is says simply isn’t true.”
Our reading: ChatGPT regularly makes things up and a fair amount of its output is false.
No horizon Evidence as of
Limitation: This does not mean fabrication was absent or rare.
Assessment: UnresolvedSaid
Gary Marcus Forecast
““Scaling” alone—building bigger and models until they absorb the entire internet — will prove useful, but only to a point.”
Our reading: Scaling alone will prove useful only up to a point.
No deadline stated Evidence as of
Limitation: The claim specifies no deadline, capability threshold, or operational test of trustworthy general intelligence.
Assessment: SupportedSaid
Gary Marcus Observation
“In short, ChatGPT might sound like the Star Trek computer, but for now, you still can’t trust it.”
Our reading: ChatGPT cannot currently be trusted.
No horizon Evidence as of
Limitation: The conclusion concerns unverified factual reliance, not every possible use.
Assessment: SupportedSaid
Gary Marcus Forecast
“Some of its advice will be head-scratchingly bad.”
Our reading: Some of GPT-4's advice will be very bad.
Settled before its deadline Evidence as of
Limitation: The study concerns a particular infectious-disease setting and does not measure all GPT-4 advice.
Assessment: SupportedSaid
Gary Marcus Forecast
“When queried on medicine it will either resist answering (if there are aggressive guardrails) or occasionally spout plausible-sounding but dangerous nonsense.”
Our reading: On medical queries GPT-4 will either refuse (with aggressive guardrails) or occasionally produce plausible but dangerous nonsense.
Settled before its deadline Evidence as of
Limitation: The decisive evaluation concerns bloodstream-infection management and the tested GPT-4 version.
Assessment: SupportedSaid
Gary Marcus Observation
“Although ChatGPT can write about anything, it is also easily confused.”
Our reading: ChatGPT is easily confused.
No horizon Evidence as of
Limitation: The arithmetic demonstrations were selected to probe weaknesses.
Assessment: SupportedSaid
Gary Marcus Forecast
“GPT-4 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
Our reading: GPT-4 will be reckless and hard to control like its predecessors.
Settled before its deadline Evidence as of
Limitation: Adversarial test results do not establish an ordinary-use failure rate.
Assessment: SupportedSaid
Gary Marcus Observation
“Alas, the genie can no longer be stuffed back in the bottle.”
Our reading: Misinformation-capable LLM technology can no longer be contained or withdrawn.
No horizon Evidence as of
Limitation: Shows availability during the period, not long-term containment.
Source: AI's Jurassic Park moment
Assessment: SupportedSaid
Gary Marcus Observation
“Large language models lack mechanisms for verifying truth; we need to find new ways to integrate them with the tools of classical AI, such as databases, webs of knowledge, and reasoning.”
Our reading: LLMs lack mechanisms for verifying truth.
No horizon Evidence as of
Limitation: The result depends on distinguishing standalone models from augmented systems.
Source: AI's Jurassic Park moment
Assessment: SupportedSaid
Gary Marcus Observation
“For another, chatGPT, just released by OpenAI, is more or less just as capable of producing similar nonsense, such as instant essays on adding wood chips to breakfast cereal.”
Our reading: ChatGPT is about as capable as Galactica of producing nonsense misinformation.
No horizon Evidence as of
Limitation: The packet found no head-to-head test of ChatGPT and Galactica, so 'more or less just as capable' is judged qualitatively, not measured.
Source: AI's Jurassic Park moment
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“they cost almost nothing to operate, and so they are on a path to reducing the cost of generating disinformation to zero.”
Our reading: Because generative AI systems like ChatGPT, Dall-E 2, and Lensa cost almost nothing to operate, they are on a path to reducing the cost of generating disinformation to zero.
No horizon Evidence as of
Limitation: The economically negligible reading of 'zero' is an interpretation.
Source: AI's Jurassic Park moment
Assessment: UnresolvedSaid
Gary Marcus Observation
“they can easily be automated to generate misinformation at unprecedented scale.”
Our reading: Generative AI systems like ChatGPT, Dall-E 2, and Lensa can easily be automated to generate misinformation at unprecedented scale.
No horizon Evidence as of
Limitation: The supplied experiment predates the November–December 2022 target period [e1].
Source: AI's Jurassic Park moment
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“If the website is flooded with worthless code examples, programmers will no longer go there, its database of over 30 million questions and answers will become untrustworthy, and the 14 year old website will die.”
Our reading: If Stack Overflow is flooded with worthless generated code, programmers will leave, its database will become untrustworthy, and the site will die.
No deadline stated Evidence as of
Only if: if the website is flooded with worthless AI-generated code examples
Source: AI's Jurassic Park moment
Assessment: SupportedSaid
Gary Marcus Causal claim
“GPT doesn’t talk randomly, because it’s pastiching things actual people said.”
Our reading: GPT's output is not random, because it recombines things real people have said.
No horizon Evidence as of
Limitation: The result depends on reading pastiching as learned recombination rather than literal copying.
Source: How come GPT can seem so brilliant one minute and so breathtakingly dumb the next?
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“When GPT sounds plausible, it is because every paraphrased bit that it pastes together is grounded in something that actual humans said, and there is often some vague (but often irrelevant) relationship between..”
Our reading: GPT sounds plausible because each piece it combines comes from something humans actually said, and the pieces are often loosely related.
No horizon Evidence as of
Limitation: The meanings of bit, paraphrase and grounded materially affect the test.
Source: How come GPT can seem so brilliant one minute and so breathtakingly dumb the next?
Assessment: SupportedSaid
Gary Marcus Observation
“It could not be immediately applied to the challenge of, say, customer service or guiding the actions of a domestic robot, or well, almost anything else, really.”
Our reading: Cicero cannot be immediately applied to tasks such as customer service or domestic robotics, or almost anything besides blitz Diplomacy.
No horizon Evidence as of
Limitation: The evidence is a developer capability statement, not an independent transfer benchmark.
Source: What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Assessment: SupportedSaid
Gary Marcus Observation
“Strikingly, and in opposition to much of the Zeitgeist, Cicero relies quite heavily on hand-crafting, both in the data sets, and in the architecture; in this sense it is in many ways more reminiscent of classical “Good Old Fashioned AI” than deep learning systems that tend to be less structured, and less customized to particular problems.”
Our reading: Against much of the Zeitgeist, Cicero relies quite heavily on hand-crafting in both its data sets and its architecture, and in this sense it is in many ways more reminiscent of classical Good Old Fashioned AI than of deep learning systems, which tend to be less structured and less customized to particular problems.
No horizon Evidence as of
Limitation: Quite heavily is assessed through substantive design choices, not a measured percentage of effort or performance.
Source: What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Assessment: UnresolvedSaid
Gary Marcus Observation
“As it turns out, the architecture of Cicero differs profoundly from most of what’s been talked about in recent years in AI.”
Our reading: Cicero's architecture differs profoundly from most AI approaches discussed in recent years.
No horizon Evidence as of
Limitation: Selected examples establish differences, not prevalence.
Source: What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Assessment: SupportedSaid
Gary Marcus Observation
“Cicero makes extensive use of machine learning, but is hardly a poster child for simply making ever bigger models (so-called “scaling maximalism”), nor for the currently popular view of “end-to-end” machine learning of in which some single general learning algorithm applies across the board, with little internal structure and zero innate knowledge.”
Our reading: Cicero makes extensive use of machine learning but is hardly a poster child for scaling maximalism or for the popular end-to-end view of machine learning with little internal structure and zero innate knowledge.
No horizon Evidence as of
Limitation: The architecture alone does not establish how much each component contributed to performance.
Source: What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“if Alexa could converse freely, user expectations would go through the roof, and mostly be unmeetable.”
Our reading: If Alexa could converse freely, user expectations would rise to a level that mostly could not be met.
No deadline stated Evidence as of
Only if: if Alexa could converse freely
Assessment: SupportedSaid
Gary Marcus Observation
“Bottom line: From the outset Large Language Models like GPT-3 have great at generating surrealist prose, and they can beat a lot of benchmarks, but they are not (and may never be) great tech for reliably inferring user intent from what users say.”
Our reading: LLMs like GPT-3 are not great technology for reliably inferring user intent from what users say.
No horizon Evidence as of
Limitation: The principal evidence is a research benchmark, not a representative household deployment study.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Turning LLMs into a product that controls your home and talks to you in a way that would be reliable enough to use at scale in millions of homes is still a long, long way away.”
Our reading: A reliable LLM-based conversational home-control product usable at scale in millions of homes is still a long, long way away.
No deadline stated Evidence as of
Limitation: Large-scale participation is company-reported.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Just like every other large language model I have seen.”
Our reading: Like Galactica, every other large language model the author has seen prevaricates (fabricates content) a lot.
No horizon Evidence as of
Limitation: False answers on deliberately challenging questions are not a direct estimate of fabrication in ordinary use.
Source: A Few Words About Bullshit
Assessment: SupportedSaid
Gary Marcus Observation
“Pitch perfect and utterly bogus imitations of science and math, presented as the real thing.”
Our reading: The Galactica examples Andrew Sundstrom shared are pitch-perfect but utterly bogus imitations of science and math, presented as the real thing.
No horizon Evidence as of
Limitation: Support depends on including the companion examples expressly linked in the post.
Source: A Few Words About Bullshit
Assessment: UnresolvedSaid
Gary Marcus Forecast
“High school students will love it, and use it to fool and intimidate (some of) their teachers.”
Our reading: High school students will love Galactica and will use it to fool and intimidate some of their teachers.
No deadline stated Evidence as of
Limitation: A search that finds no qualifying incidents cannot establish that none occurred.
Source: A Few Words About Bullshit
Assessment: UnresolvedSaid
Gary Marcus Forecast
“The problem is about to get much, much worse.”
Our reading: The problem of misinformation on Twitter is about to get much, much worse.
No deadline stated Evidence as of
Limitation: An increase in misinformation exposure could occur without an increase in posting volume.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“As the pace of machine-generated misinformation picks up, Twitter’s existing effort, Community Notes (aka Birdwatch), which is mostly done manually, by humans, is going to get left in the dust.”
Our reading: As machine-generated misinformation accelerates, Twitter's mostly manual Community Notes will be overwhelmed.
No deadline stated Evidence as of
Only if: as the pace of machine-generated misinformation picks up
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Gary Marcus Observation
“Large language models are without equal at generating misinformation, but they suck at detecting it.”
Our reading: LLMs are unmatched at generating misinformation but poor at detecting it.
No horizon Evidence as of
Limitation: No admitted comparison establishes unmatched misinformation generation during the target period.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Scaling won’t solve this.”
Our reading: Scaling LLMs will not solve their inability to reason about truth and detect misinformation.
No deadline stated Evidence as of
Limitation: Truthful answer generation and misinformation detection are related but distinct tasks.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: SupportedSaid
Gary Marcus Observation
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
Our reading: Knockoffs of GPT-3 are getting cheaper and cheaper.
No horizon Evidence as of
Limitation: The result depends on including inference operating costs in cheaper.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: SupportedSaid
Gary Marcus Causal claim
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
Our reading: Cheaper GPT-3 knockoffs will drive the cost of generating misinformation toward zero.
No horizon Evidence as of
Limitation: Support depends on interpreting zero as negligible marginal cost.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: SupportedSaid
Gary Marcus Observation
“Scaling here was largely AWOL; even the best model was only at 80.6%, and for most models, scaling had at best a neglible effect.”
Our reading: On the Ruis et al. pragmatics task, the best model reached only 80.6%, and for most models scaling had at best a negligible effect.
No horizon Evidence as of
Limitation: Support depends on reading 80.6% as the overall mean, not the maximum across every prompt configuration.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: SupportedSaid
Gary Marcus Observation
“Scaling is an empirical observation, not a guaranteed, continued law of nature.”
Our reading: AI scaling is an empirical observation, not a law of nature that is guaranteed to keep holding.
No horizon Evidence as of
Limitation: An empirical relationship can remain useful and predictive without being guaranteed.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Scaling maximalism is an interesting hypothesis, but I stand by my prediction that it won’t get us to AGI.”
Our reading: Scaling maximalism will not get us to AGI.
No deadline stated Evidence as of
Limitation: This is not a finding that AGI is absent everywhere; it is a finding that the reviewed evidence does not settle this forecast.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“Any one of them would mean we need a real paradigm shift, if we are to get to AGI.”
Our reading: If any one of the three failure modes (not enough data, not enough compute, tasks that don't scale) holds, reaching AGI will need a real paradigm shift.
No deadline stated Evidence as of
Only if: if any of the three failure modes (data, compute, non-scaling tasks) holds
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“His hypothesis, now generally called scaling maximalism, remains extremely popular, in no small part because bigger and bigger models have indeed continued to do ever more impressive things.”
Our reading: Scaling maximalism remains extremely popular, in no small part because bigger and bigger AI models have continued to do ever more impressive things.
No horizon Evidence as of
Limitation: The survey concerns NLP researchers, not all AI researchers, companies, investors, or the public.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Gary Marcus Observation
“As I have long argued, capturing this without cognitive models and common sense is really hard.”
Our reading: Capturing the pragmatic implications of language without cognitive models and common sense is really hard.
No horizon Evidence as of
Limitation: The finding does not deny the demonstrated difficulty.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: SupportedSaid
Gary Marcus Observation
“Of course we see the same thing over and over in language systems, too.”
Our reading: Language systems repeatedly show the same mix of fluent output and lack of understanding.
No horizon Evidence as of
Limitation: Support concerns the recurring behavioral pattern, not the exact anecdotes.
Assessment: UnresolvedSaid
Gary Marcus Observation
“It’s also, as ever, stuck in the liminal space between superintelligence and super-lost.”
Our reading: Make a Video, as with AI systems before it, is caught between very impressive output and basic errors, such as giving a cat a human hand or a painting that never changes.
No horizon Evidence as of
Limitation: The original moving clips were not independently inspected in this assessment.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“This is the uncanny valley we are in, and one that we occupy for a while.”
Our reading: AI systems such as video, image, and language generators are now in an uncanny valley of impressive output with basic errors, and will stay there for a while.
No deadline stated Evidence as of
Limitation: This outcome does not mean the observed persistence is unsupported; it reflects the missing duration and resolution threshold.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“We in AI have a long long way to go. Data and compute alone won’t be enough.”
Our reading: AI has a long way to go, and data and compute alone will not be enough.
No horizon Evidence as of
Limitation: Benchmark performance covers only part of the capability claim.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Gary Marcus Observation
“Musk barely even acknowledged the problem, and certainly didn’t lay out any sort of credible strategy for attacking it..”
Our reading: Musk barely acknowledged the common-sense problem and laid out no credible strategy for addressing it.
No horizon Evidence as of
Limitation: An absence claim requires stronger coverage than targeted keyword searches.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The reality is that AI needs genuine, paradigm-shift level innovation. Simply building ever bigger neural networks won’t cut it.”
Our reading: AI needs paradigm-shift innovation; simply scaling neural networks will not be enough.
No horizon Evidence as of
Limitation: The capability target and paradigm boundary are not operationally defined.
Source: Sub-Optimal
Assessment: ContradictedSaid
Gary Marcus Observation
“There was a lot of bluster (we will 100x the world’s productivity) but no road map.”
Our reading: The Optimus presentation contained bluster about productivity but no road map.
No horizon Evidence as of
Limitation: The delivery estimate was explicitly tentative [e1]. The evidence does not establish whether the proposed sequence or timing was later achieved.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Gary Marcus Observation
“Second, there was very little vision for how Tesla would build the cognitive part of the AI they will need, beyond the basics of motor control (which Boston Dynamics already does so well), nor much recognition about why robotics is so hard in the real world.”
Our reading: Tesla showed very little vision for building the cognitive AI Optimus needs beyond motor control, and little recognition of why real-world robotics is hard.
No horizon Evidence as of
Limitation: The cited notes are secondary and include automated transcription.
Source: Sub-Optimal
Assessment: SupportedSaid
Gary Marcus Observation
“Sooner or later he will recognize that humanoid robots, too, pose a whole set of challenges that lie well beyond our current grasp.”
Our reading: Humanoid robots pose a set of challenges well beyond current technology's grasp.
No horizon Evidence as of
Limitation: The decisive account addresses shared robotics limitations rather than directly testing a humanoid in a home.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Gary Marcus Observation
“Musk is almost never right about deadlines.”
Our reading: Elon Musk's deadlines for technology projects are almost never met.
No horizon Evidence as of
Limitation: The usable evidence is concentrated on Tesla.
Source: Top 5 reasons Tesla’s Optimus announcement may turn out be a dud
Assessment: SupportedSaid
Gary Marcus Observation
“Over and over, the field of AI has discovered otherwise. Guy van den Broeck’s study is a perfect example of this.”
Our reading: The AI field has repeatedly found that doing well on a benchmark designed for X does not mean a system has actually solved X.
No horizon Evidence as of
Limitation: The studies cover selected logical-reasoning and language-inference tasks, not all AI benchmarks or models [e4, e5].
Assessment: SupportedSaid
Gary Marcus Observation
“But the smart money is still on being cautious about this, and in recognizing that what the Facebook researchers said in 2019 is still the case.”
Our reading: State-of-the-art models still learn to exploit spurious statistical patterns rather than learning meaning flexibly as humans do, so benchmark gains warrant caution.
No horizon Evidence as of
Limitation: The evidence covers selected NLP models and tasks, not every leading model available by September 23, 2022 [e6, e7].
Assessment: UnresolvedSaid
Gary Marcus Observation
“On #1, the answer is a clear yes; only highly trained human artists could do better.”
Our reading: Image synthesis systems generate high quality images that only highly trained human artists could surpass.
No horizon Evidence as of
Limitation: Image quality can vary by task and evaluator.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: SupportedSaid
Gary Marcus Observation
“On #2, the answer is mixed. They do well on some inputs (like astronaut rides horse) but more poorly on others (like horse rides astronaut, which I discussed in an earlier post).”
Our reading: Image synthesis systems only inconsistently match their linguistic input to the images produced, doing well on some prompts and poorly on others.
No horizon Evidence as of
Limitation: The strongest experimental evidence covers DALL-E 2.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Nine more tries, and only one very marginal success (top right corner):”
Our reading: Stable Diffusion produced only one very marginal success in nine further attempts at a part-function prompt.
No horizon Evidence as of
Limitation: Success grading includes a subjective boundary.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Even “draw a white bicycle with green wheels”, which focuses purely on part-whole relationships without function or complex syntax, is problematic:”
Our reading: Stable Diffusion has trouble even with simple part-whole prompts such as a white bicycle with green wheels.
No horizon Evidence as of
Limitation: Failure to locate independent records does not show that Marcus's displayed results were inaccurate.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Negation is, as ever, a problem.”
Our reading: Text-to-image systems like Stable Diffusion continue to have problems with negation in prompts.
No horizon Evidence as of
Limitation: This is an evidence gap, not a finding that negation worked reliably.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Google cloaks its current work the trappings of science (article preprints with citations and graphs and so forth) but when it comes to community inspection, they’ve systematically opted out.”
Our reading: Google has systematically refused to let the scientific community inspect models such as Imagen and LaMDA.
No horizon Evidence as of
Limitation: The evidence distinguishes public availability from scientific access but does not fully document the latter.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Gary Marcus Observation
“Notwithstanding the rumors about Google Imagen, nobody has yet publicly demonstrated a machine that can relate the meanings of sentences to their parts the way a five-year-old child can.”
Our reading: No one, Imagen included, has publicly shown a machine that relates sentence meanings to their parts as well as a five-year-old.
No horizon Evidence as of
Limitation: The named age threshold is not measured by the cited benchmark.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: SupportedSaid
Gary Marcus Observation
“The dirty secret in current AI is that for all the bloviating about how current systems have mastered human language, they are still really weak on compositionality, which most linguists (since Frege, over a century ago) would agree is at the very core of how language works.”
Our reading: Current AI systems are still very weak at compositionality, which is central to how language works.
No horizon Evidence as of
Limitation: The result depends on reading current AI as prominent contemporary neural systems rather than every AI architecture.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: SupportedSaid
Gary Marcus Observation
“The progress that Alexander is vaunting shows that the system did better on a handful of items than DALL-E2 (which was not even state of the art at compositionality when it appeared), but not that the newer system (Imagen) systematically groks compositionality”
Our reading: Alexander's test shows only that Imagen beat DALL-E 2 on a handful of items, not that Imagen systematically understands compositionality.
No horizon Evidence as of
Limitation: The scores reflect Alexander's judgments, not independent verification [e1].
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: ContradictedSaid
Gary Marcus Observation
“we were right to be skeptical; so far, four and half years later, Duplex hasn’t gotten a whole lot further.)”
Our reading: Four and a half years after its announcement, Google Duplex has not progressed much past its limited demo stage.
No horizon Evidence as of
Limitation: The degree of progress is qualitative.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Gary Marcus Observation
“But many people have claimed over the years to solve the problem, and none of those systems have proven to be reliable; every proposed solution has been like Clever Hans, working in dim light, leveraging large databases to some degree, but falling apart upon careful inspection; they might get 60% on some task, but they never really master it.”
Our reading: Every claimed solution to compositionality so far has proven unreliable and falls apart on careful inspection.
No horizon Evidence as of
Limitation: The universal scope exceeds the coverage of the retrieved evaluations.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: SupportedSaid
Gary Marcus Observation
“This should be a piece of cake, but for currently popular systems, it’s not.”
Our reading: Currently popular vision-language systems cannot reliably tell apart captions that use the same words in a different order.
No horizon Evidence as of
Limitation: The result applies to the evaluated models and task.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Cool demo, yes. Real world product, anytime soon? Fuggedaboutit.”
Our reading: PaLM-SayCan will not become a real-world product anytime soon.
No deadline stated Evidence as of
Limitation: The search cannot prove the absence of every derivative.
Assessment: SupportedSaid
Gary Marcus Observation
“Performance hovered at around 75% in the best case, when the robots had a limited number of options to consider.”
Our reading: PaLM-SayCan's performance was around 75% at best, when the robot had a limited number of options.
No horizon Evidence as of
Limitation: Best case is interpreted as the best overall setting.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Facebook M was canceled, and 7 years later nobody yet can buy a super-smart (or even decently smart) robot at any price.”
Our reading: As of August 2022, nobody can buy even a decently smart robot at any price.
No horizon Evidence as of
Limitation: The sources do not provide a comprehensive market record for August 18, 2022 [e1][e2][e3].
Assessment: UnresolvedSaid
Gary Marcus Observation
“The thing is, current AI systems that (roughly speaking) do a lot of cutting and pasting without those grammar rules and trees aren’t really all that close to understanding human language.”
Our reading: Current AI systems that, roughly speaking, do a lot of cutting and pasting without grammar rules and trees aren't really all that close to understanding human language.
No horizon Evidence as of
Limitation: Benchmark achievement and general language comprehension are different claims.
Assessment: UnresolvedSaid
Gary Marcus Observation
“We don’t know how kids do it; we do know that what we are doing with machines currently isn’t really working.”
Our reading: Current machine approaches to language acquisition are not really working.
No horizon Evidence as of
Limitation: The selected tasks cover components of language competence.
Assessment: UnresolvedSaid
Gary Marcus Observation
“All normal children manage to acquire language in this sense; no existing machine does.”
Our reading: No existing machine has acquired language in the sense of mapping sentences to meanings and meanings to sentences.
No horizon Evidence as of
Limitation: Formal meaning representations cover only selected aspects of meaning.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“In our view, inadequate attention to these three factors has serious consequences, including:”
Our reading: Neglecting reference, cognitive models and compositionality causes serious problems in large language models.
No horizon Evidence as of
Limitation: The search is not an exhaustive systematic review.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“here, DALL-E mini struggles to accurately represent the implicit action for the woman, capturing the apple but not reliably capturing the eating, often sticks an apple in the man’s mouth rather than the sandwich, and in most cases misses the implication that the man and woman are supposed to be eating different things.”
Our reading: In the authors' experiments with the sentence 'The man is eating a sandwich and the woman an apple', DALL-E Mini struggles to represent the woman's implied action: it captures the apple but not reliably the eating, often puts an apple in the man's mouth instead of the sandwich, and in most cases misses that the two are eating different things.
No horizon Evidence as of
Limitation: A replication could establish reproducibility without proving what happened in the authors' experiment.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The fact that its word sequence predictions are often so good tempts many into ascribing meaning, but this is more a reflection of the tremendous size of the input corpus than of a system that genuinely maps sentence onto reference and cognitive models.”
Our reading: GPT-3's good word predictions come more from its huge training corpus than from any real mapping of sentences onto reference and cognitive models.
No horizon Evidence as of
Limitation: Results from smaller transformers, BART, and T5 are context only.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“Similarly, consider pronouns like “himself”: DALL-E does not do well with sentences like “The man is painting a picture of himself”.”
Our reading: DALL-E does poorly on sentences with reflexive pronouns such as 'The man is painting a picture of himself'.
No horizon Evidence as of
Limitation: Model identity and the standard for poor performance remain unspecified.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“Compositionality has long been a central concept in linguistics and philosophy (even Aristotle acknowledged it), yet so-called foundation models (large language models, GPT-3, BERT, deep learning approaches) sidestep it.”
Our reading: Foundation models such as GPT-3 and BERT sidestep compositionality.
No horizon Evidence as of
Limitation: The studies use different tasks and assistance levels, preventing a direct comparison [e1, e2].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: SupportedSaid
Gary Marcus Observation
“Since then, the results on benchmarks have gotten better, but there’s still something missing.”
Our reading: Since 2019 large language models have done better on benchmarks, but something is still missing from them.
No horizon Evidence as of
Limitation: TruthfulQA deliberately tests susceptibility to misconceptions; its percentages are not general error rates for all language use.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: ContradictedSaid
Gary Marcus Observation
“(LLM’s like GPT produce well-formed prose but do not produce interpretable representations of utterances that reflect structured relationships between the parts of those sentences.)”
Our reading: LLMs like GPT write well-formed prose but do not produce interpretable representations of the structured relationships between the parts of sentences.
No horizon Evidence as of
Limitation: The experiments use task-specific examples and bounded semantic-parsing domains.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;”
Our reading: Neglecting reference, cognitive models and compositionality leads LLMs to lose coherence over time and drift into empty language with no clear connection to reality.
No horizon Evidence as of
Limitation: TattleTale has limited empirical evaluation.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: SupportedSaid
Gary Marcus Observation
“noncanonical reorderings such as these remain problematic for current AI approaches.)”
Our reading: Current AI approaches still struggle with noncanonical word orders such as 'the mat is on the cat'.
No horizon Evidence as of
Limitation: The evidence concerns tested vision-language models, not every contemporary AI approach [e1].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“(b) the difficulty of large language models in distinguishing truth from falsehoods;”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to tell truth from falsehood.
No horizon Evidence as of
Limitation: TruthfulQA's questions target misconceptions; its results do not establish an overall factual-error rate [e8].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.”
Our reading: Current LLMs lack the human ability to choose whether to interpret language compositionally.
No horizon Evidence as of
Limitation: The available tests use different tasks and model configurations.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“Language is about a connection between words (or sentence) and the world; the sequences of words that large language models utter lack connection to the external world.”
Our reading: The word sequences that large language models produce have no connection to the external world, so they lack reference.
No horizon Evidence as of
Limitation: Source attribution does not by itself establish situated reference or factual correctness [e3].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: ContradictedSaid
Gary Marcus Causal claim
“Current systems simply do not postulate such cognitive models—and so, they technically can’t prove themselves wrong, because they do not even have a representation of what being “right” would look like.”
Our reading: Current systems do not postulate cognitive models, so they technically cannot prove themselves wrong, because they have no representation of what being right would look like.
No horizon Evidence as of
Limitation: The result depends on treating independently assessable self-critique as demonstrating an error.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: ContradictedSaid
Gary Marcus Causal claim
“The downstream consequence of all of this is that these models have no grasp whatsoever on truth.”
Our reading: As a downstream consequence of their lack of reference and cognitive models, large language models have no grasp whatsoever on truth.
No horizon Evidence as of
Limitation: The outcome depends on a functional reading of grasp and inclusion of fine-tuned language models.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“In such cases, DALL-E Mini typically generates images of bowls of cucumbers, and only cucumbers, leaving out the tomatoes altogether (where a human might reasonably expect there to be at least two, given the plural).”
Our reading: Given comparative prompts such as 'more cucumbers than tomatoes', DALL-E Mini usually leaves out the tomatoes entirely.
No horizon Evidence as of
Limitation: The evidence establishes neither a majority omission rate nor a rate sufficient to contradict the claim.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“Unless the field can find a way to tightly integrate large language models with cognitive models and reference, we expect that the field will eventually hit a dead end, particularly in the understanding long discourses, and series of events unfolding over time, and that the problem of misinformation will remain unsolved.”
Our reading: Unless LLMs are tightly integrated with cognitive models and reference, the field will eventually hit a dead end, especially in understanding long discourse and events over time, and misinformation will remain unsolved.
No deadline stated Evidence as of
Only if: unless LLMs are tightly integrated with cognitive models and reference
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“(c) the struggle in these models to avoid perpetuating bias and toxic speech.”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to avoid repeating bias and toxic speech.
No horizon Evidence as of
Limitation: The search is not proof that no relevant experiment exists.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: SupportedSaid
Gary Marcus Observation
“Because it shows that the people who really claim to be working on full self-driving (which Tesla waffles on1) are having plenty of problems of their own.”
Our reading: Companies working on full (Level 5) self-driving are having plenty of problems, per NHTSA ADS data.
No horizon Evidence as of
Limitation: Support is for repeated crash involvement, not technical fault or comparative danger.
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“As far as I can tell, nobody has a real solution to the outlier problem of coping with unusual circumstances; without it, full self driving will remain a pipe dream.”
Our reading: Without a solution to the outlier problem, full self-driving will not be achieved.
No deadline stated Evidence as of
Only if: without a solution to the outlier problem
Assessment: SupportedSaid
Gary Marcus Observation
“Level 5 has to handle it all; everyday stopped vehicles, weird outliers and so forth. We’re just not there.”
Our reading: Level 5 self-driving that handles everything has not been achieved.
No horizon Evidence as of
Limitation: The industry-wide finding relies on contemporaneous reporting; the primary permit corroborates a particular deployment.
Assessment: SupportedSaid
Gary Marcus Observation
“Indeed, the NHTSA report data show that even some pretty ordinary circumstances, like avoiding stopped vehicles, are continuing to prove problematic.”
Our reading: NHTSA data show even ordinary circumstances like avoiding stopped vehicles remain problematic for automated driving systems.
No horizon Evidence as of
Limitation: The decisive source is NHTSA's contemporaneous investigation, not a verified extraction of the specific June 15 summary tables.
Assessment: SupportedSaid
Gary Marcus Observation
“The basic finding was: loads of things scale, but not at all ( as foreseen both by Kurenkov in his essay last year and in my much lampooned but basically accurate essay, Deep learning is hitting a wall).”
Our reading: BIG-bench shows many capabilities improve with scale but not all, which bears out Marcus's 'Deep learning is hitting a wall' essay.
No horizon Evidence as of
Limitation: The finding does not establish a permanent scaling barrier or what happens beyond the tested model sizes [e1].
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“👉an explicit type/token distinction will be crucial”
Our reading: Whenever AGI arrives, an explicit type/token distinction will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“👉explicit cognitive models will be crucial”
Our reading: Whenever AGI arrives, explicit cognitive models will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: SupportedSaid
Gary Marcus Observation
“The massive paper looks at scaling on many measures, and sees progress on some– but not others. The stuff that isn’t scaling is the “wall”.”
Our reading: The BIG-bench paper looked at scaling on many measures and found progress on some but not others, and the measures that did not scale in it are what Marcus calls the 'wall'.
No horizon Evidence as of
Limitation: The finding applies to the evaluated models, scales and metrics.
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“👉operations over variables (including storing, retrieving and comparing values) will be crucial”
Our reading: Whenever AGI arrives, operations over variables, including storing, retrieving and comparing values, will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: SupportedSaid
Gary Marcus Observation
“so many of the unfortunate consequences of a webscraping/big data approach to AI are disproportionately borne by women and minorities.”
Our reading: Many of the harmful consequences of the webscraping/big-data approach to AI fall disproportionately on women and minorities.
No horizon Evidence as of
Limitation: Support depends on reading many qualitatively and including representational and data-exclusion harms.