Unresolved claims by Gary Marcus, page 8 of 8
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Unresolved claims by Gary Marcus (772)
The record was checked, and the evidence available by the cutoff does not settle it. Forecasts whose deadline has not arrived are listed apart, as not yet due.
Page 8 of 8: claims 701 to 772 of 772, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Observation
“Which is to say it makes stuff up, regularly; a fair bit of what it is says simply isn’t true.”
Our reading: ChatGPT regularly makes things up and a fair amount of its output is false.
No horizon Evidence as of
Limitation: This does not mean fabrication was absent or rare.
Assessment: UnresolvedSaid
Forecast
““Scaling” alone—building bigger and models until they absorb the entire internet — will prove useful, but only to a point.”
Our reading: Scaling alone will prove useful only up to a point.
No deadline stated Evidence as of
Limitation: The claim specifies no deadline, capability threshold, or operational test of trustworthy general intelligence.
Assessment: UnresolvedSaid
Causal claim
“they cost almost nothing to operate, and so they are on a path to reducing the cost of generating disinformation to zero.”
Our reading: Because generative AI systems like ChatGPT, Dall-E 2, and Lensa cost almost nothing to operate, they are on a path to reducing the cost of generating disinformation to zero.
No horizon Evidence as of
Limitation: The economically negligible reading of 'zero' is an interpretation.
Source: AI's Jurassic Park moment
Assessment: UnresolvedSaid
Observation
“they can easily be automated to generate misinformation at unprecedented scale.”
Our reading: Generative AI systems like ChatGPT, Dall-E 2, and Lensa can easily be automated to generate misinformation at unprecedented scale.
No horizon Evidence as of
Limitation: The supplied experiment predates the November–December 2022 target period [e1].
Source: AI's Jurassic Park moment
Assessment: UnresolvedSaid
Conditional forecast
“If the website is flooded with worthless code examples, programmers will no longer go there, its database of over 30 million questions and answers will become untrustworthy, and the 14 year old website will die.”
Our reading: If Stack Overflow is flooded with worthless generated code, programmers will leave, its database will become untrustworthy, and the site will die.
No deadline stated Evidence as of
Only if: if the website is flooded with worthless AI-generated code examples
Source: AI's Jurassic Park moment
Assessment: UnresolvedSaid
Causal claim
“When GPT sounds plausible, it is because every paraphrased bit that it pastes together is grounded in something that actual humans said, and there is often some vague (but often irrelevant) relationship between..”
Our reading: GPT sounds plausible because each piece it combines comes from something humans actually said, and the pieces are often loosely related.
No horizon Evidence as of
Limitation: The meanings of bit, paraphrase and grounded materially affect the test.
Source: How come GPT can seem so brilliant one minute and so breathtakingly dumb the next?
Assessment: UnresolvedSaid
Observation
“As it turns out, the architecture of Cicero differs profoundly from most of what’s been talked about in recent years in AI.”
Our reading: Cicero's architecture differs profoundly from most AI approaches discussed in recent years.
No horizon Evidence as of
Limitation: Selected examples establish differences, not prevalence.
Source: What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Assessment: UnresolvedSaid
Conditional forecast
“if Alexa could converse freely, user expectations would go through the roof, and mostly be unmeetable.”
Our reading: If Alexa could converse freely, user expectations would rise to a level that mostly could not be met.
No deadline stated Evidence as of
Only if: if Alexa could converse freely
Assessment: UnresolvedSaid
Forecast
“Turning LLMs into a product that controls your home and talks to you in a way that would be reliable enough to use at scale in millions of homes is still a long, long way away.”
Our reading: A reliable LLM-based conversational home-control product usable at scale in millions of homes is still a long, long way away.
No deadline stated Evidence as of
Limitation: Large-scale participation is company-reported.
Assessment: UnresolvedSaid
Observation
“Just like every other large language model I have seen.”
Our reading: Like Galactica, every other large language model the author has seen prevaricates (fabricates content) a lot.
No horizon Evidence as of
Limitation: False answers on deliberately challenging questions are not a direct estimate of fabrication in ordinary use.
Source: A Few Words About Bullshit
Assessment: UnresolvedSaid
Forecast
“High school students will love it, and use it to fool and intimidate (some of) their teachers.”
Our reading: High school students will love Galactica and will use it to fool and intimidate some of their teachers.
No deadline stated Evidence as of
Limitation: A search that finds no qualifying incidents cannot establish that none occurred.
Source: A Few Words About Bullshit
Assessment: UnresolvedSaid
Forecast
“The problem is about to get much, much worse.”
Our reading: The problem of misinformation on Twitter is about to get much, much worse.
No deadline stated Evidence as of
Limitation: An increase in misinformation exposure could occur without an increase in posting volume.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Conditional forecast
“As the pace of machine-generated misinformation picks up, Twitter’s existing effort, Community Notes (aka Birdwatch), which is mostly done manually, by humans, is going to get left in the dust.”
Our reading: As machine-generated misinformation accelerates, Twitter's mostly manual Community Notes will be overwhelmed.
No deadline stated Evidence as of
Only if: as the pace of machine-generated misinformation picks up
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Observation
“Large language models are without equal at generating misinformation, but they suck at detecting it.”
Our reading: LLMs are unmatched at generating misinformation but poor at detecting it.
No horizon Evidence as of
Limitation: No admitted comparison establishes unmatched misinformation generation during the target period.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Forecast
“Scaling won’t solve this.”
Our reading: Scaling LLMs will not solve their inability to reason about truth and detect misinformation.
No deadline stated Evidence as of
Limitation: Truthful answer generation and misinformation detection are related but distinct tasks.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Forecast
“Scaling maximalism is an interesting hypothesis, but I stand by my prediction that it won’t get us to AGI.”
Our reading: Scaling maximalism will not get us to AGI.
No deadline stated Evidence as of
Limitation: This is not a finding that AGI is absent everywhere; it is a finding that the reviewed evidence does not settle this forecast.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Conditional forecast
“Any one of them would mean we need a real paradigm shift, if we are to get to AGI.”
Our reading: If any one of the three failure modes (not enough data, not enough compute, tasks that don't scale) holds, reaching AGI will need a real paradigm shift.
No deadline stated Evidence as of
Only if: if any of the three failure modes (data, compute, non-scaling tasks) holds
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Causal claim
“His hypothesis, now generally called scaling maximalism, remains extremely popular, in no small part because bigger and bigger models have indeed continued to do ever more impressive things.”
Our reading: Scaling maximalism remains extremely popular, in no small part because bigger and bigger AI models have continued to do ever more impressive things.
No horizon Evidence as of
Limitation: The survey concerns NLP researchers, not all AI researchers, companies, investors, or the public.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Observation
“As I have long argued, capturing this without cognitive models and common sense is really hard.”
Our reading: Capturing the pragmatic implications of language without cognitive models and common sense is really hard.
No horizon Evidence as of
Limitation: The finding does not deny the demonstrated difficulty.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Observation
“It’s also, as ever, stuck in the liminal space between superintelligence and super-lost.”
Our reading: Make a Video, as with AI systems before it, is caught between very impressive output and basic errors, such as giving a cat a human hand or a painting that never changes.
No horizon Evidence as of
Limitation: The original moving clips were not independently inspected in this assessment.
Assessment: UnresolvedSaid
Forecast
“This is the uncanny valley we are in, and one that we occupy for a while.”
Our reading: AI systems such as video, image, and language generators are now in an uncanny valley of impressive output with basic errors, and will stay there for a while.
No deadline stated Evidence as of
Limitation: This outcome does not mean the observed persistence is unsupported; it reflects the missing duration and resolution threshold.
Assessment: UnresolvedSaid
Causal claim
“We in AI have a long long way to go. Data and compute alone won’t be enough.”
Our reading: AI has a long way to go, and data and compute alone will not be enough.
No horizon Evidence as of
Limitation: Benchmark performance covers only part of the capability claim.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Observation
“Musk barely even acknowledged the problem, and certainly didn’t lay out any sort of credible strategy for attacking it..”
Our reading: Musk barely acknowledged the common-sense problem and laid out no credible strategy for addressing it.
No horizon Evidence as of
Limitation: An absence claim requires stronger coverage than targeted keyword searches.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Causal claim
“The reality is that AI needs genuine, paradigm-shift level innovation. Simply building ever bigger neural networks won’t cut it.”
Our reading: AI needs paradigm-shift innovation; simply scaling neural networks will not be enough.
No horizon Evidence as of
Limitation: The capability target and paradigm boundary are not operationally defined.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Observation
“Second, there was very little vision for how Tesla would build the cognitive part of the AI they will need, beyond the basics of motor control (which Boston Dynamics already does so well), nor much recognition about why robotics is so hard in the real world.”
Our reading: Tesla showed very little vision for building the cognitive AI Optimus needs beyond motor control, and little recognition of why real-world robotics is hard.
No horizon Evidence as of
Limitation: The cited notes are secondary and include automated transcription.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Observation
“Musk is almost never right about deadlines.”
Our reading: Elon Musk's deadlines for technology projects are almost never met.
No horizon Evidence as of
Limitation: The usable evidence is concentrated on Tesla.
Source: Top 5 reasons Tesla’s Optimus announcement may turn out be a dud
Assessment: UnresolvedSaid
Observation
“On #1, the answer is a clear yes; only highly trained human artists could do better.”
Our reading: Image synthesis systems generate high quality images that only highly trained human artists could surpass.
No horizon Evidence as of
Limitation: Image quality can vary by task and evaluator.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Observation
“Nine more tries, and only one very marginal success (top right corner):”
Our reading: Stable Diffusion produced only one very marginal success in nine further attempts at a part-function prompt.
No horizon Evidence as of
Limitation: Success grading includes a subjective boundary.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Observation
“Even “draw a white bicycle with green wheels”, which focuses purely on part-whole relationships without function or complex syntax, is problematic:”
Our reading: Stable Diffusion has trouble even with simple part-whole prompts such as a white bicycle with green wheels.
No horizon Evidence as of
Limitation: Failure to locate independent records does not show that Marcus's displayed results were inaccurate.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Observation
“Negation is, as ever, a problem.”
Our reading: Text-to-image systems like Stable Diffusion continue to have problems with negation in prompts.
No horizon Evidence as of
Limitation: This is an evidence gap, not a finding that negation worked reliably.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Observation
“Google cloaks its current work the trappings of science (article preprints with citations and graphs and so forth) but when it comes to community inspection, they’ve systematically opted out.”
Our reading: Google has systematically refused to let the scientific community inspect models such as Imagen and LaMDA.
No horizon Evidence as of
Limitation: The evidence distinguishes public availability from scientific access but does not fully document the latter.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Observation
“Notwithstanding the rumors about Google Imagen, nobody has yet publicly demonstrated a machine that can relate the meanings of sentences to their parts the way a five-year-old child can.”
Our reading: No one, Imagen included, has publicly shown a machine that relates sentence meanings to their parts as well as a five-year-old.
No horizon Evidence as of
Limitation: The named age threshold is not measured by the cited benchmark.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Observation
“But many people have claimed over the years to solve the problem, and none of those systems have proven to be reliable; every proposed solution has been like Clever Hans, working in dim light, leveraging large databases to some degree, but falling apart upon careful inspection; they might get 60% on some task, but they never really master it.”
Our reading: Every claimed solution to compositionality so far has proven unreliable and falls apart on careful inspection.
No horizon Evidence as of
Limitation: The universal scope exceeds the coverage of the retrieved evaluations.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Forecast
“Cool demo, yes. Real world product, anytime soon? Fuggedaboutit.”
Our reading: PaLM-SayCan will not become a real-world product anytime soon.
No deadline stated Evidence as of
Limitation: The search cannot prove the absence of every derivative.
Assessment: UnresolvedSaid
Observation
“Facebook M was canceled, and 7 years later nobody yet can buy a super-smart (or even decently smart) robot at any price.”
Our reading: As of August 2022, nobody can buy even a decently smart robot at any price.
No horizon Evidence as of
Limitation: The sources do not provide a comprehensive market record for August 18, 2022 [e1][e2][e3].
Assessment: UnresolvedSaid
Observation
“The thing is, current AI systems that (roughly speaking) do a lot of cutting and pasting without those grammar rules and trees aren’t really all that close to understanding human language.”
Our reading: Current AI systems that, roughly speaking, do a lot of cutting and pasting without grammar rules and trees aren't really all that close to understanding human language.
No horizon Evidence as of
Limitation: Benchmark achievement and general language comprehension are different claims.
Assessment: UnresolvedSaid
Observation
“We don’t know how kids do it; we do know that what we are doing with machines currently isn’t really working.”
Our reading: Current machine approaches to language acquisition are not really working.
No horizon Evidence as of
Limitation: The selected tasks cover components of language competence.
Assessment: UnresolvedSaid
Observation
“All normal children manage to acquire language in this sense; no existing machine does.”
Our reading: No existing machine has acquired language in the sense of mapping sentences to meanings and meanings to sentences.
No horizon Evidence as of
Limitation: Formal meaning representations cover only selected aspects of meaning.
Assessment: UnresolvedSaid
Causal claim
“In our view, inadequate attention to these three factors has serious consequences, including:”
Our reading: Neglecting reference, cognitive models and compositionality causes serious problems in large language models.
No horizon Evidence as of
Limitation: The search is not an exhaustive systematic review.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“here, DALL-E mini struggles to accurately represent the implicit action for the woman, capturing the apple but not reliably capturing the eating, often sticks an apple in the man’s mouth rather than the sandwich, and in most cases misses the implication that the man and woman are supposed to be eating different things.”
Our reading: In the authors' experiments with the sentence 'The man is eating a sandwich and the woman an apple', DALL-E Mini struggles to represent the woman's implied action: it captures the apple but not reliably the eating, often puts an apple in the man's mouth instead of the sandwich, and in most cases misses that the two are eating different things.
No horizon Evidence as of
Limitation: A replication could establish reproducibility without proving what happened in the authors' experiment.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Causal claim
“The fact that its word sequence predictions are often so good tempts many into ascribing meaning, but this is more a reflection of the tremendous size of the input corpus than of a system that genuinely maps sentence onto reference and cognitive models.”
Our reading: GPT-3's good word predictions come more from its huge training corpus than from any real mapping of sentences onto reference and cognitive models.
No horizon Evidence as of
Limitation: Results from smaller transformers, BART, and T5 are context only.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“Similarly, consider pronouns like “himself”: DALL-E does not do well with sentences like “The man is painting a picture of himself”.”
Our reading: DALL-E does poorly on sentences with reflexive pronouns such as 'The man is painting a picture of himself'.
No horizon Evidence as of
Limitation: Model identity and the standard for poor performance remain unspecified.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“Compositionality has long been a central concept in linguistics and philosophy (even Aristotle acknowledged it), yet so-called foundation models (large language models, GPT-3, BERT, deep learning approaches) sidestep it.”
Our reading: Foundation models such as GPT-3 and BERT sidestep compositionality.
No horizon Evidence as of
Limitation: The studies use different tasks and assistance levels, preventing a direct comparison [e1, e2].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Causal claim
“(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;”
Our reading: Neglecting reference, cognitive models and compositionality leads LLMs to lose coherence over time and drift into empty language with no clear connection to reality.
No horizon Evidence as of
Limitation: TattleTale has limited empirical evaluation.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Causal claim
“(b) the difficulty of large language models in distinguishing truth from falsehoods;”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to tell truth from falsehood.
No horizon Evidence as of
Limitation: TruthfulQA's questions target misconceptions; its results do not establish an overall factual-error rate [e8].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.”
Our reading: Current LLMs lack the human ability to choose whether to interpret language compositionally.
No horizon Evidence as of
Limitation: The available tests use different tasks and model configurations.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“Language is about a connection between words (or sentence) and the world; the sequences of words that large language models utter lack connection to the external world.”
Our reading: The word sequences that large language models produce have no connection to the external world, so they lack reference.
No horizon Evidence as of
Limitation: Source attribution does not by itself establish situated reference or factual correctness [e3].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“In such cases, DALL-E Mini typically generates images of bowls of cucumbers, and only cucumbers, leaving out the tomatoes altogether (where a human might reasonably expect there to be at least two, given the plural).”
Our reading: Given comparative prompts such as 'more cucumbers than tomatoes', DALL-E Mini usually leaves out the tomatoes entirely.
No horizon Evidence as of
Limitation: The evidence establishes neither a majority omission rate nor a rate sufficient to contradict the claim.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Conditional forecast
“Unless the field can find a way to tightly integrate large language models with cognitive models and reference, we expect that the field will eventually hit a dead end, particularly in the understanding long discourses, and series of events unfolding over time, and that the problem of misinformation will remain unsolved.”
Our reading: Unless LLMs are tightly integrated with cognitive models and reference, the field will eventually hit a dead end, especially in understanding long discourse and events over time, and misinformation will remain unsolved.
No deadline stated Evidence as of
Only if: unless LLMs are tightly integrated with cognitive models and reference
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Causal claim
“(c) the struggle in these models to avoid perpetuating bias and toxic speech.”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to avoid repeating bias and toxic speech.
No horizon Evidence as of
Limitation: The search is not proof that no relevant experiment exists.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Conditional forecast
“As far as I can tell, nobody has a real solution to the outlier problem of coping with unusual circumstances; without it, full self driving will remain a pipe dream.”
Our reading: Without a solution to the outlier problem, full self-driving will not be achieved.
No deadline stated Evidence as of
Only if: without a solution to the outlier problem
Assessment: UnresolvedSaid
Conditional forecast
“👉an explicit type/token distinction will be crucial”
Our reading: Whenever AGI arrives, an explicit type/token distinction will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: UnresolvedSaid
Conditional forecast
“👉explicit cognitive models will be crucial”
Our reading: Whenever AGI arrives, explicit cognitive models will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: UnresolvedSaid
Conditional forecast
“👉operations over variables (including storing, retrieving and comparing values) will be crucial”
Our reading: Whenever AGI arrives, operations over variables, including storing, retrieving and comparing values, will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: UnresolvedSaid
Observation
“Large language models are (unless hybridized with symbolic systems, a la Jurassic X) are, on their own, inherently weak at understanding the physical and psychological world.”
Our reading: LLMs on their own, without symbolic hybridization, are inherently weak at understanding the physical and psychological world.
No horizon Evidence as of
Limitation: TRIP tests a narrower model population than the frontier generative systems discussed in the post.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Observation
“Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.”
Our reading: As Davis has found, across the full set of their Technology Review questions, the difference between GPT-3's earlier version and InstructGPT is far more modest than one might have expected: fewer outright wacky answers and some old answers now correct, but some backsliding too.
No horizon Evidence as of
Limitation: The source restriction excludes the report most directly associated with the claim.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Observation
“The core claims that Ernie and I made in Technology Review still all, so far as I can tell, hold true, nearly two years and many even larger models later. Yes, some of the specific examples we pointed out have been remedied, but the general problems have not.”
Our reading: So far as Marcus can tell, all the core claims he and Davis made in Technology Review still hold true nearly two years and many even larger models later; some of their specific examples have been remedied, but the general problems have not.
No horizon Evidence as of
Limitation: This is a gap in the retrieved evidence, not evidence that the categories had been solved.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Causal claim
“But they still don’t convey genuine comprehension, and so they are still very easy for Ernie and me (or anyone else who cares to try) to break.”
Our reading: Newer LLMs still lack genuine comprehension and are therefore still very easy for anyone to break.
No horizon Evidence as of
Limitation: Benchmark scores and qualitative examples do not establish that anyone could easily elicit failures across all relevant models [e1, e2].
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Conditional forecast
“I am willing to bet SlateStarCodex now (terms to be negotiated) that if OpenAI gives us unrestricted access to GPT-4, whenever that is released, and assuming that is basically the same architecture but with more data, that within a day of playing around with it, Ernie and I will still be able find lots of examples of failures in physical reasoning, temporal reasoning, causal reasoning, and so forth.”
Our reading: If given unrestricted access to GPT-4 and it is basically the same architecture with more data, Marcus and Davis will find many failures in physical, temporal, and causal reasoning within a day.
No deadline stated Evidence as of
Only if: OpenAI gives unrestricted access to GPT-4
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Causal claim
“The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.”
Our reading: The newer models' improvements, such as they are, come primarily from their larger and larger sets of data about how humans use word sequences, which are certainly helpful for pattern-matching machines.
No horizon Evidence as of
Limitation: An optimal scaling prescription is not a decomposition of historical progress.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Observation
“The fourth thing to realize is that we, and by this I mean humanity as a whole, still don’t really have an adequate methodology for building complex cognitive systems.”
Our reading: Humanity lacks an adequate methodology for building complex cognitive systems.
No horizon Evidence as of
Limitation: The empirical study covers a sample of practitioners and systems.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Observation
“I don’t know what you have planned for Optimus, but I can promise you that the AGI that you would need for a general-purpose domestic robot (where every home is different, and each poses its own safety risks) is way beyond what you would need for a car that drives on roads that are more or less engineered the same way from one town to the next.”
Our reading: The AGI needed for a general-purpose domestic robot, where every home is different and poses its own safety risks, is way beyond what is needed for a car driving on roads engineered more or less the same from town to town.
No horizon Evidence as of
Limitation: Household robotics evidence includes simulation and constrained kitchen tasks.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Observation
“We still don’t have anything like stable or trustworthy solutions for common sense, reasoning, language, or analogy. Truth is, I have been using this same pie chart for about 5 years, and the situation has scarcely changed.”
Our reading: AI has no stable or trustworthy solutions for common sense, reasoning, language or analogy, and this has scarcely changed in about five years.
No horizon Evidence as of
Limitation: The studies use different tasks, models and evaluation methods.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Observation
“Deep learning is pretty decent at, for example, recognizing objects, but not nearly as good at planning, or reading, or language comprehension.”
Our reading: Deep learning is decent at object recognition but much worse at planning, reading and language comprehension.
No horizon Evidence as of
Limitation: Benchmark success does not establish broad real-world comprehension.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Causal claim
“The thing is, it’s easy to convince yourself that AI problems are much easier than they are actually are, because of the long tail problem.”
Our reading: The long tail problem makes it easy to believe AI problems are much easier than they actually are.
No horizon Evidence as of
Limitation: The closest empirical evidence uses a simulated robot task.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Observation
“As Ernie Davis, Scott Aaronson, and I showed, a week or so after the release, it’s easily fooled, especially with complex sentences and words like “similar” that would require a deep understanding of language to be interpreted correctly”
Our reading: DALL-E 2 is easily fooled, especially by complex sentences and words requiring deep language understanding.
No horizon Evidence as of
Limitation: Unresolved does not mean the claim was false.
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Causal claim
“Since we know that Imagen can draw images of horses riding astronauts (if the instructions are specific enough) we know that the failure to draw a horse riding an astronaut given the prompt “a horse riding an astronaut” can only have one explanation: the system doesn’t understand what the prompt “a horse riding an astronaut” means.”
Our reading: Since Imagen can draw horses riding astronauts when the instructions are specific enough, its failure to draw one from the prompt 'a horse riding an astronaut' can only be explained by the system not understanding what that prompt means.
No horizon Evidence as of
Limitation: The available demonstration is not a controlled causal experiment.
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Causal claim
“And, no, it wasn’t about common sense at all.”
Our reading: Imagen's failure on 'a horse riding an astronaut' was not due to common sense.
No horizon Evidence as of
Limitation: The demonstration provides no repeated-sample success rates or controlled comparison separating wording from scene plausibility [e5].
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Observation
“(In reality, Imagen has no idea what weight, mass, or density are.)”
Our reading: Imagen has no understanding of weight, mass or density.
No horizon Evidence as of
Limitation: The behavioral interpretation makes the claim testable but does not settle philosophical questions about understanding.
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Causal claim
“There is no intention to create misinformation—but also no capacity to avoid it, because fundamentally, GPT-3 is a model of how words relate to one another, not a model of how language might relate to the perceived world.”
Our reading: GPT-3 cannot avoid producing misinformation because it models how words relate to each other, not how language relates to the world.
No horizon Evidence as of
Limitation: The evidence is about error rates, not about mechanism.
Source: Noam Chomsky and GPT-3
Assessment: UnresolvedSaid
Conditional forecast
“The bottom line is this, something that AI once cherished but has now forgotten: If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.”
Our reading: If we are to build AGI, we will need to learn something from humans: how they reason and understand the physical world, and how they represent and acquire language and complex concepts.
No deadline stated Evidence as of
Only if: if we are to build AGI
Assessment: UnresolvedSaid
Observation
“As I have written, I am fine with thinking of Gato as an “Alt Intelligence”— an interesting exploration in alternative ways to build intelligence—but we need to take it in context: it doesn’t work like the brain, it doesn’t learn like a child, it doesn’t understand language, it doesn’t align with human values, and it can’t be trusted with mission-critical tasks.”
Our reading: Gato does not work like the brain, learn like a child, understand language, or align with human values, and cannot be trusted with mission-critical tasks.
No horizon Evidence as of
Limitation: The evidence is the developers' original report, rather than independent validation of all five assertions [e6].