Claims checked against evidence by Gary Marcus, page 15 of 15
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1471)
Each of these claims was checked against dated evidence.
Page 15 of 15: claims 1401 to 1471 of 1471, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Observation
“Notwithstanding the rumors about Google Imagen, nobody has yet publicly demonstrated a machine that can relate the meanings of sentences to their parts the way a five-year-old child can.”
Our reading: No one, Imagen included, has publicly shown a machine that relates sentence meanings to their parts as well as a five-year-old.
No horizon Evidence as of
Limitation: The named age threshold is not measured by the cited benchmark.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: SupportedSaid
Observation
“The dirty secret in current AI is that for all the bloviating about how current systems have mastered human language, they are still really weak on compositionality, which most linguists (since Frege, over a century ago) would agree is at the very core of how language works.”
Our reading: Current AI systems are still very weak at compositionality, which is central to how language works.
No horizon Evidence as of
Limitation: The result depends on reading current AI as prominent contemporary neural systems rather than every AI architecture.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: SupportedSaid
Observation
“The progress that Alexander is vaunting shows that the system did better on a handful of items than DALL-E2 (which was not even state of the art at compositionality when it appeared), but not that the newer system (Imagen) systematically groks compositionality”
Our reading: Alexander's test shows only that Imagen beat DALL-E 2 on a handful of items, not that Imagen systematically understands compositionality.
No horizon Evidence as of
Limitation: The scores reflect Alexander's judgments, not independent verification [e1].
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: ContradictedSaid
Observation
“we were right to be skeptical; so far, four and half years later, Duplex hasn’t gotten a whole lot further.)”
Our reading: Four and a half years after its announcement, Google Duplex has not progressed much past its limited demo stage.
No horizon Evidence as of
Limitation: The degree of progress is qualitative.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Observation
“But many people have claimed over the years to solve the problem, and none of those systems have proven to be reliable; every proposed solution has been like Clever Hans, working in dim light, leveraging large databases to some degree, but falling apart upon careful inspection; they might get 60% on some task, but they never really master it.”
Our reading: Every claimed solution to compositionality so far has proven unreliable and falls apart on careful inspection.
No horizon Evidence as of
Limitation: The universal scope exceeds the coverage of the retrieved evaluations.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: SupportedSaid
Observation
“This should be a piece of cake, but for currently popular systems, it’s not.”
Our reading: Currently popular vision-language systems cannot reliably tell apart captions that use the same words in a different order.
No horizon Evidence as of
Limitation: The result applies to the evaluated models and task.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Forecast
“Cool demo, yes. Real world product, anytime soon? Fuggedaboutit.”
Our reading: PaLM-SayCan will not become a real-world product anytime soon.
No deadline stated Evidence as of
Limitation: The search cannot prove the absence of every derivative.
Assessment: SupportedSaid
Observation
“Performance hovered at around 75% in the best case, when the robots had a limited number of options to consider.”
Our reading: PaLM-SayCan's performance was around 75% at best, when the robot had a limited number of options.
No horizon Evidence as of
Limitation: Best case is interpreted as the best overall setting.
Assessment: UnresolvedSaid
Observation
“Facebook M was canceled, and 7 years later nobody yet can buy a super-smart (or even decently smart) robot at any price.”
Our reading: As of August 2022, nobody can buy even a decently smart robot at any price.
No horizon Evidence as of
Limitation: The sources do not provide a comprehensive market record for August 18, 2022 [e1][e2][e3].
Assessment: UnresolvedSaid
Observation
“The thing is, current AI systems that (roughly speaking) do a lot of cutting and pasting without those grammar rules and trees aren’t really all that close to understanding human language.”
Our reading: Current AI systems that, roughly speaking, do a lot of cutting and pasting without grammar rules and trees aren't really all that close to understanding human language.
No horizon Evidence as of
Limitation: Benchmark achievement and general language comprehension are different claims.
Assessment: UnresolvedSaid
Observation
“We don’t know how kids do it; we do know that what we are doing with machines currently isn’t really working.”
Our reading: Current machine approaches to language acquisition are not really working.
No horizon Evidence as of
Limitation: The selected tasks cover components of language competence.
Assessment: UnresolvedSaid
Observation
“All normal children manage to acquire language in this sense; no existing machine does.”
Our reading: No existing machine has acquired language in the sense of mapping sentences to meanings and meanings to sentences.
No horizon Evidence as of
Limitation: Formal meaning representations cover only selected aspects of meaning.
Assessment: UnresolvedSaid
Causal claim
“In our view, inadequate attention to these three factors has serious consequences, including:”
Our reading: Neglecting reference, cognitive models and compositionality causes serious problems in large language models.
No horizon Evidence as of
Limitation: The search is not an exhaustive systematic review.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“here, DALL-E mini struggles to accurately represent the implicit action for the woman, capturing the apple but not reliably capturing the eating, often sticks an apple in the man’s mouth rather than the sandwich, and in most cases misses the implication that the man and woman are supposed to be eating different things.”
Our reading: In the authors' experiments with the sentence 'The man is eating a sandwich and the woman an apple', DALL-E Mini struggles to represent the woman's implied action: it captures the apple but not reliably the eating, often puts an apple in the man's mouth instead of the sandwich, and in most cases misses that the two are eating different things.
No horizon Evidence as of
Limitation: A replication could establish reproducibility without proving what happened in the authors' experiment.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Causal claim
“The fact that its word sequence predictions are often so good tempts many into ascribing meaning, but this is more a reflection of the tremendous size of the input corpus than of a system that genuinely maps sentence onto reference and cognitive models.”
Our reading: GPT-3's good word predictions come more from its huge training corpus than from any real mapping of sentences onto reference and cognitive models.
No horizon Evidence as of
Limitation: Results from smaller transformers, BART, and T5 are context only.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“Similarly, consider pronouns like “himself”: DALL-E does not do well with sentences like “The man is painting a picture of himself”.”
Our reading: DALL-E does poorly on sentences with reflexive pronouns such as 'The man is painting a picture of himself'.
No horizon Evidence as of
Limitation: Model identity and the standard for poor performance remain unspecified.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“Compositionality has long been a central concept in linguistics and philosophy (even Aristotle acknowledged it), yet so-called foundation models (large language models, GPT-3, BERT, deep learning approaches) sidestep it.”
Our reading: Foundation models such as GPT-3 and BERT sidestep compositionality.
No horizon Evidence as of
Limitation: The studies use different tasks and assistance levels, preventing a direct comparison [e1, e2].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: SupportedSaid
Observation
“Since then, the results on benchmarks have gotten better, but there’s still something missing.”
Our reading: Since 2019 large language models have done better on benchmarks, but something is still missing from them.
No horizon Evidence as of
Limitation: TruthfulQA deliberately tests susceptibility to misconceptions; its percentages are not general error rates for all language use.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: ContradictedSaid
Observation
“(LLM’s like GPT produce well-formed prose but do not produce interpretable representations of utterances that reflect structured relationships between the parts of those sentences.)”
Our reading: LLMs like GPT write well-formed prose but do not produce interpretable representations of the structured relationships between the parts of sentences.
No horizon Evidence as of
Limitation: The experiments use task-specific examples and bounded semantic-parsing domains.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Causal claim
“(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;”
Our reading: Neglecting reference, cognitive models and compositionality leads LLMs to lose coherence over time and drift into empty language with no clear connection to reality.
No horizon Evidence as of
Limitation: TattleTale has limited empirical evaluation.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: SupportedSaid
Observation
“noncanonical reorderings such as these remain problematic for current AI approaches.)”
Our reading: Current AI approaches still struggle with noncanonical word orders such as 'the mat is on the cat'.
No horizon Evidence as of
Limitation: The evidence concerns tested vision-language models, not every contemporary AI approach [e1].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Causal claim
“(b) the difficulty of large language models in distinguishing truth from falsehoods;”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to tell truth from falsehood.
No horizon Evidence as of
Limitation: TruthfulQA's questions target misconceptions; its results do not establish an overall factual-error rate [e8].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.”
Our reading: Current LLMs lack the human ability to choose whether to interpret language compositionally.
No horizon Evidence as of
Limitation: The available tests use different tasks and model configurations.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“Language is about a connection between words (or sentence) and the world; the sequences of words that large language models utter lack connection to the external world.”
Our reading: The word sequences that large language models produce have no connection to the external world, so they lack reference.
No horizon Evidence as of
Limitation: Source attribution does not by itself establish situated reference or factual correctness [e3].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: ContradictedSaid
Causal claim
“Current systems simply do not postulate such cognitive models—and so, they technically can’t prove themselves wrong, because they do not even have a representation of what being “right” would look like.”
Our reading: Current systems do not postulate cognitive models, so they technically cannot prove themselves wrong, because they have no representation of what being right would look like.
No horizon Evidence as of
Limitation: The result depends on treating independently assessable self-critique as demonstrating an error.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: ContradictedSaid
Causal claim
“The downstream consequence of all of this is that these models have no grasp whatsoever on truth.”
Our reading: As a downstream consequence of their lack of reference and cognitive models, large language models have no grasp whatsoever on truth.
No horizon Evidence as of
Limitation: The outcome depends on a functional reading of grasp and inclusion of fine-tuned language models.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Observation
“In such cases, DALL-E Mini typically generates images of bowls of cucumbers, and only cucumbers, leaving out the tomatoes altogether (where a human might reasonably expect there to be at least two, given the plural).”
Our reading: Given comparative prompts such as 'more cucumbers than tomatoes', DALL-E Mini usually leaves out the tomatoes entirely.
No horizon Evidence as of
Limitation: The evidence establishes neither a majority omission rate nor a rate sufficient to contradict the claim.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Conditional forecast
“Unless the field can find a way to tightly integrate large language models with cognitive models and reference, we expect that the field will eventually hit a dead end, particularly in the understanding long discourses, and series of events unfolding over time, and that the problem of misinformation will remain unsolved.”
Our reading: Unless LLMs are tightly integrated with cognitive models and reference, the field will eventually hit a dead end, especially in understanding long discourse and events over time, and misinformation will remain unsolved.
No deadline stated Evidence as of
Only if: unless LLMs are tightly integrated with cognitive models and reference
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Causal claim
“(c) the struggle in these models to avoid perpetuating bias and toxic speech.”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to avoid repeating bias and toxic speech.
No horizon Evidence as of
Limitation: The search is not proof that no relevant experiment exists.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: SupportedSaid
Observation
“Because it shows that the people who really claim to be working on full self-driving (which Tesla waffles on1) are having plenty of problems of their own.”
Our reading: Companies working on full (Level 5) self-driving are having plenty of problems, per NHTSA ADS data.
No horizon Evidence as of
Limitation: Support is for repeated crash involvement, not technical fault or comparative danger.
Assessment: UnresolvedSaid
Conditional forecast
“As far as I can tell, nobody has a real solution to the outlier problem of coping with unusual circumstances; without it, full self driving will remain a pipe dream.”
Our reading: Without a solution to the outlier problem, full self-driving will not be achieved.
No deadline stated Evidence as of
Only if: without a solution to the outlier problem
Assessment: SupportedSaid
Observation
“Level 5 has to handle it all; everyday stopped vehicles, weird outliers and so forth. We’re just not there.”
Our reading: Level 5 self-driving that handles everything has not been achieved.
No horizon Evidence as of
Limitation: The industry-wide finding relies on contemporaneous reporting; the primary permit corroborates a particular deployment.
Assessment: SupportedSaid
Observation
“Indeed, the NHTSA report data show that even some pretty ordinary circumstances, like avoiding stopped vehicles, are continuing to prove problematic.”
Our reading: NHTSA data show even ordinary circumstances like avoiding stopped vehicles remain problematic for automated driving systems.
No horizon Evidence as of
Limitation: The decisive source is NHTSA's contemporaneous investigation, not a verified extraction of the specific June 15 summary tables.
Assessment: SupportedSaid
Observation
“The basic finding was: loads of things scale, but not at all ( as foreseen both by Kurenkov in his essay last year and in my much lampooned but basically accurate essay, Deep learning is hitting a wall).”
Our reading: BIG-bench shows many capabilities improve with scale but not all, which bears out Marcus's 'Deep learning is hitting a wall' essay.
No horizon Evidence as of
Limitation: The finding does not establish a permanent scaling barrier or what happens beyond the tested model sizes [e1].
Assessment: UnresolvedSaid
Conditional forecast
“👉an explicit type/token distinction will be crucial”
Our reading: Whenever AGI arrives, an explicit type/token distinction will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: UnresolvedSaid
Conditional forecast
“👉explicit cognitive models will be crucial”
Our reading: Whenever AGI arrives, explicit cognitive models will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: SupportedSaid
Observation
“The massive paper looks at scaling on many measures, and sees progress on some– but not others. The stuff that isn’t scaling is the “wall”.”
Our reading: The BIG-bench paper looked at scaling on many measures and found progress on some but not others, and the measures that did not scale in it are what Marcus calls the 'wall'.
No horizon Evidence as of
Limitation: The finding applies to the evaluated models, scales and metrics.
Assessment: UnresolvedSaid
Conditional forecast
“👉operations over variables (including storing, retrieving and comparing values) will be crucial”
Our reading: Whenever AGI arrives, operations over variables, including storing, retrieving and comparing values, will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: SupportedSaid
Observation
“so many of the unfortunate consequences of a webscraping/big data approach to AI are disproportionately borne by women and minorities.”
Our reading: Many of the harmful consequences of the webscraping/big-data approach to AI fall disproportionately on women and minorities.
No horizon Evidence as of
Limitation: Support depends on reading many qualitatively and including representational and data-exclusion harms.
Assessment: SupportedSaid
Forecast
“I would be very surprised if that fundamental lack of reliability did not also hold true for GPT-4. Maybe it will take three paragraphs in a prompt rather than one to break it, but a big Transformer alone does not a a theory of the world create.”
Our reading: GPT-4 will very likely still have a fundamental lack of reliability, though it may take longer prompts to break.
Settled before its deadline Evidence as of
Limitation: Support applies to recurring factual and reasoning unreliability, not a claim that GPT-4 has no useful capabilities.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Observation
“Large language models are (unless hybridized with symbolic systems, a la Jurassic X) are, on their own, inherently weak at understanding the physical and psychological world.”
Our reading: LLMs on their own, without symbolic hybridization, are inherently weak at understanding the physical and psychological world.
No horizon Evidence as of
Limitation: TRIP tests a narrower model population than the frontier generative systems discussed in the post.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Observation
“Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.”
Our reading: As Davis has found, across the full set of their Technology Review questions, the difference between GPT-3's earlier version and InstructGPT is far more modest than one might have expected: fewer outright wacky answers and some old answers now correct, but some backsliding too.
No horizon Evidence as of
Limitation: The source restriction excludes the report most directly associated with the claim.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Observation
“The core claims that Ernie and I made in Technology Review still all, so far as I can tell, hold true, nearly two years and many even larger models later. Yes, some of the specific examples we pointed out have been remedied, but the general problems have not.”
Our reading: So far as Marcus can tell, all the core claims he and Davis made in Technology Review still hold true nearly two years and many even larger models later; some of their specific examples have been remedied, but the general problems have not.
No horizon Evidence as of
Limitation: This is a gap in the retrieved evidence, not evidence that the categories had been solved.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Causal claim
“But they still don’t convey genuine comprehension, and so they are still very easy for Ernie and me (or anyone else who cares to try) to break.”
Our reading: Newer LLMs still lack genuine comprehension and are therefore still very easy for anyone to break.
No horizon Evidence as of
Limitation: Benchmark scores and qualitative examples do not establish that anyone could easily elicit failures across all relevant models [e1, e2].
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Conditional forecast
“I am willing to bet SlateStarCodex now (terms to be negotiated) that if OpenAI gives us unrestricted access to GPT-4, whenever that is released, and assuming that is basically the same architecture but with more data, that within a day of playing around with it, Ernie and I will still be able find lots of examples of failures in physical reasoning, temporal reasoning, causal reasoning, and so forth.”
Our reading: If given unrestricted access to GPT-4 and it is basically the same architecture with more data, Marcus and Davis will find many failures in physical, temporal, and causal reasoning within a day.
No deadline stated Evidence as of
Only if: OpenAI gives unrestricted access to GPT-4
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Causal claim
“The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.”
Our reading: The newer models' improvements, such as they are, come primarily from their larger and larger sets of data about how humans use word sequences, which are certainly helpful for pattern-matching machines.
No horizon Evidence as of
Limitation: An optimal scaling prescription is not a decomposition of historical progress.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Observation
“The fourth thing to realize is that we, and by this I mean humanity as a whole, still don’t really have an adequate methodology for building complex cognitive systems.”
Our reading: Humanity lacks an adequate methodology for building complex cognitive systems.
No horizon Evidence as of
Limitation: The empirical study covers a sample of practitioners and systems.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Forecast
“In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user.”
Our reading: In 2029, AI will not be able to reliably build bug-free code over 10,000 lines from a natural-language spec or from talking with a non-expert user, not counting glued-together library code.
Not yet due Evidence as of
Limitation: The cited demonstrations are developer reports, not independent evaluations satisfying the whole contract.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Observation
“You said, for instance in 2015, that (truly) self-driving cars were two years away; you’ve pretty much said the same thing every year since. It still hasn’t happened.”
Our reading: Musk said in 2015 that truly self-driving cars were two years away and has pretty much said the same thing every year since, and truly self-driving cars still had not arrived.
No horizon Evidence as of
Limitation: Several predictions are documented through contemporary reporting because readable original transcripts were unavailable.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Observation
“I don’t know what you have planned for Optimus, but I can promise you that the AGI that you would need for a general-purpose domestic robot (where every home is different, and each poses its own safety risks) is way beyond what you would need for a car that drives on roads that are more or less engineered the same way from one town to the next.”
Our reading: The AGI needed for a general-purpose domestic robot, where every home is different and poses its own safety risks, is way beyond what is needed for a car driving on roads engineered more or less the same from town to town.
No horizon Evidence as of
Limitation: Household robotics evidence includes simulation and constrained kitchen tasks.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Forecast
“In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc.”
Our reading: In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc., where the key is going beyond the literal text.
Not yet due Evidence as of
Limitation: The studies do not establish current frontier performance across the complete task or what systems will achieve in 2029.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Observation
“We still don’t have anything like stable or trustworthy solutions for common sense, reasoning, language, or analogy. Truth is, I have been using this same pie chart for about 5 years, and the situation has scarcely changed.”
Our reading: AI has no stable or trustworthy solutions for common sense, reasoning, language or analogy, and this has scarcely changed in about five years.
No horizon Evidence as of
Limitation: The studies use different tasks, models and evaluation methods.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Observation
“Deep learning is pretty decent at, for example, recognizing objects, but not nearly as good at planning, or reading, or language comprehension.”
Our reading: Deep learning is decent at object recognition but much worse at planning, reading and language comprehension.
No horizon Evidence as of
Limitation: Benchmark success does not establish broad real-world comprehension.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Forecast
“In 2029, AI will not be able to take arbitrary proofs from the mathematical literature written in natural language and convert them into a symbolic form suitable for symbolic verification.”
Our reading: In 2029, AI will not be able to take arbitrary proofs written in natural language from the mathematical literature and convert them into a symbolic form suitable for symbolic verification.
Not yet due Evidence as of
Limitation: The research and developer reports were read, but their generated proof artifacts were not independently rerun in this assessment.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Observation
“Even within perception 3D perception remains a challenge, and scene understanding is by no means solved.”
Our reading: 3D perception and scene understanding remain unsolved for current AI.
No horizon Evidence as of
Limitation: The studies cover particular tasks rather than every form of perception or scene understanding.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Observation
“But for the kinds of AI techniques that are currently popular, and which rely more on big data than on reasoning, it’s a very serious issue.”
Our reading: The long-tail problem of rare events is a very serious issue for the currently popular AI techniques, which rely more on big data than on reasoning.
No horizon Evidence as of
Limitation: Evidence is strongest for visual recognition.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Forecast
“In 2029, AI will not be able to work as a competent cook in an arbitrary kitchen (extending Steve Wozniak’s cup of coffee benchmark).”
Our reading: In 2029, AI will not be able to work as a competent cook in an arbitrary kitchen.
Not yet due Evidence as of
Limitation: The cooking demonstrations and unfamiliar-home results concern different systems and tasks.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: MixedSaid
Causal claim
“Debugging with deep learning is wicked hard, because nobody (a) really understands how it works, and (b) knows how to fix problems other to collect more data and add more layers and so forth.”
Our reading: Debugging deep learning is wicked hard because nobody really understands how it works or knows how to fix problems other than collecting more data, adding more layers and so forth.
No horizon Evidence as of
Limitation: The evidence concerns neural language models, not every deep learning application.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Causal claim
“The thing is, it’s easy to convince yourself that AI problems are much easier than they are actually are, because of the long tail problem.”
Our reading: The long tail problem makes it easy to believe AI problems are much easier than they actually are.
No horizon Evidence as of
Limitation: The closest empirical evidence uses a simulated robot task.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Forecast
“And it’s why I absolutely guarantee you won’t be shipping level 5 self-driving cars this year or next, no matter what you might have told Chris Anderson at TED.”
Our reading: Tesla will not ship level 5 self-driving cars in 2022 or 2023.
Deadline passed Evidence as of
Limitation: The assessment concerns delivered capability, not whether Tesla sold an option named Full Self-Driving.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Forecast
“In 2029, AI will not be able to watch a movie and tell you accurately what is going on (what I called the comprehension challenge in The New Yorker, in 2014).”
Our reading: In 2029, AI will not be able to watch a movie and accurately describe what is going on.
Not yet due Evidence as of
Limitation: The search does not establish that no qualifying system exists.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Observation
“But you and I both know that they continue to be a major problem that as yet has no known robust solution.”
Our reading: Outliers remain a major AI problem with no known robust solution.
No horizon Evidence as of
Limitation: The absence of a broadly validated solution is an inference from the inspected public research record, not proof that every possible solution was absent.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: SupportedSaid
Observation
“That’s where the deep understanding of language should live—but, despite the PR, it isn’t there.”
Our reading: Imagen lacks deep understanding of language despite Google's PR.
No horizon Evidence as of
Limitation: The result depends on interpreting deep understanding through the role-and-relationship behavior discussed in the post.
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Observation
“As Ernie Davis, Scott Aaronson, and I showed, a week or so after the release, it’s easily fooled, especially with complex sentences and words like “similar” that would require a deep understanding of language to be interpreted correctly”
Our reading: DALL-E 2 is easily fooled, especially by complex sentences and words requiring deep language understanding.
No horizon Evidence as of
Limitation: Unresolved does not mean the claim was false.
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Causal claim
“Since we know that Imagen can draw images of horses riding astronauts (if the instructions are specific enough) we know that the failure to draw a horse riding an astronaut given the prompt “a horse riding an astronaut” can only have one explanation: the system doesn’t understand what the prompt “a horse riding an astronaut” means.”
Our reading: Since Imagen can draw horses riding astronauts when the instructions are specific enough, its failure to draw one from the prompt 'a horse riding an astronaut' can only be explained by the system not understanding what that prompt means.
No horizon Evidence as of
Limitation: The available demonstration is not a controlled causal experiment.
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Causal claim
“And, no, it wasn’t about common sense at all.”
Our reading: Imagen's failure on 'a horse riding an astronaut' was not due to common sense.
No horizon Evidence as of
Limitation: The demonstration provides no repeated-sample success rates or controlled comparison separating wording from scene plausibility [e5].
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Observation
“(In reality, Imagen has no idea what weight, mass, or density are.)”
Our reading: Imagen has no understanding of weight, mass or density.
No horizon Evidence as of
Limitation: The behavioral interpretation makes the claim testable but does not settle philosophical questions about understanding.
Source: Horse rides astronaut
Assessment: ContradictedSaid
Forecast
“They do not gravitate towards human language, and they never develop a cohesive undestanding of the world, no matter how much experience they have.”
Our reading: Systems like GPT-3 do not gravitate towards human language, and they never develop a cohesive understanding of the world, no matter how much experience they have.
Settled before its deadline Evidence as of
Limitation: The contradiction applies to the language-learning component of the combined assertion.
Source: Noam Chomsky and GPT-3
Assessment: UnresolvedSaid
Causal claim
“There is no intention to create misinformation—but also no capacity to avoid it, because fundamentally, GPT-3 is a model of how words relate to one another, not a model of how language might relate to the perceived world.”
Our reading: GPT-3 cannot avoid producing misinformation because it models how words relate to each other, not how language relates to the world.
No horizon Evidence as of
Limitation: The evidence is about error rates, not about mechanism.
Source: Noam Chomsky and GPT-3
Assessment: UnresolvedSaid
Conditional forecast
“The bottom line is this, something that AI once cherished but has now forgotten: If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.”
Our reading: If we are to build AGI, we will need to learn something from humans: how they reason and understand the physical world, and how they represent and acquire language and complex concepts.
No deadline stated Evidence as of
Only if: if we are to build AGI
Assessment: UnresolvedSaid
Observation
“As I have written, I am fine with thinking of Gato as an “Alt Intelligence”— an interesting exploration in alternative ways to build intelligence—but we need to take it in context: it doesn’t work like the brain, it doesn’t learn like a child, it doesn’t understand language, it doesn’t align with human values, and it can’t be trusted with mission-critical tasks.”
Our reading: Gato does not work like the brain, learn like a child, understand language, or align with human values, and cannot be trusted with mission-critical tasks.
No horizon Evidence as of
Limitation: The evidence is the developers' original report, rather than independent validation of all five assertions [e6].