No, Virginia, AGI is not imminent
We recorded 20 claims from this post. 12 were checked against evidence. 8 can’t be checked. 2 more are held. Read the post at its source.
Claims from this post (20)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“I seriously doubt it.”
What we tested: Marcus seriously doubts that AI is close to beating typical human performance on virtually all cognitive tasks.
Assessment: UnresolvedSaid
“in my view it’s a subtle redefinition that dramatically moves the goal posts.”
What we tested: In Marcus's view, Legg's definition of AGI as better than typical human performance on virtually all cognitive tasks is a subtle redefinition that dramatically moves the goal posts, compared with how Legg and Marcus publicly discussed the term last year.
Assessment: SupportedSaid
“But it is also well-known throughout the industry that benchmarks are often quite gameable.”
What we tested: AI benchmarks are often quite gameable.
What happened: The review documents benchmark shortcuts across vision, language, reinforcement learning and algorithmic decision-making. This supports a recurring, publicly recognized problem rather than an isolated vulnerability. (arXiv, )
Assessment: SupportedSaid
“Basically, beating benchmarks doesn’t tell you as much as you might think. Often when you do, it’s a statistical artifact.”
What we tested: Beating AI benchmarks is often a statistical artifact and says less than people assume.
What happened: Models scoring well on MNLI failed controlled examples that invalidated their syntactic shortcuts. The experiment links apparent benchmark competence to heuristics that do not reliably perform the intended inference. (arXiv, )
Assessment: UnresolvedSaid
“This is even more true four years later, now that we have systems that ingest massive amounts of data from both public and commercial sources, quite possibly including eg sample law school and medical exams etc.”
What we tested: Because current systems ingest massive data possibly including sample exams, benchmark success is even less indicative of intelligence than in 2019.
Assessment: SupportedSaid
“The dirty little secret that pretty much everyone in the AI community knows is that doing well on a benchmark is not actually tantamount to being intelligent.”
What we tested: Doing well on a benchmark does not amount to being intelligent.
What happened: CheckList identifies basic linguistic failures in commercial and leading models across three tasks despite strong benchmark results. These empirical failures support the inference that benchmark success alone does not establish broad, reliable competence. (arXiv, )
Assessment: UnresolvedSaid
“Without a sense of what it is in the training corpus, it’s almost impossible to conclude anything about what a machine truly understands and how general that is.”
What we tested: Without knowing the training corpus, one can hardly conclude what a machine understands or how general it is.
Assessment: SupportedSaid
“Machines often solve them for the wrong reasons. Building them is an unsolved problem within AI.”
What we tested: Machines often solve benchmarks for the wrong reasons and building good benchmarks is an unsolved problem.
What happened: The 2023 version describes benchmark design as an ongoing difficulty and reports that single-sentence label leakage persists despite mitigation efforts. The event date denotes that research-state assessment, not the date of every underlying experiment. (arXiv; Michael Saxon and colleagues, )
Assessment: UnresolvedSaid
“We are not even close to machines that exceed the capacity of typical humans to cope with unusual circumstances while driving”
What we tested: Machines are not close to exceeding typical humans at coping with unusual circumstances while driving.
Assessment: Can’t be checkedSaid
“Humans can write summaries and reliably answer many questions about original films and television programs that they watch; I doubt LLMs (or any other extant AI technology) can do this at all.”
What we tested: Marcus doubts LLMs or any extant AI can summarize and reliably answer questions about original films and TV programs.
Assessment: UnresolvedSaid
“Humans can write summaries of things without hallucination; LLMs cannot do so reliably.”
What we tested: LLMs cannot reliably write summaries without hallucinating.
Assessment: SupportedSaid
“Humans can learn rules of games like chess from modest amounts of explicit instruction; LLMs can’t stick to explicit rules.”
What we tested: LLMs cannot stick to explicit rules such as the rules of chess.
What happened: The accompanying demonstrations also document illegal GPT-3.5 moves with specialized prompting and a GPT-4 failure to obey the instruction to output only its own move. The same report includes successful legal games, so it supports unreliable adherence rather than total inability. (Adam Yedidia, LessWrong, )
Assessment: Can’t be checkedSaid
“And frankly, we still have a long way to go on all the things I always harp on: reasoning, compositionality, planning, factuality, and so on.”
What we tested: AI still has a long way to go on reasoning, compositionality, planning, and factuality.
Assessment: Can’t be checkedSaid
“No AGI worth its salt would brick multidigit arithmetic.”
What we tested: A genuine AGI must reliably do multidigit arithmetic.
Assessment: Can’t be checkedSaid
“It’s a huge downgrade in our goals for artificial general intelligence to say “better than the average untrained human on many tasks”; the point is that we should expect machines to do what we ask them to do;”
What we tested: Saying AGI means 'better than the average untrained human on many tasks' is a huge downgrade in our goals for AGI, because we should expect machines to do what we ask them to do.
Assessment: Can’t be checkedSaid
“An artificial general intelligence should do the things that it is asked to do, and decline to do those that it can’t, and have the wisdom to know the difference.”
What we tested: An AGI should do what it is asked, decline what it cannot do, and know the difference.
Assessment: Can’t be checkedSaid
“Dropping from “as smart as the Star Trek computer” to better than Joe Sixpack should not be our goal.”
What we tested: The AI field should not lower its AGI goal from Star Trek computer-level to merely beating an average person.
Assessment: UnresolvedSaid
“18 months ago, before the LLM euphoria, we all took for granted that AGI wasn’t just about beating ordinary humans.”
What we tested: Around early 2022, before LLM euphoria, the field assumed AGI meant more than beating ordinary humans.
Assessment: SupportedSaid
“What would make artificial general intelligence general is generality: the ability to cope with new things. And that’s precisely where current AI is still lacking.”
What we tested: Generality, the ability to cope with new things, is what would make AGI general, and that is precisely where current AI is still lacking.
What happened: ConceptARC tested generalization across variations of concepts. Humans substantially exceeded specialized solvers; GPT-4 scored below 30% on 15 of 16 concepts, supporting a substantial limitation under these test conditions. (arXiv; Arseny Moskvichev, Victor Vikram Odouard and Melanie Mitchell, )
Assessment: Can’t be checkedSaid
“Let’s not dumb down our standards, and wind up with a bunch of machines we can’t rely on, just so that big tech can prematurely declare victory, when there is plenty of work left to do.”
What we tested: AGI standards should not be lowered so big tech can prematurely declare victory, since much work remains.
Held from this post (2)
These records did not pass every check, so nothing about them is published. Held is not a finding.