Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

This one important fact about current AI explains almost everything

We recorded 10 claims from this post. 4 were checked against evidence. 6 can’t be checked. 9 more are held. Read the post at its source.

Claims from this post (10)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: Can’t be checkedSaid

    “Those who do not have wasted a tremendous amount of money.”

    What we tested: People in AI who do not understand that machine learning handles outliers poorly have wasted a tremendous amount of money.

  2. Assessment: Can’t be checkedSaid

    “If it’s not directly in the training set, but merely superficially like some previous examples, there could be trouble.”

    What we tested: When an input is not directly in the training set but only superficially resembles earlier examples, machine learning systems may fail.

  3. Assessment: Can’t be checkedSaid

    “Everyone commenting on or thinking about AI should learn it by heart, if not verbatim, certainly the gist.”

    What we tested: Everyone commenting on or thinking about AI should learn by heart, if not word for word then at least the gist, Koopman's point that machine learning trains on many examples and, when it meets something it lacks many examples of, may have no idea what is going on.

  4. Assessment: UnresolvedSaid

    “The AI world pretty much divides into two groups: those who understand why current machine learning techniques suck at outliers, and therefore struggle at things like driverless cars and high-level reasoning in unusual circumstances — and those who don’t.”

    What we tested: Current machine learning techniques handle outliers poorly and so struggle with driverless cars and high-level reasoning in unusual circumstances.

  5. Assessment: UnresolvedSaid

    “An entire industry has been built - and will collapse - because people aren’t getting it.”

    What we tested: The industry built on generative AI will collapse because people don't understand the outlier problem.

  6. Assessment: Can’t be checkedSaid

    “The people who have temporarily gotten rich or famous on AI have done so by pretending that this outlier problem simply doesn’t exist, or that a remedy for it is imminent.”

    What we tested: The people who have temporarily gotten rich or famous on AI have done so by pretending that the outlier problem doesn't exist or that a remedy for it is imminent.

  7. Assessment: SupportedSaid

    “Over a quarter century later, handling outliers is still the Achilles’ Heel of neural networks.”

    What we tested: More than a quarter century after his 1998 work, handling outliers (distribution shift) is still the Achilles' heel of neural networks.

    What happened: The evaluation reports failures across several unfamiliar visual domains, supporting a recurring weakness in contemporary generative neural systems. It also finds improvements from in-context learning, while reporting that this approach remains vulnerable to distribution shifts. (arXiv; Zhang and colleagues, )

  8. Assessment: MixedSaid

    “Symbolic systems have always been good for outliers; neural networks have always struggled with them. For 30 years, this has been a constant.”

    What we tested: For 30 years, neural networks have struggled with outliers while symbolic systems handled them well.

    What held: The gSCAN experiments report substantial failures when neural models must apply compositional rules to unfamiliar combinations. This supports the recurring neural-difficulty part of the claim, but does not establish symbolic superiority across the entire period. (arXiv; Ruis and colleagues, )

    What did not: In a comparison of seven models, the meta-trained neural approach combined systematicity and flexibility that the symbolic and ordinary neural comparators did not jointly achieve. This contradicts a stable categorical division between successful symbolic generalization and unsuccessful neural generalization. (Nature, )

  9. Assessment: Can’t be checkedSaid

    “Once you do, almost everything that people like Altman and Musk and Kurzweil are currently saying about AGI being nigh seems like sheer fantasy, on par with imagining that really tall ladders will soon make it to the moon.”

    What we tested: Once one understands that current neural networks struggle with outliers, almost everything that people like Altman, Musk and Kurzweil are currently saying about AGI being near seems like sheer fantasy, like imagining that really tall ladders will soon reach the moon.

  10. Assessment: Can’t be checkedSaid

    “You can’t think we are close to AGI once you realize that as yet we have no general solution to the outlier problem. You just can’t.”

    What we tested: Marcus asserts emphatically that no one who realizes we do not yet have a general solution to the outlier problem can think we are close to AGI.

Held from this post (9)

These records did not pass every check, so nothing about them is published. Held is not a finding.