Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?

We recorded 12 claims from this post. 7 were checked against evidence. 5 can’t be checked. 6 more are held. Read the post at its source.

Claims from this post (12)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: Can’t be checkedSaid

    “In 2017, a team of Google researchers invented the Transformer, arguably the biggest advance in AI in the last 10 years.”

    What we tested: The Transformer, invented by a team of Google researchers in 2017, is arguably the biggest advance in AI in the last 10 years.

  2. Assessment: ContradictedSaid

    “we were right to be skeptical; so far, four and half years later, Duplex hasn’t gotten a whole lot further.)”

    What we tested: Four and a half years after its announcement, Google Duplex has not progressed much past its limited demo stage.

    What happened: Google reported over a million completed bookings and over three million business-information updates across eight countries, alongside the quoted automation rate. These reported deployments show substantial operational progress beyond a limited demonstration. (Google, )

  3. Assessment: SupportedSaid

    “The dirty secret in current AI is that for all the bloviating about how current systems have mastered human language, they are still really weak on compositionality, which most linguists (since Frege, over a century ago) would agree is at the very core of how language works.”

    What we tested: Current AI systems are still very weak at compositionality, which is central to how language works.

    What happened: Winoground found large human-model gaps across several vision-language model families on image-caption matching that depends on composition. The finding supports a broad weakness within the tested contemporary systems. (arXiv; Thrush and colleagues, )

  4. Assessment: Can’t be checkedSaid

    “For me, compositionality has always been the touchstone for AI; when I say AI has hit a wall, it’s compositionality, more than anything else, that I have been talking about:”

    What we tested: AI has hit a wall, and that wall is mainly compositionality.

  5. Assessment: SupportedSaid

    “This should be a piece of cake, but for currently popular systems, it’s not.”

    What we tested: Currently popular vision-language systems cannot reliably tell apart captions that use the same words in a different order.

    What happened: The group score requires all relevant caption-image comparisons to be correct; every tested model scored below its random baseline. Some models exceeded chance on the less demanding text score, but remained far behind humans and did not reliably solve the complete matching task. (arXiv; Thrush and colleagues, )

  6. Assessment: UnresolvedSaid

    “Notwithstanding the rumors about Google Imagen, nobody has yet publicly demonstrated a machine that can relate the meanings of sentences to their parts the way a five-year-old child can.”

    What we tested: No one, Imagen included, has publicly shown a machine that relates sentence meanings to their parts as well as a five-year-old.

  7. Assessment: UnresolvedSaid

    “But many people have claimed over the years to solve the problem, and none of those systems have proven to be reliable; every proposed solution has been like Clever Hans, working in dim light, leveraging large databases to some degree, but falling apart upon careful inspection; they might get 60% on some task, but they never really master it.”

    What we tested: Every claimed solution to compositionality so far has proven unreliable and falls apart on careful inspection.

  8. Assessment: UnresolvedSaid

    “Google cloaks its current work the trappings of science (article preprints with citations and graphs and so forth) but when it comes to community inspection, they’ve systematically opted out.”

    What we tested: Google has systematically refused to let the scientific community inspect models such as Imagen and LaMDA.

  9. Assessment: Can’t be checkedSaid

    “(Dare I make obvious guess? They probably tried and failed, and don’t want to admit it.)”

    What we tested: Google probably tested Imagen on Winoground, got poor results, and doesn't want to admit it.

  10. Assessment: SupportedSaid

    “The progress that Alexander is vaunting shows that the system did better on a handful of items than DALL-E2 (which was not even state of the art at compositionality when it appeared), but not that the newer system (Imagen) systematically groks compositionality”

    What we tested: Alexander's test shows only that Imagen beat DALL-E 2 on a handful of items, not that Imagen systematically understands compositionality.

    What happened: The page specifies five prompts, reports DALL-E 2 at 0/5 and Imagen at 3/5 under Alexander's scoring, and contains no systematic retest of earlier successes. Alexander also expressly declines to claim full mastery, so the methodological limitation does not establish that he claimed otherwise. (Astral Codex Ten, )

  11. Assessment: Can’t be checkedSaid

    “Having a Silicon Valley celebrity like Alexander weigh in for Google is great PR for Google, but not so great for the public understanding of AI.”

    What we tested: Having a Silicon Valley celebrity like Alexander weigh in for Google is great PR for Google but not so great for the public understanding of AI.

  12. Assessment: Can’t be checkedSaid

    “A will to believe in AI will never replace the need for careful science.”

    What we tested: A will to believe in AI will never replace the need for careful science.

Held from this post (6)

These records did not pass every check, so nothing about them is published. Held is not a finding.