Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

Seven lies in four sentences

We recorded 6 claims from this post. 3 were checked against evidence. 3 can’t be checked. 2 more are held. Read the post at its source.

Claims from this post (6)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: SupportedSaid

    “Rumors that LLMs have solved then hallucination problem are greatly exaggerated.”

    What we tested: LLMs have not solved the hallucination problem.

    What happened: The revised paper identifies hallucination as an unresolved contemporary problem and reports experiments reducing it through verification. The event date identifies this current-state assessment, not the undisclosed dates of individual experimental runs. (arXiv; Shehzaad Dhuliawala and colleagues, )

  2. Assessment: UnresolvedSaid

    “By my count, though I acknowledge I may have missed one, there are at least 7 falsehoods.”

    What we tested: A recent Microsoft Bing answer about the House Speaker contained at least seven falsehoods.

  3. Assessment: Can’t be checkedSaid

    “The worst part is not that every single sentence contains at least one lie, but that the whole thing sounds detailed and real. To someone who wasn’t following matters closely, it might well all sound plausible.”

    What we tested: The worst part of Bing's false answer is not that every sentence contains at least one falsehood but that the whole thing sounds detailed and real, so that to someone who wasn't following matters closely it might well all sound plausible.

  4. Assessment: Can’t be checkedSaid

    “Ezra and I harped on their propensity to generate bullshit because bullshit was and is their fundamental product.”

    What we tested: Bullshit was and still is the fundamental product of LLMs.

  5. Assessment: UnresolvedSaid

    “This all happens not because LLMs are smart, in the sense of being able to compare what they say with the world, but simply because they are good at making new sentences that sound old sentences, the essence of autocomplete.”

    What we tested: LLMs produce plausible falsehoods because they mimic existing sentences like autocomplete rather than checking what they say against the world.

  6. Assessment: Can’t be checkedSaid

    “we really ought to commit to building AI that can.”

    What we tested: The field should commit to building AI that can distinguish facts from fiction.

Held from this post (2)

These records did not pass every check, so nothing about them is published. Held is not a finding.