Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

CONFIRMED: LLMs have indeed reached a point of diminishing returns

We recorded 13 claims from this post. 6 were checked against evidence. 7 can’t be checked. 5 more are held. Read the post at its source.

Claims from this post (13)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: UnresolvedSaid

    “Just as I argued here in April 2024, LLMs have reached a point of diminishing returns.”

    What we tested: LLMs have reached a point of diminishing returns from scaling.

  2. Assessment: Can’t be checkedSaid

    “The economics are likely to be grim.”

    What we tested: The economics of LLMs are likely to be grim.

  3. Assessment: Can’t be checkedSaid

    “When everyone realizes this, the financial bubble may burst quickly; even NVidia might take a hit, when people realize the extent to which its valuation was based on a false premise.”

    What we tested: Once people realize LLM economics don't work, the AI financial bubble may burst quickly and even Nvidia might take a hit.

  4. Assessment: UnresolvedSaid

    “Given the cost of chips, profits will be elusive.”

    What we tested: Because chips are costly, LLM companies will struggle to make profits.

  5. Assessment: UnresolvedSaid

    “LLMs such as they are, will become a commodity; price wars will keep revenue low.”

    What we tested: LLMs such as they are will become a commodity, and price wars will keep revenue low.

  6. Assessment: Can’t be checkedSaid

    “LLMs will not disappear, even if improvements diminish, but the economics will likely never make sense: additional training is expensive, the more scaling, the more costly.”

    What we tested: LLMs will not disappear even if improvements diminish, but their economics will likely never make sense because additional training is expensive and more scaling is more costly.

  7. Assessment: SupportedSaid

    “Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs.”

    What we tested: Marcus and many others, including Melanie Mitchell, Subbarao Kambhampati, Emily Bender and Ernie Davis, have long emphasized that there are principled limits with LLMs.

    What happened: Bender and Koller's paper makes an explicit conceptual-limit argument more than four years before the post. The metadata gives July 2020; July 31 is used as a conservative availability bound. (Association for Computational Linguistics, )

  8. Assessment: UnresolvedSaid

    “Meanwhile, precious little investment has been made in other approaches.”

    What we tested: Very little investment has gone into AI approaches other than LLMs.

  9. Assessment: Can’t be checkedSaid

    “If LLMs won’t get the US to trustworthy AI, and our adversaries invest in alternative approaches, we could easily be outfoxed.”

    What we tested: If LLMs don't lead to trustworthy AI and adversaries invest in alternatives, the US could easily be outfoxed.

  10. Assessment: Can’t be checkedSaid

    “The US has been putting all its AI eggs in the LLM basket, and that may well prove to be an epic, massive mistake.”

    What we tested: The US has been putting all its AI eggs in the LLM basket, and that may well prove to be an epic, massive mistake.

  11. Assessment: UnresolvedSaid

    “If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation.”

    What we tested: If enthusiasm for GenAI dwindles and market valuations plummet, AI and LLMs will not disappear, and LLMs will still have their place as tools for statistical approximation.

  12. Assessment: Can’t be checkedSaid

    “But that place may be smaller; it is entirely possible that LLMs on their own will never live up to last year’s wild expectations.”

    What we tested: LLMs' role may be smaller than hoped, and LLMs on their own may never live up to the wild expectations of 2023.

  13. Assessment: Can’t be checkedSaid

    “Reliable, trustworthy AI is surely achievable, but we may need to go back to the drawing board to get there.”

    What we tested: Reliable, trustworthy AI is achievable, but getting there may require new approaches beyond current LLMs.

Held from this post (5)

These records did not pass every check, so nothing about them is published. Held is not a finding.