Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

LLMs + Coding Agents = Security Nightmare

We recorded 12 claims from this post. 5 were checked against evidence. 7 can’t be checked. 8 more are held. Read the post at its source.

Claims from this post (12)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: SupportedSaid

    “Gary has written here endlessly about the troubles with reliability, apparently inherent, in LLMs. If you write code with an LLM, you are asking for trouble; the kind of garden-variety hallucinations that Gary has described in, for example, biographies, have parallels in LLM-generated code.”

    What we tested: LLMs have apparently inherent reliability problems, and code written with them contains hallucinations, making LLM coding risky.

    What happened: The reported comparison found invented dependencies in foundation models and coding agents, supporting a concrete reliability and supply-chain risk. It also found predominantly clean foundation-model outputs and substantial reductions with agent reasoning and validation, which limit any claim of pervasive or unavoidable failure. (Trend Micro, )

  2. Assessment: SupportedSaid

    “We are particularly worried about agents that software developers are starting to use, because they are often granted considerable authority and access to far-ranging tools, opening up immense security vulnerabilities.”

    What we tested: The authors are particularly worried about agents that software developers are starting to use, because these agents are often given considerable authority and access to far-ranging tools, which opens up immense security vulnerabilities.

    What happened: The researcher reported that Copilot could persistently edit workspace files without approval, then demonstrated injected instructions changing its approval configuration and executing commands. June 29 is the documented disclosure date by which the exploit existed. (Embrace The Red — Johann Rehberger, )

  3. Assessment: Can’t be checkedSaid

    “(Nvidia advised, rightly, that everyone should disable Auto-Run mode if they’ve activated it, but we fear that many users may use it anyway, because it is so fast.)”

    What we tested: Many Cursor users may keep using Auto-Run mode despite the security risk because it is fast.

  4. Assessment: Can’t be checkedSaid

    “the user has the option to accept a code change or not, but with so many changes to make at such a rapid pace, a developer in hurrry (which is almost every developer) could easily miss an attack, and ok the change or run a command”

    What we tested: Even with manual approval, hurried developers could easily approve malicious changes suggested by coding agents.

  5. Assessment: Can’t be checkedSaid

    “Given current implementations of the technology, it's hard to imagine enough patches on the planet to thwart them all.”

    What we tested: Given current implementations, it is unlikely that patches can thwart all data-injection attacks on generative AI systems.

  6. Assessment: Can’t be checkedSaid

    “As long as we have agents roaming the internet and otherwise incorporating data that they don’t fully understand – and LLMs don’t ever fully understand the data they are leveraging – there is enormous risk.”

    What we tested: LLMs never fully understand the data they leverage.

  7. Assessment: SupportedSaid

    “And that means that high levels of agency, combined with the ease of manipulation of LLM-based software, is a recipe for chaos.”

    What we tested: High agency combined with easily manipulated LLM software leads to chaos.

    What happened: NVIDIA describes an August Black Hat demonstration in which a malicious issue redirected a computer-use agent into executing an attacker command, and a separate Cursor dependency-installation route that also granted remote control. Cursor blocked the initial direct injection, and the report makes autonomous execution and insufficient isolation central conditions of the successful attacks. (NVIDIA, )

  8. Assessment: UnresolvedSaid

    “Agency plus LLMs have already led multiple “vibe coders” to lose databases, with new reports coming in every day or two, stemming simply from the unreliability inherent in LLMs (that Gary has so often harped on).”

    What we tested: Agency combined with LLMs has already led multiple vibe coders to lose databases, with new reports every day or two, and these losses stem simply from the unreliability built into LLMs.

  9. Assessment: UnresolvedSaid

    “The bad news is that although this one was stopped, many others won’t be. No one patch would be enough, or even a thousand; there are just too many variations on the overall theme”

    What we tested: Many attacks on AI coding tools won't be stopped; not even a thousand patches would suffice given the number of variations.

  10. Assessment: Can’t be checkedSaid

    “The best defense would be not using agentic coding altogether.”

    What we tested: The best security defense is to not use agentic coding at all.

  11. Assessment: Can’t be checkedSaid

    “But the tools are so seductive that we doubt many developers will resist.”

    What we tested: Few developers are likely to refrain from using agentic coding tools.

  12. Assessment: Can’t be checkedSaid

    “Sure the advice is good, as far is it goes, but if the goods inside are valuable enough, motivated thieves may well still find a way.”

    What we tested: Even with mitigations, motivated attackers may still compromise valuable systems using coding agents.

Held from this post (8)

These records did not pass every check, so nothing about them is published. Held is not a finding.