“The bottom line is this: LLMs are certainly coding more, but it’s not clear that the code they are creating is secure or trustworthy.”
Our reading, the sentence that was checked:
LLMs are writing more code, but it is unclear whether that code is secure or trustworthy.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, About that Matt Shumer post that has nearly 50 million views, paragraph 14.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-about-that-matt-shumer-post-that-1636a3-a4
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-02-11. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- LLMs are certainly coding more
- it's not clear that the code they are creating is secure or trustworthy
Our interpretation
- Two parts: (a) LLMs are writing a growing share of code; (b) the security and trustworthiness of that code had not been established as of the post.
- Ongoing-state claim with no named period. The target period runs from late 2024 to the post date: the most recent data on AI's share of code available before the post covered late 2024 and early 2025, and the security studies date from 2025.
- Part (b) is a claim of uncertainty. Studies showing frequent vulnerabilities support it. Strong evidence that AI code is reliably secure would contradict it.
Counts as supported if: Data show the share of code written with AI rising, and credible studies show AI-generated code frequently has security flaws, so its trustworthiness is not established.
Counts as contradicted if: Data show AI coding is not growing, or credible evidence before the post shows AI-generated code is reliably secure and trustworthy.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The claim has two parts: (a) LLMs were writing more code, and (b) the security and trustworthiness of that code had not been established by the post on 11 February 2026. The test period runs from 1 October 2024 to 11 February 2026. On (a), a peer-reviewed study in Science, publicized on 22 January 2026, found that AI-assisted coding in the US rose from about 5% in 2022 to nearly 30% in the last quarter of 2024. The data came from Python contributions on GitHub (e1). On (b), Veracode reported on 30 July 2025 that LLMs introduced vulnerabilities in 45% of its test cases. It said models were getting better at coding accurately but not at security (e2). A preprint posted on 2 December 2025 found that 61% of solutions from SWE-Agent with Claude 4 Sonnet were functionally correct, but only 10.5% were secure (e3). The packet has no evidence from before the post showing that AI-generated code was reliably secure. Both parts meet the support condition.
Limitations
- The code-share data cover open-source Python on GitHub through early 2025. They do not cover all code or the months just before the post.
- The security evidence comes from benchmark tasks in a vendor report and a preprint, not from audits of deployed code.
- Correctness is measured only on the preprint's benchmark tasks and mentioned in general terms by Veracode. Maintainability and other aspects of trustworthiness are not measured.
Evidence (3 items)
Supports the claim: Complexity Science Hub via EurekAlert! (study in Science),
“In the U.S., AI-assisted coding jumped from around 5% in 2022 to nearly 30% in the last quarter of 2024.”
Why it matters here: A peer-reviewed measurement shows the share of AI-assisted code rising sharply. This supports 'LLMs are certainly coding more'.
- Where
- paragraph 5
- Publisher
- Complexity Science Hub via EurekAlert! (study in Science)
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Supports the claim: Veracode,
“Our research shows models are getting better at coding accurately but are not improving at security.
Why it matters here: The same report found vulnerabilities in 45% of cases and no improvement in security over time. This supports the claim that security was not established.
- Where
- paragraph 8
- Publisher
- Veracode
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: arXiv (Zhao et al., Carnegie Mellon University),
Although 61% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 10.5% are secure.
Why it matters here: Agent-written code that passes functional tests is mostly insecure on real-world tasks. Code that works is not necessarily trustworthy.
- Where
- Abstract
- Publisher
- arXiv (Zhao et al., Carnegie Mellon University)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, About that Matt Shumer post that has nearly 50 million views
The bottom line is this: LLMs are certainly coding more, but it’s not clear that the code they are creating is secure or trustworthy.
- Where
- paragraph 14
- Published
- 2026-02-11
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-about-that-matt-shumer-post-that-1636a3-a4: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-about-that-matt-shumer-post-that-1636a3-a4.
HeldBy this site
The final-text check failed: rationale_supported_by_evidence; overclaims: Trustworthiness beyond security, such as correctness and maintainability, is not measured directly..
About assessment marcus-about-that-matt-shumer-post-that-1636a3-a3, which was not published.