Sorry, GenAI is NOT going to 10x computer programming
We recorded 7 claims from this post. 2 were checked against evidence. 5 can’t be checked. 1 more is held. Read the post at its source.
Claims from this post (7)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
“The data are coming in – and it’s not.”
What we tested: Emerging data show generative AI is not improving programmer productivity by a factor of 10.
What happened: The published study reported no significant improvement in pull-request cycle time or throughput. Its results support the claim about the contemporary research record, although they cannot establish a universal absence of benefits. (Uplevel, )
Assessment: SupportedSaid
“Two recent studies show nothing like “10x” improvement :”
What we tested: Two recent studies of generative AI coding tools show nothing like a 10x productivity improvement.
What happened: The primary announcement identifies approximately 800 developers and reports no significant improvement in its efficiency measures. That is plainly below tenfold. (Uplevel, )
Assessment: Can’t be checkedSaid
“Yet another earlier study (with earlier models) showed evidence of users writing less secure code, which of course could lead to a net loss of productivity long term.”
What we tested: Less secure code written with AI coding assistants could lead to a net loss of productivity in the long term.
Assessment: Can’t be checkedSaid
“A good IDE (versus none) probably is a much bigger, much less expensive, much less hyped improvement that helps more people more reliably.”
What we tested: Having a good IDE, compared with having none, is probably a much bigger, much less expensive, and much less hyped improvement than generative AI coding tools, one that helps more people more reliably.
Assessment: Can’t be checkedSaid
“10x-ing requires deep conceptual understanding – exactly what GenAI lacks.”
What we tested: Generative AI cannot 10x programming because doing so requires deep conceptual understanding, which GenAI lacks.
Assessment: Can’t be checkedSaid
“Writing lines of code without that understanding can only help so much.”
What we tested: Generating code without conceptual understanding provides only limited productivity help.
Assessment: Can’t be checkedSaid
“Use it to type faster, not as a substitute for clear thinking about algorithms + data structures.”
What we tested: Programmers should use generative AI to type faster, not as a substitute for clear thinking about algorithms and data structures.
Held from this post (1)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The criteria are ambiguous. Our reading, not assessed: Evidence points to modest productivity improvements from generative AI coding tools, with possible security and technical-debt costs, not 10x improvement.