Decoding (and debunking) Hard Fork’s Kevin Roose
We recorded 16 claims from this post. 9 were checked against evidence. 7 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (16)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Observation
“Anyone can download the code, and probably most or all of the AI systems were trained on this and other similar examples.”
Our reading: Most or all AI coding systems were probably trained on existing fridge-photo recipe code such as Deep-food.
No horizon
Assessment: SupportedSaid
Observation
“Here Roose has displayed the cardinal sin of looking only at regurgitation rather than generalization. History has told us over and over that systems can be good at the former and still face enormous challenges at the latter.”
Our reading: AI systems can be good at regurgitating known code while still facing enormous challenges generalizing to new problems.
No horizon Evidence as of
Limitation: Training similarity and robustness failures do not by themselves prove verbatim memorization.
Assessment: UnresolvedSaid
Observation
“it’s often easy to get a solution that is 80% correct, and extremely difficult to get the last 20%. Driverless cars have been like that for decades.”
Our reading: In AI it is often easy to reach 80% correctness and extremely difficult to get the last 20%, as with driverless cars for decades.
No horizon Evidence as of
Limitation: Distance automated is not equivalent to correctness across all driving situations.
Assessment: SupportedSaid
Observation
“A new study of the latest system, GPT 4.5 showed that by on one standardized measure over 1/3 of the outputs are still hallucinations.”
Our reading: On one standardized measure, over a third of GPT-4.5's outputs are hallucinations.
No horizon Evidence as of
Limitation: The exact percentage is supported by Computerworld's report of OpenAI's evaluation, not a readable primary numeric table.
Assessment: Can’t be checkedSaid
Forecast
“Errors in the code of automatic code writing systems will likely persist for decades, especially when they are forced to construct genuinely novel systems.”
Our reading: Errors in code from automatic code-writing systems will likely persist for decades, especially for genuinely novel systems.
No horizon
Qualifier: especially for genuinely novel systems
Assessment: UnresolvedSaid
Forecast
“I can’t imagine that noncoders are really going to get those systems to work well enough for prime time (except perhaps in very very narrow use cases), let alone know what do when the back end internet calls break.”
Our reading: Marcus says he can't imagine noncoders getting systems built by automated code tools to work well enough for prime time, except perhaps in very narrow use cases.
No deadline stated Evidence as of
Limitation: Both sources are company accounts without independent operational verification.
Assessment: Can’t be checkedSaid
Causal claim
“None of this may matter much in a lunch-recipe recommender, but errors in an automatic email writing system could cause serious harm.”
Our reading: Errors in AI-generated code for something like an automatic email writing system could cause serious harm.
No horizon
Qualifier: in an automatic email writing system
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“In the end, this essay, like a lot of Roose’s pieces, comes off as marketing for AI companies, even though it is framed as reporting. The Times owes its readers better.”
Our reading: Roose's essay, like a lot of his pieces, comes off as marketing for AI companies even though it is framed as reporting, and the Times owes its readers better.
No horizon
Scope: like a lot of Roose's pieces
Assessment: UnresolvedSaid
Observation
“They never let you know that a significant number of real world coders are deeply frustrated with automated coding tools like Cursor.AI (which is in part powered by Anthropic’s Claude):”
Our reading: A significant number of real-world coders are deeply frustrated with automated coding tools such as Cursor, which is partly powered by Anthropic's Claude.
No horizon Evidence as of
Limitation: Significant number and deeply frustrated remain qualitative.
Assessment: UnresolvedSaid
Forecast
“People are increasingly going to discourage American children from learning to code based on naive enthusiasm from people like Roose.”
Our reading: Hype about AI coding will increasingly lead people to discourage American children from learning to code.
No deadline stated Evidence as of
Limitation: Public attitudes toward coding are not the same as actual advice to children.
Assessment: Can’t be checkedSaid
Causal claim
“This will leave the US flat-foooted.”
Our reading: People increasingly discouraging American children from learning to code, out of naive enthusiasm about AI, will leave the US flat-footed.
No horizon
Assessment: SupportedSaid
Observation
“Already, of course, DeepSeek has shown that our once strong lead is diminishing.”
Our reading: DeepSeek has shown that the US's once-strong lead is shrinking.
No horizon Evidence as of
Limitation: The report was published after the claim, although its measurements cover the relevant earlier period.
Assessment: UnresolvedSaid
Forecast
“At least for another decade or two and probably more, big projects are always going to require software architects who know what they are doing.”
Our reading: For at least another 10 to 20 years, and probably longer, big software projects will require competent human software architects.
Not yet due Evidence as of
Limitation: The distinction between a substantial benchmark implementation and a big software project requires interpretation.
Assessment: UnresolvedSaid
Observation
“Nine years later and hundreds of startups later, my understanding is that not one radiologist has been replaced, and many parts of the world actually have a shortage of radiologists.”
Our reading: Marcus's understanding is that nine years and hundreds of startups after Hinton's 2016 prediction, not one radiologist has been replaced, and many parts of the world have a shortage of radiologists.
No horizon Evidence as of
Limitation: The outcome depends on reading replacement as job displacement rather than substitution on individual tasks.
Assessment: Can’t be checkedSaid
Forecast
“With people like Roose hyping automated coding, we may well find ourselves with too few coders.”
Our reading: Hype about automated coding may well leave society with too few coders.
No horizon
Qualifier: given hype about automated coding
Assessment: Can’t be checkedSaid
Causal claim
“By hyping the tech largely uncritically, the Rooses and Newtons of the world make the tech oligarchy stronger; society inevitably has to bear the costs.”
Our reading: Uncritical AI hype by journalists strengthens the tech oligarchy, and society bears the costs.
No horizon
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: Code for Roose's recipe-from-photo app already existed online, and the AI coding tools he used were presumably trained on it.