Deep Research, Deep Bullshit, and the potential (model) collapse of science
We recorded 13 claims from this post. 12 were checked against evidence. 1 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (13)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“Sam Altman just released a fancier remake Deep Research, which, like Meta’s predecessor, can write science-sounding articles on demand, on any topic of your choice.”
Our reading: OpenAI's Deep Research is essentially a fancier remake of Meta's Galactica that can write science-sounding articles on any topic on demand.
No horizon Evidence as of
Limitation: The evidence supports a shared writing function, not technical equivalence or equal performance.
Assessment: UnresolvedSaid
Forecast
“We can be sure that DeepResearch will get heavily used.”
Our reading: Deep Research will be heavily used.
No deadline stated Evidence as of
Limitation: The disclosure does not quantify Deep Research's absolute scale or frequency of use.
Assessment: SupportedSaid
Observation
“The basic conjecture then — which has already been confirmed to some degree – was that LLMs would be used to write garbage content undermining the value of the internet itself.”
Our reading: His 2023 conjecture that LLMs would be used to write garbage content undermining the internet's value has already been confirmed to some degree.
No horizon Evidence as of
Limitation: The Wikipedia study was a preprint and relied on imperfect detectors.
Assessment: UnresolvedSaid
Forecast
“Deep Research, because it works faster and expands the reach of what can be automatically generated, is going to make that problem worse.”
Our reading: Deep Research is going to make the problem of the internet being polluted with AI-generated garbage content worse, because it works faster and expands the reach of what can be automatically generated.
No deadline stated Evidence as of
Limitation: Missing product attribution is not evidence that no contribution occurred.
Assessment: UnresolvedSaid
Forecast
“Scammers will use it, for example, to write “medical reports” that are half-baked (in order to sell ads on the webistes that report such “research”).”
Our reading: Scammers will use Deep Research to write half-baked 'medical reports' to sell ads on websites.
No deadline stated Evidence as of
Limitation: The search cannot establish that no qualifying case exists.
Assessment: Can’t be checkedSaid
Forecast
“But it is not just s scammers; the larger problem may be naive people who believe that the outputs are legit, or who use it as a shortcut for writing journal articles, missing the errors it produces.”
Our reading: The larger problem with Deep Research may be naive people who trust its outputs or use it to write journal articles, missing its errors.
No horizon
Assessment: SupportedSaid
Observation
“And that’s just not realistic given the current state of AI.”
Our reading: Current AI cannot reliably get synthesis of online information and multi-step research right.
No horizon Evidence as of
Limitation: The evidence directly concerns Deep Research, not every contemporary AI system.
Assessment: UnresolvedSaid
Forecast
“Unfortunately, virtually everything that Deep Research produces will pass a LGTM test ( “it ‘looks good to me’”); people will assume it’s legit, even when it is isn’t.”
Our reading: Virtually all Deep Research output will superficially look good and people will assume it is legitimate even when it is not.
No deadline stated Evidence as of
Limitation: The absence of a located study is an evidence gap, not proof that the predicted behavior does not occur.
Assessment: UnresolvedSaid
Forecast
“Few people will fact-check its output carefully, since it will all superficially look ok.”
Our reading: Few people will carefully fact-check Deep Research's output, since it will all superficially look ok.
No deadline stated Evidence as of
Limitation: Checking behavior may vary substantially with expertise and task stakes.
Assessment: UnresolvedSaid
Forecast
“The output of DeepResearch will inevitably feed further models; we will wind up with more of this:”
Our reading: Deep Research output will inevitably be used to train further models, contributing to model collapse.
No deadline stated Evidence as of
Limitation: Training datasets may not disclose product-level provenance.
Assessment: UnresolvedSaid
Forecast
“And it’s not just the models that will be infected, but the scientific literature itself.”
Our reading: The scientific literature itself, not just models, will be infected by Deep Research output.
No deadline stated Evidence as of
Limitation: This assessment uses the contextual meaning of infected as erroneous contamination.
Assessment: UnresolvedSaid
Forecast
“Because Deep Research lowers the effort, making it trivial to write legit-sounding articles, paper mill factories (there is already such a thing, designed to game citation counts, “an illegitimate, yet lucrative business“ that is growing) will massively ramp up production, choking the peer review process and saturating the market with hard-to-find, hard-to-rectify errors.”
Our reading: Because Deep Research lowers the effort and makes it trivial to write legit-sounding articles, paper mills will massively ramp up production, choking the peer review process and saturating the market with hard-to-find, hard-to-rectify errors.
No deadline stated Evidence as of
Limitation: The search cannot establish that no undisclosed paper-mill use exists.
Assessment: UnresolvedSaid
Forecast
“When — not if — Deep Research begins to flood the medical zone with shite, we are all in trouble.”
Our reading: Deep Research will flood the medical field with garbage content, causing serious trouble.
No deadline stated Evidence as of
Limitation: Benchmark errors establish capability limitations, not the prevalence of harmful medical content in circulation.
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: A Deep Research essay on the McKinley Tariff of 1890, which circulated and was initially praised, was not so great on inspection.
- The two reviews read the claim differently