Diving deep into OpenAI’s new study on LLM’s and bioweapons
We recorded 22 claims from this post. 7 were checked against evidence. 15 can’t be checked. 7 more are held. Read the post at its source.
Claims from this post (22)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“I am here to tell you that OpenAI’s research, analyzed carefully, shows no such thing, and that there is plenty of reason to be concerned.”
Our reading: OpenAI's bioweapons research, analyzed carefully, does not show that GPT-4 poses at most a slight risk of helping create biological threats, and there is plenty of reason to be concerned.
No horizon Evidence as of
Limitation: OpenAI itself acknowledged uncertainty; Marcus's criticism should not imply that the study concealed all these limitations.
Assessment: SupportedSaid
Observation
“Any fool can simply eyeball these data and see that in virtually every condition, people who have access both to the internet and LLMs (darker bars) are more accurate than people without (lighter bars).”
Our reading: In OpenAI's study, participants with both internet and LLM access were more accurate than those without in virtually every condition.
No horizon Evidence as of
Limitation: Higher observed means do not establish significance or practical danger.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Anyone reading this article, OpenAI included, ought to be pretty worried by these results.”
Our reading: Readers, including OpenAI, ought to be pretty worried by the accuracy results in the paper.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Unfortunately footnote [C] is extremely dubious.”
Our reading: Footnote C of OpenAI's paper, which says the results are not statistically significant, is extremely dubious.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“But here there is an obvious a priori test: does using an LLM make people more accurate? That’s what the whole paper is about. You don’t need a Bonferroni correction for that, and shouldn’t be using it.”
Our reading: Because the paper has an obvious a priori hypothesis, a Bonferroni correction is unnecessary and should not have been used.
No horizon
Qualifier: there is an obvious a priori hypothesis
Assessment: SupportedSaid
Observation
“In fact, if you simply collapsed all the measurements of accuracy, and did the single most obvious test here, a simple t-test, the results would (as Footnote C implies) be significant.”
Our reading: If all the accuracy measurements in OpenAI's study were collapsed and tested with a simple t-test, the results would be statistically significant, as Footnote C implies.
No horizon Evidence as of
Limitation: The result does not establish that this analysis was uniquely appropriate or that multiple-comparison adjustment was unnecessary.
Assessment: Can’t be checkedSaid
Conditional forecast
“Statistical significance is in part a function of sample size; with a bigger sample size, I doubt there would be any question about significance at all.”
Our reading: With a bigger sample size, the author doubts there would be any question that the effect of LLM access is statistically significant.
No horizon
Only if: if the sample size were bigger
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The threshold itself is arbitrary, and it is rarely good practice to use thresholds in statistical analysis, because doing so throw away information”
Our reading: The 8-of-10 threshold OpenAI used is arbitrary, and using thresholds in statistical analysis is rarely good practice because doing so throws away information.
No horizon
Assessment: SupportedSaid
Observation
“For one thing, in the expert population, the effect is in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation, and release).”
Our reading: In OpenAI's threshold analysis, among experts the effect was in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation and release).
No horizon Evidence as of
Limitation: These observed threshold proportions alone do not establish statistical significance or real-world biological threat capability.
Assessment: Can’t be checkedSaid
Observation
“What the results really seem to be showing is that (a) the nonexperts aren’t so much to be worried about, because few get over the hump, but (b) experts really do seem to be helped, in every aspect of development.”
Our reading: The study results seem to show that nonexperts are not much of a worry because few clear the threshold, but experts do seem to be helped in every aspect of bioweapon development.
No horizon
Qualifier: as interpretation of the study results
Assessment: Can’t be checkedSaid
Conditional forecast
“Crucially, if even one malicious expert gets over the hump and creates a deadly pathogen; that’s huge. The results make it look like that sort of thing might be possible.”
Our reading: If even one malicious expert uses an LLM to get over the threshold and create a deadly pathogen that would be hugely consequential, and the study results suggest that might be possible.
No horizon
Only if: if even one malicious expert succeeds
Assessment: Can’t be checkedSaid
Causal claim
“Indeed, in a domain like this, thresholds can be deeply misleading, because we don’t really know the relation between a threshold in the experiment and what would matter in the real world.”
Our reading: In the bioweapons domain, experimental thresholds can be deeply misleading because the relation between an experimental threshold and real-world significance is unknown.
No horizon
Qualifier: in the bioweapons domain
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“There is zero reason to get much comfort from this threshold analysis.”
Our reading: There is no reason to take much comfort from OpenAI's threshold analysis.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“Unfortunately, in the specific domain in which we are talking about, bioweapons risk, even a tiny effect could literally be world-changing, in a deeply negative way.”
Our reading: In the bioweapons domain, even a tiny effect of LLMs on capability could be world-changing in a deeply negative way.
No horizon
Qualifier: in the bioweapons domain
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Without question the finding of greater facility by experts on all five tasks deserves followup with a larger group of experts, and a single preregistered test with no Bonferroni correction.”
Our reading: The finding that experts did better on all five tasks should be followed up with a larger group of experts and a single preregistered test without Bonferroni correction.
No horizon
Qualifier: larger group of experts
Assessment: UnresolvedSaid
Observation
“It is shocking that neither OpenAI nor the media noted this (they also did not discuss in any detail one other paper that reached a somewhat similar conclusion, by Kevin Esvelt’s lab).”
Our reading: Neither OpenAI nor the media noted the fourfold difference in expert success on pathogen formulation, nor discussed in detail a similar paper from Kevin Esvelt's lab.
No horizon Evidence as of
Limitation: The supplied evidence items omit OpenAI's publication and do not provide a sufficiently comprehensive contemporary media record.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“but the media and the public need to realize that company white papers are not peer-reviewed articles. Particularly when it comes to safety-critical questions, peer-review is essential.”
Our reading: The media and public must recognize that company white papers are not peer-reviewed, and peer review is essential for safety-critical questions.
No horizon
Qualifier: safety-critical questions
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“It should go without saying that any risk-benefit analyses that with respect to the tradeoffs involved in AI needs to reflect studies that are not just methodologically sound but also interpreted accurately.”
Our reading: AI risk-benefit analyses need to be based on studies that are both methodologically sound and accurately interpreted.
No horizon
Assessment: UnresolvedSaid
Observation
“At present, we do not know.”
Our reading: It is currently unknown whether open-source models without guardrails are more useful to bad actors than commercial models with guardrails.
No horizon Evidence as of
Limitation: Earlier research is context because the claim concerns the knowledge state on February 4.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Good peer reviewers should push not only on sound statistical practice but also on important real-world questions like whether open-source models without guardrails (or with guardrails removed, which is easy to do in open-source models) might be more useful to bad actors than commercial models with guardrails.”
Our reading: Peer reviewers should press on statistical practice and on whether open-source models without guardrails are more useful to bad actors than guardrailed commercial models.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If an LLM equips even one team of lunatics with the ability to build, weaponize and distribute even one pathogen as deadly as covid-19, it will be really, really big deal.”
Our reading: If an LLM equips even one team of lunatics with the ability to build, weaponize and distribute even one pathogen as deadly as covid-19, that will be a really, really big deal.
No horizon
Only if: if an LLM equips even one team of lunatics to build, weaponize and distribute even one pathogen as deadly as covid-19
Assessment: ContradictedSaid
Observation
“As a journal reviewer, I would also note that they reported neither the p value they obtained, nor the test of statistical significance they applied; in a competent peer-reviewed journal, that’s a no-no.”
Our reading: OpenAI's paper reported neither the p value obtained nor the significance test applied, which would be unacceptable in a competent peer-reviewed journal.
No horizon Evidence as of
Limitation: The p-values are in a linked CSV rather than stated directly in footnote C.
Held from this post (7)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: In OpenAI's study the overall effect is sizeable: experts gained a .88 point lift on average across five measures on a 0-to-10 scale.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews read the claim differently