“As a journal reviewer, I would also note that they reported neither the p value they obtained, nor the test of statistical significance they applied; in a competent peer-reviewed journal, that’s a no-no.”
Our reading, the sentence that was checked:
OpenAI's paper reported neither the p value obtained nor the significance test applied, which would be unacceptable in a competent peer-reviewed journal.
Assessment: Contradicted Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Diving deep into OpenAI’s new study on LLM’s and bioweapons, footnote 1.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-when-looked-at-carefully-openais-c9a306-a1
We recorded 22 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-04. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- reported neither the p value they obtained
- nor the test of statistical significance they applied
Our interpretation
- Assess the factual reporting allegation against the study, appendix and linked statistical summaries.
- The surrounding criticism concerns accuracy comparisons and footnote C.
- The judgment about acceptable journal practice is normative and is not separately graded.
Counts as supported if: Neither obtained p-values nor the applied significance tests are reported in the study or accompanying materials.
Counts as contradicted if: The materials report obtained p-values and identify the applied tests. If only one is reported and the other demonstrably absent, assess the two components separately.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Contradicted
The dated evidence goes against what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The study's methods appendix names the tests, and its linked accuracy summary reports numerical p-values. Both factual omissions alleged by Marcus are contradicted when the publication and accompanying materials are read together. This does not decide whether the analysis was appropriate or whether a journal should have accepted the paper.
Limitations
- The p-values are in a linked CSV rather than stated directly in footnote C.
- The CSV quotation preserves a complete data record because the source has no prose sentences.
- A separately timestamped release-day archive of the linked files was not authenticated.
Evidence (2 items)
Goes against the claim: OpenAI,
For completeness, we also repeated this procedure using the Mann-Whitney U-test, and observed the same results.
Why it matters here: The methods paragraph names one-sided t-tests and the Mann–Whitney check. The applied tests were therefore identified.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Goes against the claim: OpenAI,
,T-test p-value,0.3998,0.9584,0.0445,0.0683,0.3170,0.2399,0.2165,0.0721,0.0073,0.0160,0.1042,0.0027
Why it matters here: The linked summary explicitly reports obtained p-values, including 0.0027 for expert total accuracy. This is a published value, not an evaluator's reconstruction.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Diving deep into OpenAI’s new study on LLM’s and bioweapons
As a journal reviewer, I would also note that they reported neither the p value they obtained, nor the test of statistical significance they applied; in a competent peer-reviewed journal, that’s a no-no.
- Where
- footnote 1
- Published
- 2024-02-04
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-when-looked-at-carefully-openais-c9a306-a1: Current. Outcome: Contradicted. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-when-looked-at-carefully-openais-c9a306-a1.