“However, just in terms of percentage right vs. wrong, the improvement was slight; It now got 50% right, 40% wrong, and 10% borderline (marginally better than the 45% right before).”
Our reading, the sentence that was checked:
From 2020 to June 2022, GPT-3 improved only slightly on the authors' commonsense examples, going from 45% to 50% right.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How Not to Test GPT-3, paragraph 37.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-not-to-test-gpt-3-c48e17-a2
We recorded 12 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-02-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- in June 2022 we reran all the examples that we had run before
- 50% right, 40% wrong, and 10% borderline
- marginally better than the 45% right before
- the improvement was slight
Our interpretation
- The claim reports the authors' own informal experiment: GPT-3 (2020 version) at 45% right, and text-davinci-002 in June 2022 at 50% right / 40% wrong / 10% borderline, on their commonsense examples.
- Verification needs an independent record of the prompts, outputs and scoring. The authors' own report cannot serve as evidence that they were right.
Counts as supported if: An independent rerun or audit of the authors' example set confirming roughly 45% right in 2020 and 50% right in June 2022.
Counts as contradicted if: An independent audit of the same examples and outputs showing materially different success rates, or showing that the two runs did not cover the same examples.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The 45% and 50% figures come from an informal experiment by Marcus and Davis. The only evidence item (e1) is their own NYU report from June 2022. Under the contract, the authors' own report counts only as context and cannot confirm their tallies. Its table lists 71 of 157 tests (45%) as successes in 2020 and 40 of 80 tests (50%) in 2022. Because the test counts differ, it is unclear whether the two percentages cover the same examples. However, e1 does not show how the example sets or the scoring differed. The packet's search found no independent rerun or audit. On this evidence, the claim can be neither confirmed nor refuted.
Limitations
- No independent rerun or audit of the authors' example set was found. That does not prove none exists.
- e1 notes that the system is not deterministic, so a rerun could give different results.
- The packet notes that the 2020 model versions are no longer available, which limits replication.
Evidence (1 item)
Context only: Ernest Davis, NYU (co-author of the post),
There are some examples that GPT-3 got wrong in 2020 and got right in 2022, but there are also examples of the reverse.
Why it matters here: The underlying report is by the post's own authors. Its table lists 157 tests at 45% in 2020 and 80 tests at 50% in 2022, so the sets differ in size. It cannot be used as independent confirmation.
- Publisher
- Ernest Davis, NYU (co-author of the post)
- Kind of source
- Secondary summary
- Describes events of
- Public from
What would settle it
Someone other than the authors would need to rescore the full 2020 and 2022 example sets and outputs, or replicate the tests on the same examples, and confirm or reject roughly 45% right in 2020 and 50% right in June 2022.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How Not to Test GPT-3
However, just in terms of percentage right vs. wrong, the improvement was slight; It now got 50% right, 40% wrong, and 10% borderline (marginally better than the 45% right before).
- Where
- paragraph 37
- Published
- 2023-02-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-not-to-test-gpt-3-c48e17-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.