“Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.”
Our reading, the sentence that was checked:
As Davis has found, across the full set of their Technology Review questions, the difference between GPT-3's earlier version and InstructGPT is far more modest than one might have expected: fewer outright wacky answers and some old answers now correct, but some backsliding too.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus, paragraph 57.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-does-it-mean-when-an-ai-fails-cdba9a-a1
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-06-07. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- across the full set of our Technology Review questions (scope, stated by the speaker)
- far more modest than one might have expected (qualifier, stated by the speaker)
From the speaker
- across the full set of our Technology Review questions
- GPT-3’s earlier version to the current version (InstructGPT)
- far more modest than one might have expected
- fewer outright wacky answers
- some old answers now correct
- some backsliding, too
Our interpretation
- The plain reading is a limited overall improvement on the complete question set, with fewer plainly nonsensical responses, some corrected failures, and some newly incorrect answers.
- The surrounding discussion supplies Alexander’s six-question comparison as the reference for expectations. It does not supply a numerical threshold for modest improvement.
- The target is the comparison newly reported around the post date. I use June 1–7, 2022 for this early-June observation; the earlier GPT-3 answers serve as its historical baseline.
- Backsliding means an earlier correct response becoming incorrect on the same question. It does not by itself establish a decline in the model’s average capability.
Counts as supported if: An admissible comparison covering the complete question set shows a limited aggregate improvement, fewer plainly nonsensical answers, at least one corrected earlier failure, and at least one earlier success becoming incorrect.
Counts as contradicted if: An admissible complete-set comparison shows a clearly large aggregate improvement under consistent scoring, or establishes that one of the claimed directional changes did not occur.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The search located the directly relevant full-set report, but Marcus coauthored it. The supplied rules prohibit using his own writing to establish that he was right. Alexander’s independent test reports four corrected answers among six earlier failures, but that selection cannot reveal backsliding or establish the overall change across the complete set. The admissible evidence therefore does not settle the claim. This is an evidence limitation, not evidence that Marcus was wrong.
Limitations
- The source restriction excludes the report most directly associated with the claim.
- A small set selected from earlier failures cannot determine complete-set performance.
- Modest improvement and wacky answers lack numerical definitions.
Evidence (1 item)
Context only: Scott Alexander, Astral Codex Ten,
Of these six prompts that GPT-3 original failed, GPT-3 advanced gets four unambiguously right.
Why it matters here: Alexander reports corrections on a subset of earlier failures. This cannot establish the complete-set improvement or reveal regressions on earlier successes, so it neither settles nor contradicts Marcus’s full claim.
- Publisher
- Scott Alexander, Astral Codex Ten
- Kind of source
- Independent measurement
- Describes events of
- Public from
What would settle it
An independent audit of authenticated outputs from the original and early-June 2022 tests, covering the complete question set and applying consistent scoring, with matched counts of corrections, regressions, unchanged failures, and nonsensical responses.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.
- Where
- paragraph 57
- Published
- 2022-06-07
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-does-it-mean-when-an-ai-fails-cdba9a-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-does-it-mean-when-an-ai-fails-cdba9a-a1.