What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
We recorded 14 claims from this post. 7 were checked against evidence. 7 can’t be checked. 2 more are held. Read the post at its source.
Claims from this post (14)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“I think all that’s scandalous.”
Our reading: Marcus thinks it is scandalous that OpenAI never followed through on its promise of GPT-3 access to him and Davis and never replied to his request for DALL-E 2 access, and that Google Brain never replied to his request for Imagen access.
No horizon
Qualifier: I think
Assessment: UnresolvedSaid
Observation
“The core claims that Ernie and I made in Technology Review still all, so far as I can tell, hold true, nearly two years and many even larger models later. Yes, some of the specific examples we pointed out have been remedied, but the general problems have not.”
Our reading: So far as Marcus can tell, all the core claims he and Davis made in Technology Review still hold true nearly two years and many even larger models later; some of their specific examples have been remedied, but the general problems have not.
No horizon Evidence as of
Limitation: This is a gap in the retrieved evidence, not evidence that the categories had been solved.
Assessment: Can’t be checkedSaid
Observation
“But I see no reason whatsoever to think that the underlying problem — a lack of cognitive models of the world —have been remedied.”
Our reading: Marcus sees no reason whatsoever to think that the underlying problem of these models, a lack of cognitive models of the world, has been remedied.
No horizon
Qualifier: I see no reason whatsoever to think
Assessment: UnresolvedSaid
Causal claim
“But they still don’t convey genuine comprehension, and so they are still very easy for Ernie and me (or anyone else who cares to try) to break.”
Our reading: Newer LLMs still lack genuine comprehension and are therefore still very easy for anyone to break.
No horizon Evidence as of
Limitation: Benchmark scores and qualitative examples do not establish that anyone could easily elicit failures across all relevant models [e1, e2].
Assessment: UnresolvedSaid
Causal claim
“The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.”
Our reading: The newer models' improvements, such as they are, come primarily from their larger and larger sets of data about how humans use word sequences, which are certainly helpful for pattern-matching machines.
No horizon Evidence as of
Limitation: An optimal scaling prescription is not a decomposition of historical progress.
Assessment: Can’t be checkedSaid
Observation
“And yes, there has been improvement eg from GPT-2 to GPT-3—but zero qualitative change.”
Our reading: Improvement from GPT-2 to GPT-3 has been quantitative only, with zero qualitative change.
No horizon
Assessment: Can’t be checkedSaid
Observation
“Either way, though, there is still no real physical, social, or psychological understanding, just richer parroting of larger corpora (and some reinforcement learning to keep the system more on check).”
Our reading: Current LLMs have no real physical, social, or psychological understanding; they are richer parroting of larger corpora.
No horizon
Assessment: UnresolvedSaid
Conditional forecast
“I am willing to bet SlateStarCodex now (terms to be negotiated) that if OpenAI gives us unrestricted access to GPT-4, whenever that is released, and assuming that is basically the same architecture but with more data, that within a day of playing around with it, Ernie and I will still be able find lots of examples of failures in physical reasoning, temporal reasoning, causal reasoning, and so forth.”
Our reading: If given unrestricted access to GPT-4 and it is basically the same architecture with more data, Marcus and Davis will find many failures in physical, temporal, and causal reasoning within a day.
No deadline stated Evidence as of
Only if: OpenAI gives unrestricted access to GPT-4
Assessment: Can’t be checkedSaid
Conditional forecast
“Maybe we will find fewer absolute howlers per attempt than we did with GPT-3, but it still won’t be so proficient as to have developed a genuine causal model of the world.”
Our reading: If GPT-4 is basically the same architecture as GPT-3 but with more data, it will not be proficient enough to have developed a genuine causal model of the world, though Marcus and Davis may find fewer absolute howlers per attempt than with GPT-3.
No horizon
Only if: assuming GPT-4 is basically the same architecture but with more data
Assessment: Can’t be checkedSaid
Causal claim
“The core problem that follows from that has always been that these systems aren’t stable or trustworthy; a crumbling house follows from a rotten foundation.”
Our reading: Because LLMs lack causal world models, they are not stable or trustworthy.
No horizon
Assessment: SupportedSaid
Forecast
“I would be very surprised if that fundamental lack of reliability did not also hold true for GPT-4. Maybe it will take three paragraphs in a prompt rather than one to break it, but a big Transformer alone does not a a theory of the world create.”
Our reading: GPT-4 will very likely still have a fundamental lack of reliability, though it may take longer prompts to break.
Settled before its deadline Evidence as of
Limitation: Support applies to recurring factual and reasoning unreliability, not a claim that GPT-4 has no useful capabilities.
Assessment: UnresolvedSaid
Observation
“Large language models are (unless hybridized with symbolic systems, a la Jurassic X) are, on their own, inherently weak at understanding the physical and psychological world.”
Our reading: LLMs on their own, without symbolic hybridization, are inherently weak at understanding the physical and psychological world.
No horizon Evidence as of
Limitation: TRIP tests a narrower model population than the frontier generative systems discussed in the post.
Assessment: Can’t be checkedSaid
Observation
“Scott’s more marked success on six questions may have been a fluke.”
Our reading: Scott Alexander's finding that InstructGPT got four of six previously failed prompts right may have been a fluke.
No horizon
Assessment: UnresolvedSaid
Observation
“Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.”
Our reading: As Davis has found, across the full set of their Technology Review questions, the difference between GPT-3's earlier version and InstructGPT is far more modest than one might have expected: fewer outright wacky answers and some old answers now correct, but some backsliding too.
No horizon Evidence as of
Limitation: The source restriction excludes the report most directly associated with the claim.
Held from this post (2)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: Marcus says that he and Davis have been able to analyze these AI systems at all only because of the kindness of strangers, in an era when, he says, big corporations pretend to do public science but refuse to share their work.