Statistics versus Understanding: The Essence of What Ails Generative AI
We recorded 6 claims from this post. 3 were checked against evidence. 3 can’t be checked. 3 more are held. Read the post at its source.
Claims from this post (6)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“The problem with “Foundation Models” (a common term for Generative AI) is that they have never provided the firm, reliable foundation that their name implies.”
What we tested: Foundation models (generative AI) have never provided the firm, reliable foundation their name implies.
Assessment: UnresolvedSaid
“In our own brief experiments with GPT-3 (OpenAI has refused us proper scientific access for over a year) we found cases like the following, which reflects a complete failure to understand human biology.”
What we tested: In Marcus and Davis's experiments, GPT-3 produced outputs showing a complete failure to understand human biology.
Assessment: Can’t be checkedSaid
“GPT-3 has no idea what grape juice is, or what cranberry juice is, or what pouring, sniffing, smelling, or drinking are, or what it means to be dead.”
What we tested: GPT-3 has no idea what grape juice or cranberry juice is, what pouring, sniffing, smelling or drinking are, or what it means to be dead.
Assessment: Can’t be checkedSaid
“Generative AI systems have always tried to use statistics as a proxy for deeper understanding, and it’s never entirely worked.”
What we tested: Generative AI systems have always tried to use statistics as a proxy for deeper understanding, and it has never entirely worked.
Assessment: UnresolvedSaid
“That’s why statistically improbable requests like horses riding astronauts have always been challenging for generative image systems.”
What we tested: Because they rely on statistics instead of understanding, generative image systems have always struggled with statistically improbable requests such as horses riding astronauts.
Assessment: UnresolvedSaid
“Once again statistical frequency (most guitarists play righthanded) won out over linguistic understanding, to the consternation of Hendrix fans everywhere.”
What we tested: In Wyatt Walls's left-handed guitarist example, statistical frequency (most guitarists play right-handed) once again won out over linguistic understanding.
Held from this post (3)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: The critic's objection that the system had not been taught relevant information about left-handed drawing is nonsense, because a few seconds on Google Images turns up lots of people drawing with their left hand.