How come GPT can seem so brilliant one minute and so breathtakingly dumb the next?
We recorded 9 claims from this post. 6 were checked against evidence. 3 can’t be checked. 7 more are held. Read the post at its source.
Claims from this post (9)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
“Chance is definitely part of what’s going on. But it’s not quite the right way to understand the juxtaposition of brilliance and stupidity that we see within GPT.”
What we tested: Chance plays a part, but it does not properly explain why GPT mixes brilliance and stupidity.
What happened: Human detection accuracy was about 86% for the degraded control and 52% for GPT-3, with outputs selected programmatically. This supports systematic competence beyond random text, although the control changes several factors together. (Brown et al., arXiv, )
Assessment: SupportedSaid
“The impressive thing about GPT is that it spits out hundreds of perfectly fluent, often plausible prose at a regular clip, with no human filtering required.”
What we tested: GPT reliably produces large amounts of fluent, often plausible prose without needing humans to filter it.
What happened: The first four conversations contain repeated readable, generally grammatical responses across recipes, social media, mood, and AI. The transcript exceeds hundreds of words and is explicitly described as unedited; this is evidence of output quality, not validation of its advice. (Jeff Turner, )
Assessment: UnresolvedSaid
“Almost everything it says is fluent and at least vaguely plausible.”
What we tested: Almost all of GPT's output is fluent and at least vaguely plausible, not random or word salad.
Assessment: Can’t be checkedSaid
“GPT-3 has no idea how the world works (and on this Bender and I would agree);”
What we tested: GPT-3 does not understand how the world works.
Assessment: Can’t be checkedSaid
“But it is mimic that knows not whereof it speaks.”
What we tested: GPT-3 is a mimic that does not understand what it is saying.
Assessment: UnresolvedSaid
“GPT”s heavy use of a technique called embeddings makes it really good at substituting synonyms and more broadly related phrases, but the same tendency towards substitution often lead it astray.”
What we tested: GPT's reliance on embeddings makes it good at substituting related phrases, and the same tendency often causes it to make mistakes.
Assessment: SupportedSaid
“GPT doesn’t talk randomly, because it’s pastiching things actual people said.”
What we tested: GPT's output is not random, because it recombines things real people have said.
What happened: The study documents training on web text, books and Wikipedia, alongside controlled assessments of generated articles. Together these support learned language patterns as the basis of structured output, without establishing literal cut-and-paste generation. (OpenAI researchers / arXiv, )
Assessment: UnresolvedSaid
“When GPT sounds plausible, it is because every paraphrased bit that it pastes together is grounded in something that actual humans said, and there is often some vague (but often irrelevant) relationship between..”
What we tested: GPT sounds plausible because each piece it combines comes from something humans actually said, and the pieces are often loosely related.
Assessment: Can’t be checkedSaid
“It was just named a Best Tech Article of 2022, and I believe that is still incredibly relevant, even with all the advances of the last 8 months.”
What we tested: Marcus believes his essay 'Deep Learning is Hitting a Wall' is still incredibly relevant, even with all the advances of the last 8 months.
Held from this post (7)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: GPT never fully masters abstract, universally quantified relationships.
- The two reviews reached different assessments. Our reading, not assessed: GPT does not know which entities properly go with which properties.
- The two reviews reached different assessments. Our reading, not assessed: GPT-3 produces output like the churro-surgery text because it is the king of pastiche, imitating the style of human-written text.
- The record does not meet the current rules. Our reading, not assessed: Over the course of training, GPT sometimes loses track of the precise relations ('bindings') between entities and their properties.
- The record does not meet the current rules. Our reading, not assessed: Current neural networks still struggle to learn universal abstract knowledge, as recent arithmetic studies show.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: For now, a human is still needed to judge which of GPT's plausible fragments actually belong together.