Learning Language is Harder Than You Think
We recorded 7 claims from this post. 5 were checked against evidence. 2 can’t be checked. 1 more is held. Read the post at its source.
Claims from this post (7)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Observation
“The thing is, current AI systems that (roughly speaking) do a lot of cutting and pasting without those grammar rules and trees aren’t really all that close to understanding human language.”
Our reading: Current AI systems that, roughly speaking, do a lot of cutting and pasting without grammar rules and trees aren't really all that close to understanding human language.
No horizon Evidence as of
Limitation: Benchmark achievement and general language comprehension are different claims.
Assessment: SupportedSaid
Observation
“it’s even less clear how it would work with words like no and not (a major sore spot for current neural networks).”
Our reading: Handling negation words like 'no' and 'not' is a major weak spot for current neural networks.
No horizon Evidence as of
Limitation: The supporting performance results concern RoBERTa and selected tasks; they do not establish a weakness in every contemporary neural network.
Assessment: UnresolvedSaid
Observation
“We don’t know how kids do it; we do know that what we are doing with machines currently isn’t really working.”
Our reading: Current machine approaches to language acquisition are not really working.
No horizon Evidence as of
Limitation: The selected tasks cover components of language competence.
Assessment: UnresolvedSaid
Observation
“All normal children manage to acquire language in this sense; no existing machine does.”
Our reading: No existing machine has acquired language in the sense of mapping sentences to meanings and meanings to sentences.
No horizon Evidence as of
Limitation: Formal meaning representations cover only selected aspects of meaning.
Assessment: UnresolvedSaid
Observation
“Indeed systems like GPT-3 are spectacular at memorization, but many onlookers recognize them for what they are, stochastic imitations of their input, with very little real coherence and very little comprehension.”
Our reading: Systems like GPT-3 are excellent at memorization but are stochastic imitations of their input with very little coherence or comprehension.
No horizon Evidence as of
Limitation: This is not a finding that GPT-3 understood language in a human-like way.
Assessment: Can’t be checkedSaid
Observation
“And it is worth noting that even if they did work, there would be, of course, no guarantee that they would work in a fashion that was similar to humans; AlphaGo plays a mean game of Go but that doesn’t mean it gives insight into how human players learn Go.)”
Our reading: Even if systems like GPT-3 did work, there would be no guarantee that they would work in a fashion similar to humans.
No horizon
Qualifier: even if they did work
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If we want to make progress, we need to start by recognizing the enormity of the problem.”
Our reading: To make progress on language acquisition (human or machine), researchers must first recognize how hard the problem is.
No horizon
Qualifier: if we want to make progress
Held from this post (1)
These records did not pass every check, so nothing about them is published. Held is not a finding.