Three ideas from linguistics that everyone in AI should know
We recorded 26 claims from this post. 17 were checked against evidence. 9 can’t be checked. 12 more are held. Read the post at its source.
Claims from this post (26)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“Since then, the results on benchmarks have gotten better, but there’s still something missing.”
Our reading: Since 2019 large language models have done better on benchmarks, but something is still missing from them.
No horizon Evidence as of
Limitation: TruthfulQA deliberately tests susceptibility to misconceptions; its percentages are not general error rates for all language use.
Assessment: UnresolvedSaid
Observation
“Language is about a connection between words (or sentence) and the world; the sequences of words that large language models utter lack connection to the external world.”
Our reading: The word sequences that large language models produce have no connection to the external world, so they lack reference.
No horizon Evidence as of
Limitation: Source attribution does not by itself establish situated reference or factual correctness [e3].
Assessment: ContradictedSaid
Observation
“(LLM’s like GPT produce well-formed prose but do not produce interpretable representations of utterances that reflect structured relationships between the parts of those sentences.)”
Our reading: LLMs like GPT write well-formed prose but do not produce interpretable representations of the structured relationships between the parts of sentences.
No horizon Evidence as of
Limitation: The experiments use task-specific examples and bounded semantic-parsing domains.
Assessment: UnresolvedSaid
Causal claim
“In our view, inadequate attention to these three factors has serious consequences, including:”
Our reading: Neglecting reference, cognitive models and compositionality causes serious problems in large language models.
No horizon Evidence as of
Limitation: The search is not an exhaustive systematic review.
Assessment: UnresolvedSaid
Causal claim
“(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;”
Our reading: Neglecting reference, cognitive models and compositionality leads LLMs to lose coherence over time and drift into empty language with no clear connection to reality.
No horizon Evidence as of
Limitation: TattleTale has limited empirical evaluation.
Assessment: UnresolvedSaid
Causal claim
“(b) the difficulty of large language models in distinguishing truth from falsehoods;”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to tell truth from falsehood.
No horizon Evidence as of
Limitation: TruthfulQA's questions target misconceptions; its results do not establish an overall factual-error rate [e8].
Assessment: UnresolvedSaid
Causal claim
“(c) the struggle in these models to avoid perpetuating bias and toxic speech.”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to avoid repeating bias and toxic speech.
No horizon Evidence as of
Limitation: The search is not proof that no relevant experiment exists.
Assessment: Can’t be checkedSaid
Causal claim
“But if all a system is doing is predicting the next word in what someone says, its understanding is inevitably superficial.”
Our reading: A system that only predicts the next word will inevitably have only superficial understanding.
No horizon
Qualifier: if the system only predicts the next word
Assessment: Can’t be checkedSaid
Observation
“In our view, large language models are basically sidestepping this entire set of issues, and to their peril.”
Our reading: Large language models sidestep the problems of reference and shared understanding, and this puts them at risk.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Moving beyond prediction into reference, particularly in a system that is inherently uninterpretable, is an immense leap that (a) cannot be taken for granted and (b) should be a focus of research.”
Our reading: Moving beyond prediction into reference, especially in a system that is inherently uninterpretable, is an immense leap that cannot be taken for granted and should be a focus of research.
No horizon
Qualifier: particularly in a system that is inherently uninterpretable
Assessment: ContradictedSaid
Causal claim
“Current systems simply do not postulate such cognitive models—and so, they technically can’t prove themselves wrong, because they do not even have a representation of what being “right” would look like.”
Our reading: Current systems do not postulate cognitive models, so they technically cannot prove themselves wrong, because they have no representation of what being right would look like.
No horizon Evidence as of
Limitation: The result depends on treating independently assessable self-critique as demonstrating an error.
Assessment: ContradictedSaid
Causal claim
“The downstream consequence of all of this is that these models have no grasp whatsoever on truth.”
Our reading: As a downstream consequence of their lack of reference and cognitive models, large language models have no grasp whatsoever on truth.
No horizon Evidence as of
Limitation: The outcome depends on a functional reading of grasp and inclusion of fine-tuned language models.
Assessment: UnresolvedSaid
Conditional forecast
“Unless the field can find a way to tightly integrate large language models with cognitive models and reference, we expect that the field will eventually hit a dead end, particularly in the understanding long discourses, and series of events unfolding over time, and that the problem of misinformation will remain unsolved.”
Our reading: Unless LLMs are tightly integrated with cognitive models and reference, the field will eventually hit a dead end, especially in understanding long discourse and events over time, and misinformation will remain unsolved.
No deadline stated Evidence as of
Only if: unless LLMs are tightly integrated with cognitive models and reference
Assessment: SupportedSaid
Observation
“noncanonical reorderings such as these remain problematic for current AI approaches.)”
Our reading: Current AI approaches still struggle with noncanonical word orders such as 'the mat is on the cat'.
No horizon Evidence as of
Limitation: The evidence concerns tested vision-language models, not every contemporary AI approach [e1].
Assessment: Can’t be checkedSaid
Observation
“Rather, it means identify the second of the balls that are blue, regardless of which position; systems like GPT-3 likely have trouble with this sort of thing, as Ernie Davis just reported to us in an email.”
Our reading: Systems like GPT-3 likely have trouble with compositional phrases such as 'the second blue ball'.
No horizon
Assessment: UnresolvedSaid
Observation
“Compositionality has long been a central concept in linguistics and philosophy (even Aristotle acknowledged it), yet so-called foundation models (large language models, GPT-3, BERT, deep learning approaches) sidestep it.”
Our reading: Foundation models such as GPT-3 and BERT sidestep compositionality.
No horizon Evidence as of
Limitation: The studies use different tasks and assistance levels, preventing a direct comparison [e1, e2].
Assessment: UnresolvedSaid
Observation
“But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.”
Our reading: Current LLMs lack the human ability to choose whether to interpret language compositionally.
No horizon Evidence as of
Limitation: The available tests use different tasks and model configurations.
Assessment: Can’t be checkedSaid
Rhetoric
“Yet even more wild is the widespread belief that language models lacking the major foundational components of human language (e.g., bounded compositionality mixed with unbounded generativity) will have any hope of meaningfully interfacing with audio/visual models in a way that accurately reflects human conceptual structure.”
Our reading: The authors call it even more wild than the expected unification of modalities in foundation models that people widely believe language models lacking major foundational components of human language (e.g., bounded compositionality mixed with unbounded generativity) have any hope of meaningfully interfacing with audio/visual models in a way that accurately reflects human conceptual structure.
No horizon
Scope: language models lacking the major foundational components of human language
Assessment: UnresolvedSaid
Observation
“In such cases, DALL-E Mini typically generates images of bowls of cucumbers, and only cucumbers, leaving out the tomatoes altogether (where a human might reasonably expect there to be at least two, given the plural).”
Our reading: Given comparative prompts such as 'more cucumbers than tomatoes', DALL-E Mini usually leaves out the tomatoes entirely.
No horizon Evidence as of
Limitation: The evidence establishes neither a majority omission rate nor a rate sufficient to contradict the claim.
Assessment: UnresolvedSaid
Observation
“here, DALL-E mini struggles to accurately represent the implicit action for the woman, capturing the apple but not reliably capturing the eating, often sticks an apple in the man’s mouth rather than the sandwich, and in most cases misses the implication that the man and woman are supposed to be eating different things.”
Our reading: In the authors' experiments with the sentence 'The man is eating a sandwich and the woman an apple', DALL-E Mini struggles to represent the woman's implied action: it captures the apple but not reliably the eating, often puts an apple in the man's mouth instead of the sandwich, and in most cases misses that the two are eating different things.
No horizon Evidence as of
Limitation: A replication could establish reproducibility without proving what happened in the authors' experiment.
Assessment: UnresolvedSaid
Observation
“Similarly, consider pronouns like “himself”: DALL-E does not do well with sentences like “The man is painting a picture of himself”.”
Our reading: DALL-E does poorly on sentences with reflexive pronouns such as 'The man is painting a picture of himself'.
No horizon Evidence as of
Limitation: Model identity and the standard for poor performance remain unspecified.
Assessment: Can’t be checkedSaid
Causal claim
“A good language understanding (or language production) system almost certainly needs to have some large-scale learning as a vital component.”
Our reading: A good language understanding or language production system almost certainly needs some large-scale learning as a vital component.
No horizon Evidence as of
Limitation: The evidence does not establish that the proposed necessity is true or false.
Assessment: Can’t be checkedSaid
Forecast
“Ultimately, the right approach to large-scale learning may not rest on machinery that is finely geared to predicting sequences of words, but rather on developing a new approach to machine learning that is fundamentally geared towards learning how hierarchically structured sets of words map onto meanings that are used in the service of updating cognitive models.”
Our reading: The right approach to large-scale learning may need a new kind of machine learning that maps hierarchically structured words onto meanings used to update cognitive models, rather than word-sequence prediction.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“For that, we may yet need another revolution.”
Our reading: Getting to real language understanding may require another revolution in AI.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“Considerable efforts are being made towards making “cross-modal” models like Gato that take in visual input as well strings of words; we think this is well worth investigating but unlikely in itself to suffice.”
Our reading: Cross-modal models like Gato are worth investigating but are unlikely by themselves to solve the lack of reference.
No horizon
Qualifier: in itself
Assessment: UnresolvedSaid
Causal claim
“The fact that its word sequence predictions are often so good tempts many into ascribing meaning, but this is more a reflection of the tremendous size of the input corpus than of a system that genuinely maps sentence onto reference and cognitive models.”
Our reading: GPT-3's good word predictions come more from its huge training corpus than from any real mapping of sentences onto reference and cognitive models.
No horizon Evidence as of
Limitation: Results from smaller transformers, BART, and T5 are context only.
Held from this post (12)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The criteria are ambiguous. Our reading, not assessed: DALL-E Mini does not seem to grasp ellipsis, comparatives or pronouns, which children learn easily.
- The record does not meet the current rules. Our reading, not assessed: Systems like DALL-E clearly struggle with compositionality.
- The criteria are ambiguous. Our reading, not assessed: The authors see no reason to think that current large language models encode stable representations of the world at all, and say that all these models are literally doing is predicting the next words in sentences.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: Large language models do not produce cognitive models of the world, at least none that anyone has been able to use reliably.
- The criteria are ambiguous. Our reading, not assessed: The authors suggest that all the attention placed on predicting sequences of words has come at the expense of developing a theory of how such processes might culminate in cognitive models of the world.
- The criteria are ambiguous. Our reading, not assessed: Asked for a blue cube on a red cube, DALL-E puts the red cube on top about as often as not, losing the relation between the parts.
- The two reviews reached different assessments. Our reading, not assessed: Because GPT-3's words do not refer to the world or an updatable cognitive model, it tends to fabricate misinformation, and this is hard to fix.
- The criteria are ambiguous. Our reading, not assessed: The large language model approach ignores the linguistics and cognitive psychology research on models of meaning.
- The criteria are ambiguous. Our reading, not assessed: Current AI systems do not really build cognitive models of the kind humans use to understand stories and the world.
- The criteria are ambiguous. Our reading, not assessed: The internal states visited by current neural network systems that cluster similar sentences, as in Piantadosi's argument, are pretty but lack reference and lack ties to dynamically updated cognitive models.
- The two reviews read the claim differently