Elegant and powerful new result that seriously undermines large language models
We recorded 9 claims from this post. 5 were checked against evidence. 4 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (9)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
“The shocking (yet, in historical context, see below, unsurprising) answer is no:”
What we tested: LLMs trained on 'A is B' cannot automatically infer that 'B is A'.
What happened: Table 1 reports strong directional asymmetry, and section 2.1.2 finds no detectable advantage for the correct reversed name over random names. Section 1 explicitly excludes in-context learning from this failure. (arXiv; Berglund et al., )
Assessment: SupportedSaid
“On made-up facts, in a first experiment, the model was at zero percent correct, on celebrities, in a second experiment, performance was still dismal.”
What we tested: In the Berglund et al. experiments, the model scored zero percent on reversed made-up facts and performed dismally on reversed celebrity facts.
What happened: Table 1 gives GPT-3-175B synthetic reverse accuracies of 0.0% and 0.1% for the two directions. Celebrity reversal succeeds in 33% of cases, counting a correct answer in any of ten attempts, versus 79% for identifying parents. (arXiv; Berglund et al., )
Assessment: Can’t be checkedSaid
“None of what I raised then has really been adequately addressed in the intervening decades.”
What we tested: The generalization problems Marcus raised in 2001 about neural networks have not been adequately addressed since.
Assessment: SupportedSaid
“Forty one years later, neural networks (at least of the popular variety) still struggle with this. They still remain pointillistic masses of blurry memory, never as systematic as reasoning machines ought to be.”
What we tested: Popular neural networks still struggle with systematic relational reasoning and are not as systematic as reasoning machines should be.
What happened: The study evaluates GPT-3, ChatGPT and GPT-4 on multiplication, relational logic puzzles and dynamic programming. Performance deteriorates sharply as compositional complexity increases, supporting the claim that popular models struggle with systematic reasoning. (arXiv; Dziri et al., )
Assessment: Can’t be checkedSaid
“The will to believe in neural networks is frequently so strong that counterevidence is often dismissed or ignored, for much too long.”
What we tested: The will to believe in neural networks is frequently so strong that counterevidence is often dismissed or ignored for much too long.
Assessment: Can’t be checkedSaid
“If, after training on virtually the entire internet, you know Tom is Mary Lee‘s son, but can’t figure out without special prompting that Mary Lee therefore is Tom’s mother, you have no business running all the world’s software.”
What we tested: A system that, after training on virtually the entire internet, knows Tom is Mary Lee's son but can't figure out without special prompting that Mary Lee is therefore Tom's mother has no business running all the world's software.
Assessment: UnresolvedSaid
“This symmetry fail is mighty big, a mighty persistent error that has endured for decades.”
What we tested: The symmetry failure is a big, persistent error in neural networks that has endured for decades.
Assessment: UnresolvedSaid
“It’s just a matter before people start to realize that we need some genuinely new ideas in the field, either new mechanisms (perhaps neurosymbolic2), or different approaches altogether.”
What we tested: It is just a matter [of time] before people start to realize that the field needs some genuinely new ideas, either new mechanisms (perhaps neurosymbolic) or different approaches altogether.
Assessment: Can’t be checkedSaid
“Computers should be expected to make optimal use of information given, regardless of human limitations.”
What we tested: Computers should be expected to make optimal use of the information given, regardless of human limitations.
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: In many real-world problems the space of possible examples can never be fully covered, and heavily data-driven systems like LLMs that lack explicit variables and operations over variables are out of luck when they try to extrapolate beyond the space of their training examples; this was true in 2001 and is still true now.
- Attribution is uncertain
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: Bing's neurosymbolic supplementation to pure LLMs gives some benefit on these reversal problems where there are known facts, though it is not yet clear how general the benefit is, and Marcus would expect Bing to have trouble on the made-up facts used in Experiment 1.
- The two reviews read the claim differently