LLMs don’t do formal reasoning - and that is a HUGE problem
We recorded 9 claims from this post. 4 were checked against evidence. 5 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (9)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Everyone actively working with AI should read it”
Our reading: Everyone actively working with AI should read the new Apple researchers' article on LLMs, or at least senior author Mehrdad Farajtabar's X thread summarizing it.
No horizon
Scope: everyone actively working with AI
Assessment: SupportedSaid
Observation
“Performance is ok on small problems, but quickly tails off.”
Our reading: o1 does acceptably on small planning problems, but its performance drops quickly as problems get larger.
No horizon Evidence as of
Limitation: The model tested was o1-preview.
Assessment: SupportedSaid
Observation
“Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.”
Our reading: Both older and newer LLMs have repeatedly been observed to get worse as multiplication problems get larger.
No horizon Evidence as of
Limitation: The Deng results are a researcher's social-media post plus a news report, not a peer-reviewed paper.
Assessment: SupportedSaid
Observation
“Failure to follow the rules of chess is another continuing failure of formal reasoning:”
Our reading: LLMs still fail to follow the rules of chess, which is an ongoing failure of formal reasoning.
No horizon Evidence as of
Limitation: The evidence is one puzzle benchmark with a single prompt format. Other formats or full games may give different illegal-move rates.
Assessment: Can’t be checkedSaid
Forecast
“(We are, however, unlikely ever to get systematic data on this, since the company isn’t transparent about what it has done or what the results are.)”
Our reading: The public is unlikely ever to get systematic data on robotaxi reasoning failures, because Tesla is not transparent.
No horizon
Qualifier: because Tesla is not transparent
Assessment: Can’t be checkedSaid
Forecast
“Elon Musk’s putative robotaxis are likely to suffer from a similar affliction: they may well work safely for the most common situations, but are also likely struggle to reason abstractly enough in some circumstances.”
Our reading: Tesla's planned robotaxis will likely fail to reason abstractly enough in some circumstances, although they may work safely in the most common situations.
No horizon
Qualifier: in some circumstances
Assessment: MixedSaid
Observation
“I strongly believe the current results are robust.”
Our reading: Marcus strongly believes that the current results on LLMs' reasoning failures, including the Apple study and the other work he cites, are robust.
No horizon Evidence as of
Limitation: 'I strongly believe' is a statement of belief; only the robustness of the results is graded.
Assessment: Can’t be checkedSaid
Causal claim
“What I argued in 2001, in The Algebraic Mind, still holds: symbol manipulation, in which some knowledge is represented truly abstractly in terms of variables and operations over those variables, much as we see in algebra and traditional computer programming, must be part of the mix.”
Our reading: As Marcus argued in The Algebraic Mind in 2001, symbol manipulation, in which some knowledge is represented truly abstractly in terms of variables and operations over them as in algebra and traditional programming, must be part of the mix.
No horizon Evidence as of
Limitation: The packet cites no evidence about whether current AI systems use symbolic components, or how well systems without them perform.
Assessment: Can’t be checkedSaid
Causal claim
“Neurosymbolic AI — combining such machinery with neural networks – is likely a necessary condition for going forward.”
Our reading: Neurosymbolic AI, which combines symbol manipulation with neural networks, is likely a necessary condition for going forward.
No horizon
Qualifier: likely
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: o1's accuracy also drops as multiplication problems get larger.
- Attribution is uncertain
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: LLM performance often falls apart as problems are made bigger, which is another manifestation of their lack of sufficiently abstract, formal reasoning.