AI still lacks “common” sense, 70 years later
We recorded 15 claims from this post. 13 were checked against evidence. 2 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (15)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“Depressingly, despite all of the apparent progress in the last few years, much of what we raised in 2015 and re-emphasized in 2019 remains unsolved.”
Our reading: Much of the commonsense reasoning problem the authors raised in 2015 and 2019 remains unsolved in AI as of January 2025.
No horizon Evidence as of
Limitation: The result depends on interpreting 'unsolved' as lacking broad reliability.
Assessment: UnresolvedSaid
Observation
“Three approaches to commonsense reasoning have become popular in the last fifteen years; none has proved fully satisfactory.”
Our reading: None of the three popular approaches to commonsense reasoning (simulation, LLM emergence, video generation) has proved fully satisfactory.
No horizon Evidence as of
Limitation: Partial support for two approaches is insufficient to score the entire conjunction supported.
Assessment: UnresolvedSaid
Observation
“To date, though simulation has proved extremely valuable for all kinds of scientific and engineering applications, as well as video games and in creating CGI for films, it has proven much less effective for commonsense reasoning for AI.”
Our reading: Physical simulation has proven much less effective for AI commonsense reasoning than for scientific, engineering, and entertainment applications.
No horizon Evidence as of
Limitation: The evidence supplies no common effectiveness measure or broad comparison across the named domains.
Assessment: UnresolvedSaid
Causal claim
“Rosie the Robot is not doing our housework, and that is in part because the kind of physical reasoning a household robot would need to do can’t be done purely by running simulations.”
Our reading: General-purpose household robots like Rosie the Robot are not doing our housework, and this is in part because the physical reasoning such a robot would need cannot be done purely by running simulations.
No horizon Evidence as of
Limitation: The available experiments cover selected manipulation tasks.
Assessment: UnresolvedSaid
Causal claim
“Any one example can be fixed, but examples keep rolling in because no robust solution has been found.”
Our reading: LLM commonsense failure examples keep appearing because no robust solution has been found, even though individual examples can be fixed.
No horizon Evidence as of
Limitation: Spatial reasoning studies cover only part of commonsense reasoning.
Assessment: UnresolvedSaid
Observation
“Two and a half years later, more recent models no longer makes the same specific mistakes that GPT-3 did, but still lack a robust understanding of the real external world.”
Our reading: Recent LLMs no longer make GPT-3's specific mistakes but still lack a robust understanding of the real external world.
No horizon Evidence as of
Limitation: The packet contains no independent matched retests of the cited GPT-3 failure cases.
Assessment: UnresolvedSaid
Causal claim
“For example, OpenAI’s o1 is able to create and execute Python code that can do geometric calculation, but because it does not understand how objects are embedded in space and interact in space or, indeed, what spatial relationships actually mean, it often makes very basic mistakes.”
Our reading: OpenAI's o1 often makes very basic spatial mistakes because it does not understand how objects are embedded and interact in space.
No horizon Evidence as of
Limitation: The cited studies do not establish the behavior of released o1 during the target period.
Assessment: SupportedSaid
Observation
“Ten months later, Sora is finally available to the public, and it’s apparent that it too is struggling with physical reasoning.”
Our reading: Sora, now publicly available, struggles with physical reasoning.
No horizon Evidence as of
Limitation: The announcement does not quantify the physical-error rate or independently reproduce the examples in the post.
Assessment: Can’t be checkedSaid
Causal claim
“But fundamentally its videos are more like visual collages than the renderings of game engines, which is why they sometimes have a surreal or magic trick quality.”
Our reading: Sora's videos are fundamentally more like visual collages than game-engine renderings, which is why they sometimes have a surreal or magic-trick quality.
No horizon
Qualifier: more like (comparative, not categorical)
Assessment: SupportedSaid
Observation
“If general purpose domestic robots are a long way away, “AI agents” —AI systems that act in the world on user’s behalf—are already here to a limited degree.”
Our reading: AI agents, meaning AI systems that act in the world on users' behalf, are already here to a limited degree, even though general-purpose domestic robots are a long way away.
No horizon Evidence as of
Limitation: This establishes limited availability and functionality, not general reliability or widespread adoption.
Assessment: SupportedSaid
Forecast
“They will be a major focus of attention in the field in 2025.”
Our reading: AI agents will be a major focus of attention in the AI field in 2025.
Deadline passed Evidence as of
Limitation: 'Major focus' has no numeric threshold. The evidence shows agents were prominent but does not rank them against other themes in the field.
Assessment: SupportedSaid
Observation
“But what was most striking to us was not the unsurprising fact that these systems are not close to ready for prime time (an employee that bad would probably be immediately fired), but the examples from the list of what went wrong:”
Our reading: Current LLM agents (Claude, Gemini, GPT-4 etc.) are not close to ready for prime time on real-world workplace tasks.
No horizon Evidence as of
Limitation: 'Ready for prime time' has no explicit threshold and is a judgment.
Assessment: SupportedSaid
Observation
“Hoping that reliable commonsense will simply “emerge” from large databases and video has proven unrealistic, again and again.”
Our reading: The hope that reliable commonsense will emerge from large text and video datasets has repeatedly proven unrealistic.
No horizon Evidence as of
Limitation: Reliable commonsense remains an interpretive standard.
Assessment: SupportedSaid
Observation
“But one important component of physical and conceptual understanding is reasoning about entities and their properties, and current approaches have consistently fallen short on this.”
Our reading: Current AI approaches have consistently fallen short on reasoning about entities and their properties.
No horizon Evidence as of
Limitation: This outcome depends on reading consistently as recurring gaps, not failure on every task.
Assessment: Can’t be checkedSaid
Conditional forecast
“In our view, it is only once AI researchers grapple directly with the challenging problems inherent in commonsense reasoning about entities that interact in space and persist in time that the field of general-purpose AI will finally begin to mature.”
Our reading: General-purpose AI will begin to mature only once researchers directly tackle commonsense reasoning about entities interacting in space and persisting in time.
No horizon
Only if: only once researchers grapple directly with commonsense reasoning about entities in space and time
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: AI agents will require common sense about objects, people, and entities to work reliably in the real world.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: Sora's outputs are unreliable pure image streams, not segmentations of the world into objects with properties.
- The criteria are ambiguous. Our reading, not assessed: In recent Optimus demonstrations, all or most of the required commonsense was apparently supplied by human teleoperators.