The mirage of visual understanding in current frontier models
We recorded 5 claims from this post. 2 were checked against evidence. 3 can’t be checked. 2 more are held. Read the post at its source.
Claims from this post (5)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
“AGI this stuff ain’t.”
What we tested: The frontier models described in the study are not AGI.
Assessment: SupportedSaid
“This study reinforces what Anh Totti Nguyen has been saying for a long time, in a series of underappreciated papers like Vision Language Models are Blind that I keep trying to draw attention to.”
What we tested: The new Stanford study reinforces what Anh Totti Nguyen has long been saying in a series of underappreciated papers, such as 'Vision Language Models are Blind'.
What happened: The study reports strong image-free benchmark performance and fabricated visual descriptions. Its published conclusion corroborates the broader warning that benchmark success can overstate visual understanding; the event date identifies the publication of this finding. (arXiv, )
Assessment: Can’t be checkedSaid
“Also, re the very active discussion on AI and jobs: although some white collar jobs (e.g., entry-level coder or market research assistant) may be in near-term jeopardy, many of those that require visual understanding (architect, cartographer, civil engineer, film editor, medical illustrator, urban planner, etc) probably aren’t vulnerable until entirely new techniques are developed.”
What we tested: Some white collar jobs, such as entry-level coder or market research assistant, may be in jeopardy from AI in the near term.
Assessment: Can’t be checkedSaid
“Also, re the very active discussion on AI and jobs: although some white collar jobs (e.g., entry-level coder or market research assistant) may be in near-term jeopardy, many of those that require visual understanding (architect, cartographer, civil engineer, film editor, medical illustrator, urban planner, etc) probably aren’t vulnerable until entirely new techniques are developed.”
What we tested: Many jobs requiring visual understanding (architect, cartographer, civil engineer, film editor, medical illustrator, urban planner) probably won't be vulnerable to AI until entirely new techniques are developed.
Assessment: Can’t be checkedSaid
“If your humanoid robot can’t understand the visual world, it’s just a demo, and not something you can trust.”
What we tested: A humanoid robot that can't understand the visual world is just a demo and not something you can trust.
Held from this post (2)
These records did not pass every check, so nothing about them is published. Held is not a finding.