About that OpenAI “breakthrough”
We recorded 10 claims from this post. 3 were checked against evidence. 7 can’t be checked. 2 more are held. Read the post at its source.
Claims from this post (10)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“Me? I am pretty skeptical; I was only half-kidding when I wrote this:”
What we tested: Marcus is pretty skeptical about OpenAI's reported Q* breakthrough.
Assessment: Can’t be checkedSaid
“Although there is some talk of OpenAI already trying to test the new technique out, it wouldn’t be realistic for them to have entirely changed the world in a matter of months.”
What we tested: It is not realistic that OpenAI's Q* technique has already entirely changed the world within a few months.
Assessment: Can’t be checkedSaid
“Part of the joke is that I don’t actually think Bing is powered already by the alleged breakthrough.”
What we tested: Marcus does not think Bing is already powered by OpenAI's alleged Q* breakthrough.
Assessment: Can’t be checkedSaid
“OpenAI could in fact have a breakthrough that fundamentally changes the world.”
What we tested: It is possible that OpenAI has a breakthrough that fundamentally changes the world.
Assessment: UnresolvedSaid
“But “breakthroughs” rarely turn to be general to live up to initial rosy expectations. Often advances work in some contexts, not otherwise.”
What we tested: Claimed AI breakthroughs rarely prove general enough to live up to initial expectations, often working only in some contexts.
Assessment: Can’t be checkedSaid
“We are probably still many “breakthroughs” away from true level 5 driverless cars.”
What we tested: True level 5 driverless cars are probably still many breakthroughs away.
Assessment: SupportedSaid
“AV’s still aren’t general enough that you can just plop down a car that was tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.”
What we tested: Autonomous vehicles still are not general enough that you can just plop down a car tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.
What happened: Waymo describes initial manual driving, subsequent specialist feedback, virtual evaluation of the new location, and refinements from local experience. This directly supports the need for preparation before expecting dependable operation in unfamiliar places. (Waymo, )
Assessment: SupportedSaid
“getting their algorithm to work for one object (which, cheesily, turned out to be a special Cube instrumented with sensors and LEDS, unlike a Rubik’s cube you’d buy in a store) in carefully controlled lab circumstances hardly guaranteed that the solution would work more broadly in the complex open-ended real world.”
What we tested: OpenAI's 2019 Rubik's cube robot result, achieved on a sensor-instrumented cube in controlled lab conditions, did not guarantee generalization to the open-ended real world.
What happened: The preceding discussion identifies arbitrary-object interaction as a long-term goal and a completely standard cube as unfinished work. Sections 3 and 7 document the laboratory apparatus and sensor-equipped Giiker experiments, rather than an established solution for open-ended manipulation. (OpenAI; original report hosted by Matthias Plappert, )
Assessment: Can’t be checkedSaid
“It’s all still an empirical question; I for one certainly don’t yet know enough about the details of Q* to judge with certainty.”
What we tested: Whether Q* pans out is still an empirical question, and Marcus himself does not yet know enough about the details of Q* to judge it with certainty.
Assessment: Can’t be checkedSaid
“At least so far, Q* ain’t one.”
What we tested: At least so far, OpenAI's Q* is not among the problems Marcus is worried about.
Held from this post (2)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: OpenAI's touted Rubik's cube robotics breakthrough went nowhere, and OpenAI quietly closed its robotics division a year or two later.
- The two reviews reached different assessments. Our reading, not assessed: Autonomous cars are still not reliable enough for prime time; no breakthrough has gotten them over the threshold.