The truth about Google’s amazing new demo
We recorded 6 claims from this post. 3 were checked against evidence. 3 can’t be checked. 2 more are held. Read the post at its source.
Claims from this post (6)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“The video is almost never the true story.”
What we tested: Tech demo videos almost never tell the true story.
Assessment: SupportedSaid
“The whole project was quietly dropped a year or two later.”
What we tested: OpenAI quietly dropped its Rubik's cube robotics project a year or two after the demo.
What happened: The next sentence dates the move to the preceding October, and the reproduced company statement says further robotics research stopped. This establishes October 2020 rather than treating the July 2021 reporting date as the shutdown date. (TechCrunch, )
Assessment: UnresolvedSaid
“Regular readers will recall how that turned out: the Rubik’s cube was instrumented with sensors, and not straight off the shelf, and what worked for the Rubik’s cube in the demo video didn’t turn out have a whole lot of application in the real world.”
What we tested: OpenAI's Rubik's cube robot demo relied on a sensor-instrumented cube and had little real-world application.
Assessment: UnresolvedSaid
“What we are going to see, in the fullness of time, I promise you, is that Gemini is more or less in the same ball park as GPT-4, handy for a bunch of things, but untethered in reality, still with dicey, unpredictable reasoning, and a very limited understanding of the world.”
What we tested: Gemini will be shown to still have dicey, unpredictable reasoning and a very limited understanding of the world.
Assessment: Can’t be checkedSaid
“Don’t let the PR fool you.”
What we tested: People should not be fooled by Google's PR about Gemini.
Assessment: Can’t be checkedSaid
“I will give the last word to Filip Piekniewski, who did the right test: try something that isn’t in the vast corpus, that requires real reasoning, and see how it runs…”
What we tested: Filip Piekniewski's test of Gemini, which tried something not in the vast training corpus that requires real reasoning and checked how it ran, was the right test.
Held from this post (2)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: Gemini will turn out to be roughly in the same ballpark as GPT-4: useful for some things but not grounded in reality.
- The two reviews reached different assessments. Our reading, not assessed: Google's Gemini demo video was dubbed and edited to look like real-time interaction built from stills, misrepresenting the system.