Horse rides Astronaut, redux
We recorded 8 claims from this post. 7 were checked against evidence. 1 can’t be checked. 2 more are held. Read the post at its source.
Claims from this post (8)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
“As Ernie Davis, Scott Aaronson, and I showed, a week or so after the release, it’s easily fooled, especially with complex sentences and words like “similar” that would require a deep understanding of language to be interpreted correctly”
What we tested: DALL-E 2 is easily fooled by complex sentences and words requiring deep language understanding.
Assessment: Can’t be checkedSaid
“DALL-E 2 is, I dare say, not as smart as it seems.”
What we tested: DALL-E 2 is not as smart as it appears.
Assessment: UnresolvedSaid
“Again, here is my first and only try. Not a single horse riding an astronaut:”
What we tested: On a first attempt in February 2024, the image model produced no image of a horse riding an astronaut for the 'horse rides astronaut' style prompt.
Assessment: UnresolvedSaid
“The lesson I drew from from that, back then, in May 2022, was the problem lies not in Dall-E’s drawing ability, but in its ability to cope with noncanonical prompts.”
What we tested: DALL-E's failure stems from inability to cope with noncanonical prompts rather than from limits of its drawing ability.
Assessment: UnresolvedSaid
“In other words, the problem was with its language understanding, rather than with illustration per se. The model was perfectly capable of drawing a horse riding an astronaut, it just didn’t know that the phrase horse rides astronaut was a request for a horse riding an astronaut.”
What we tested: DALL-E's failure on 'horse rides astronaut' was due to deficient language understanding, not illustration ability.
Assessment: UnresolvedSaid
“One winner out of three to be exact.”
What we tested: After several retries, Gemini produced only one correct horse-riding-astronaut image out of three.
Assessment: UnresolvedSaid
“I think we can safely reach the same conclusion as before. Gemini can draw the desired image. But it’s still like pulling teeth to get there.”
What we tested: Gemini can draw a horse riding an astronaut, but getting it to do so is very difficult.
Assessment: ContradictedSaid
“In some ways we have made progress, but in the ones I keep harping on—factuality and compositionality—we have not.”
What we tested: Generative AI has made no progress on factuality and compositionality since 2022, despite progress in other respects.
What happened: The report compares factual accuracy on TruthfulQA and reports gains over earlier models. This contradicts a categorical absence of factuality progress, subject to the company's evaluation methods and contamination caveat. (OpenAI via arXiv, )
Held from this post (2)
These records did not pass every check, so nothing about them is published. Held is not a finding.