Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

Form, function, and the giant gulf between drawing a picture and understanding the world

We recorded 6 claims from this post. 5 were checked against evidence. 1 can’t be checked. 5 more are held. Read the post at its source.

Claims from this post (6)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: Can’t be checkedSaid

    “In assessing progress towards general intelligence, the critical question should be, how much do systems like Dall-E, Imagen, Midjourney, and Stable Diffusion really understand the world, such that they can reason on and act on that knowledge?”

    What we tested: Progress towards general intelligence from image systems like DALL-E, Imagen, Midjourney and Stable Diffusion should be assessed by how much they understand the world so they can reason and act on it.

  2. Assessment: UnresolvedSaid

    “On #1, the answer is a clear yes; only highly trained human artists could do better.”

    What we tested: Image synthesis systems generate high quality images that only highly trained human artists could surpass.

  3. Assessment: SupportedSaid

    “On #2, the answer is mixed. They do well on some inputs (like astronaut rides horse) but more poorly on others (like horse rides astronaut, which I discussed in an earlier post).”

    What we tested: Image synthesis systems only inconsistently match their linguistic input to the images produced, doing well on some prompts and poorly on others.

    What happened: The study's human judgments show some successful matching alongside widespread relational failures. Its results were reported by July; the precise image-generation dates are not supplied. (arXiv; Colin Conwell and Tomer Ullman, )

  4. Assessment: UnresolvedSaid

    “Nine more tries, and only one very marginal success (top right corner):”

    What we tested: Stable Diffusion produced only one very marginal success in nine further attempts at a part-function prompt.

  5. Assessment: UnresolvedSaid

    “Negation is, as ever, a problem.”

    What we tested: Text-to-image systems like Stable Diffusion continue to have problems with negation in prompts.

  6. Assessment: UnresolvedSaid

    “Even “draw a white bicycle with green wheels”, which focuses purely on part-whole relationships without function or complex syntax, is problematic:”

    What we tested: Stable Diffusion has trouble even with simple part-whole prompts such as a white bicycle with green wheels.

Held from this post (5)

These records did not pass every check, so nothing about them is published. Held is not a finding.