Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

GPT-4’s successes, and GPT-4’s failures

We recorded 13 claims from this post. 7 were checked against evidence. 6 can’t be checked. 4 more are held. Read the post at its source.

Claims from this post (13)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: SupportedSaid

    “It can see (though we don’t have a lot of details on that yet); it does astonishingly well on a whole bunch of standardized tests, like LSATs, GREs, and SATs.”

    What we tested: GPT-4 has vision capability and performs astonishingly well on standardized tests such as the LSAT, GRE, and SAT.

    What happened: The report documents launch-model image processing and reports approximately the 88th percentile on LSAT, 93rd on SAT reading/writing, 89th on SAT mathematics, 80th on GRE quantitative and 99th on GRE verbal. GRE writing was approximately the 54th percentile; event_date denotes the launch model's capability, not an asserted examination date. (OpenAI, published on arXiv, )

  2. Assessment: UnresolvedSaid

    “It doesn’t actually solve any of the core problems of truthfulness and reliability that I laid out in my infamous March 2022 essay Deep Learning is Hitting a Wall.”

    What we tested: GPT-4 does not solve any of the core truthfulness and reliability problems identified in Marcus's March 2022 essay.

  3. Assessment: UnresolvedSaid

    “Alignment is still shaky; you still wouldn’t be able to use it reliably to guide robots or scientific discovery, the kinds of things that made me excited about A(G)I in the first place.”

    What we tested: GPT-4's alignment remains shaky and it cannot be used reliably to guide robots or scientific discovery.

  4. Assessment: SupportedSaid

    “Outliers remain a problem, too.”

    What we tested: Outliers remain a problem for GPT-4.

    What happened: OpenAI acknowledges that specially constructed prompts can still defeat GPT-4's safeguards. This supports the ordinary reading that unusual inputs remain a source of failures. (OpenAI, )

  5. Assessment: Can’t be checkedSaid

    “Although GPT-4 is clearly better than GPT-3 and 3.5, it is not so far as I can tell, qualitatively better, only quantitatively better; as noted above, the limits remain more or less the same.”

    What we tested: So far as Marcus can tell, GPT-4, though clearly better than GPT-3 and 3.5, is only quantitatively and not qualitatively better, and its limits remain more or less the same.

  6. Assessment: Can’t be checkedSaid

    “Massive scaling has not thus far lead to revolution.”

    What we tested: As of March 2023, massive scaling has not led to a revolution in AI.

  7. Assessment: Can’t be checkedSaid

    “The quantitative improvements may (or may not) have considerable commercial implications in particular domains, but it’s not the massive win for the scaling hypotheses that memes like this had forecast.”

    What we tested: GPT-4's quantitative improvements may or may not have considerable commercial implications in particular domains, but GPT-4 is not the massive win for the scaling hypotheses that memes like the one he cites had forecast.

  8. Assessment: UnresolvedSaid

    “On its own, it still requires frequent, massive retraining to be current with the news; GPT 3.5 knew nothing of 2022; GPT-4 seems to know little of 2023.”

    What we tested: On its own, GPT-4 still requires frequent massive retraining to stay current with the news.

  9. Assessment: SupportedSaid

    “We don’t know how big it is; we don’t know what the architecture is, we don’t know how much energy was used; we don’t how many processors were used; we don’t know what it was trained on etc.”

    What we tested: OpenAI has not disclosed GPT-4's size, architecture, energy use, processor count, or training data.

    What happened: Sutskever directly confirms withholding size and training data. The article also documents the absent energy and hardware information and reproduces OpenAI's statement withholding architectural and training details. (The Verge, )

  10. Assessment: Can’t be checkedSaid

    “It is a step backwards for science, because it sets a new precedent for pretending to be scientific while revealing absolutely nothing.”

    What we tested: GPT-4's release is a step backwards for science because it sets a new precedent for pretending to be scientific while revealing absolutely nothing.

  11. Assessment: UnresolvedSaid

    “Because of all that, we don’t know what explains its success, and we don’t know how to predict its failures:”

    What we tested: Because GPT-4's size, architecture, energy use, processors, and training data have not been disclosed, we don't know what explains its success and we don't know how to predict its failures.

  12. Assessment: Can’t be checkedSaid

    “One more giant step for hype, but not necessarily a giant step for science, AGI, or humanity.”

    What we tested: GPT-4 is a big step for hype but not necessarily for science, AGI, or humanity.

  13. Assessment: Can’t be checkedSaid

    “and (b) it puts all of us in an extremely poor position to predict what GPT-4 consequences will be for society, if we have no idea of what is in the training set and no way of anticipating which problems it will work on and which it will not.”

    What we tested: If we have no idea what is in GPT-4's training set and no way of anticipating which problems it will work on and which it will not, all of us are in an extremely poor position to predict what GPT-4's consequences will be for society.

Held from this post (4)

These records did not pass every check, so nothing about them is published. Held is not a finding.