Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Ed Zitron Where's Your Ed At Published

Expectations Versus Reality

We recorded 28 claims from this post. 18 were checked against evidence. 10 can’t be checked. 16 more are held. Read the post at its source.

Claims from this post (28)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: SupportedSaid

    “Yet these tiny little problems all lead to one overwhelming issue: that Sora isn’t so much a tool to make movies as it is a slot machine that spits out footage that may or may not be of any use at all.”

    What we tested: Sora's problems add up to one overwhelming issue: it is less a tool for making movies than a slot machine that produces footage that may or may not be of any use at all.

    What happened: Cederberg's first-hand account describes inconsistent character details, unwanted features and extensive selection and correction. The same interview also documents a completed film and useful creative applications, so it supports unpredictability rather than total uselessness. (fxguide, )

  2. Assessment: Can’t be checkedSaid

    “Sora’s problem isn’t really the amount of time it takes to generate a shot, but the imprecision that comes from using a generative artificial intelligence that doesn’t really know anything.”

    What we tested: Sora's main problem is imprecision caused by generative AI not really knowing anything.

  3. Assessment: SupportedSaid

    “To digress, none of this shit actually works if you’re making an actual movie.”

    What we tested: None of Sora's generative workflow actually works if you are making an actual movie.

    What happened: The interview documents missing control over continuity between separately generated shots. It supports the specific reliability concern, but does not establish that every movie use was impractical. (fxguide, )

  4. Assessment: UnresolvedSaid

    “Sora’s attempts to replace filmmakers are dead on arrival because these are impractical and ineffective solutions to problems that nobody complained about other than Hollywood executives.”

    What we tested: Sora will not succeed in replacing filmmakers because it is an impractical and ineffective solution.

  5. Assessment: UnresolvedSaid

    “To make Sora kind of useful, OpenAI will have to find a way to dramatically increase the precision of prompts, reduce hallucinations to almost nothing, and vastly increase processing power across the board.”

    What we tested: Sora will only become somewhat useful if OpenAI dramatically improves prompt precision, nearly eliminates hallucinations and vastly increases processing power.

  6. Assessment: Can’t be checkedSaid

    “Sora hasn’t even been launched, save to a few hand-picked companies that got to test an early version, meaning that the 10-to-20 minute wait between generations is something that’s likely to increase once more people use the product, and that’s before you consider how incredibly expensive it is to run a significantly more complex model than ChatGPT, which is already unprofitable for OpenAI to run.”

    What we tested: Sora's 10-20 minute generation wait is likely to increase once more people use it.

  7. Assessment: Can’t be checkedSaid

    “but the thing that I think he’s leaving out is that it may take an energy breakthrough (and more chips) for generative AI to approach any level of necessity”

    What we tested: In the author's view, it may take an energy breakthrough and more chips for generative AI to reach any level of necessity.

  8. Assessment: MixedSaid

    “OpenAI failed to make a more efficient model for Microsoft late last year, and while GPT-5 is meant to be “materially better,” it isn’t obvious what “better” means when GPT-4 performs worse at some tasks than its predecessor.”

    What we tested: OpenAI failed to build a more efficient model for Microsoft in late 2023, and GPT-4 performs worse than its predecessor at some tasks.

    What held: This supports the substance of the reported efficiency-project failure. It is secondary reporting, not a primary project record or independent corroboration of The Information's sources. (The Indian Express, )

    What did not: The account places cancellation in the middle of 2023, contradicting the literal late-2023 timing while corroborating that a failure occurred. The year-only event date avoids inventing a precise cancellation month. (Synthedia, )

  9. Assessment: UnresolvedSaid

    “and he still can’t get past any of these very obvious problems, partly because he isn’t technical and thus can’t really solve any of the problems himself, and partly because the problems he’s facing are burdened by the laws of mathematics and physics.”

    What we tested: Despite his connections and funding, Sam Altman still can't get past these obvious problems, partly because he isn't technical and so can't really solve them himself, and partly because the problems are burdened by the laws of mathematics and physics.

  10. Assessment: UnresolvedSaid

    “Generative AI is extremely expensive because even the simplest prompts require GPT-4 to run highly-complex mathematical equations on GPUs that cost $10,000 apiece.”

    What we tested: Generative AI is extremely expensive because even simple GPT-4 prompts require complex computation on $10,000 GPUs.

  11. Assessment: ContradictedSaid

    “Generative AI hallucinates because it doesn’t have a consciousness or any ability to learn or know anything.”

    What we tested: Generative AI hallucinates because it lacks consciousness and any ability to learn or know anything.

    What happened: The experiments demonstrate usable abilities acquired through training and adaptation to examples. They contradict an absolute absence of learning under the operational reading, without establishing consciousness. (arXiv; Tom B. Brown and colleagues, )

  12. Assessment: Can’t be checkedSaid

    “Or perhaps the truth was a little simpler: there wasn’t much artificial intelligence happening at all.”

    What we tested: Perhaps there wasn't much artificial intelligence happening at all in Rabbit's Large Action Model.

  13. Assessment: ContradictedSaid

    “Since Rabbit is acting as a third-party, and doesn’t use a standard like Oauth or SAML, it needs to keep user credentials in their original form. This is just insanely dangerous.”

    What we tested: Rabbit must store user credentials in original form, which is insanely dangerous.

    What happened: The report describes inspection during April 29–May 10; the event date marks that testing window's end. It found that service-issued session cookies, rather than entered usernames and passwords, were collected for storage. (Obscurity Labs, )

  14. Assessment: SupportedSaid

    “Putting aside the remarkable security concerns, the “Large Action Model” also looks like barely-functioning dogshit.”

    What we tested: Setting aside its security concerns, Rabbit's Large Action Model looks like it barely functions.

    What happened: This is the review's cumulative result as of publication: neither ride booking nor food ordering had succeeded. The surrounding passage describes repeated attempts and errors. (The Verge, David Pierce, )

  15. Assessment: SupportedSaid

    “Like I said, useful. But it isn’t AI.”

    What we tested: The Playwright scripting Rabbit uses is not AI.

    What happened: The versioned README describes browser automation and demonstrates explicit navigation, clicking and script execution. The assessment infers from that documented mechanism that these scripted actions are not themselves an AI model. (Microsoft, Playwright project, )

  16. Assessment: UnresolvedSaid

    “If, say, Uber changes the layout of its website, or changes the flow of certain actions (like booking a cab, or changing the pickup location), it would effectively break that entire part of the service.”

    What we tested: If an app like Uber changes its website layout or flows, the corresponding Rabbit R1 functionality would break.

  17. Assessment: SupportedSaid

    “Worse still, it lacks very, very basic features that you’d expect in a device from a decade ago — timers, reminders, calendars, photos, access to your emails, things that one would assume were obvious for an “AI assistant,””

    What we tested: The Rabbit R1 lacks basic features such as timers, reminders, calendars, photos and email access.

    What happened: The launch review identifies four of the five omissions. Its May 1 update reports other improvements without changing this feature list. (Tom's Guide, )

  18. Assessment: Can’t be checkedSaid

    “Companies that deliver features late should not get credit for features they are yet to deliver, and companies like Rabbit that engage in the language of science fiction to paper over their lazily-developed half-solutions should get buried for doing so.”

    What we tested: Companies should get no credit for features not yet delivered, and companies like Rabbit that hype half-solutions should be harshly criticized.

  19. Assessment: UnresolvedSaid

    “While Rabbit has tens of millions of dollars of venture funding, Rabbit’s business is about as functional as the R1, and I would bet money that it finds a way to try to charge its 100,000 or so customers some sort of subscription fee to stay afloat, just after the single most conspicuous tech reviewer called it “barely reviewable.””

    What we tested: Rabbit will try to charge its roughly 100,000 customers a subscription fee to stay afloat.

  20. Assessment: SupportedSaid

    “Funnily enough, this is exactly what the R1 is — an Android app shoved into a bright orange box.”

    What we tested: The Rabbit R1 is essentially an Android app in a box.

    What happened: The investigator identifies three APKs on MediaTek's Android software and describes extracting the firmware and running the launcher on phones. This supports the ordinary architectural reading, while showing that 'an app' simplifies a small collection of applications. (Rithwik Jayasimha, reproduced by Thread Reader, )

  21. Assessment: Can’t be checkedSaid

    “It’s time — it’s been time for a while — to accept that artificial intelligence isn’t the digital panacea that will automate our entire lives, and that reality may be far more mediocre than what Sam Altman and his ilk have been selling.”

    What we tested: People should accept that AI will not automate our entire lives and may be far more mediocre than Sam Altman claims.

  22. Assessment: Can’t be checkedSaid

    “because something that can automate your calendar, your emails, your music and your travel in a way that you can actually rely on requires operating system-level access and processing of information that may never exist.”

    What we tested: Something that can reliably automate your calendar, emails, music and travel requires operating-system-level access and information processing that may never exist.

  23. Assessment: ContradictedSaid

    “Large Language Models are — and always will be — incapable of reasoning or consciousness”

    What we tested: LLMs are and always will be incapable of reasoning or consciousness.

    What happened: SATBench reports performance well above its 50% chance baseline on generated, solver-checked puzzles, without allowing external tools during model evaluation. Reasoning-trace checks and partial human validation support a limited inference capability, although they also reveal errors. (Anjiang Wei and colleagues, arXiv, )

  24. Assessment: MixedSaid

    “Every single story about artificial intelligence seems to begin with “well, AI can’t do much yet, but…” and then lean into sloppy science fiction that masks that these “autonomous agents” are effectively chatbots that can answer questions based on a document, tech that’s existed for a decade or more.”

    What we tested: So-called autonomous agents are effectively chatbots answering questions from documents, a decade-old technology.

    What held: The authors describe Watson's February 2011 system deriving answers using unstructured text and matching evidence from passages. This establishes the document-question-answering capability more than a decade before the post, without establishing equivalence to modern agent architectures. (Association for Logic Programming; Adam Lally and Paul Fodor, )

    What did not: WebVoyager observes websites, chooses actions and executes them through a browser, with an evaluation reporting 55.7% task success in this original version. Those interactive capabilities exceed answering questions from a supplied document, even though reliability was incomplete. (Hongliang He and colleagues, arXiv, )

  25. Assessment: UnresolvedSaid

    “AI agents are error-prone, and the likelihood of failure in an automated task like moving a calendar invite or ordering an Uber is exponentially higher and less tolerable in a consumer setting.”

    What we tested: AI agents are error-prone, and in a consumer setting the likelihood of failure in automated tasks like moving a calendar invite or ordering an Uber is exponentially higher and less tolerable.

  26. Assessment: Can’t be checkedSaid

    “Companies like Google, OpenAI and Rabbit should be treated with suspicion when they make flowery promises, and derision when they continually fail to do anything useful.”

    What we tested: Companies like Google, OpenAI and Rabbit should be treated with suspicion when they make flowery promises, and with derision when they continually fail to do anything useful.

  27. Assessment: Can’t be checkedSaid

    “The tech industry needs to start building things for real people again.”

    What we tested: The tech industry should build products that solve real problems for ordinary people.

  28. Assessment: Can’t be checkedSaid

    “Venture capital investment is at a five-year-low because an alarming amount of VCs don’t seem to understand technology or business, and seem entirely married to hype cycles.”

    What we tested: Venture capital investment is at a five-year low because an alarming number of VCs seem not to understand technology or business and seem entirely married to hype cycles.

Held from this post (16)

These records did not pass every check, so nothing about them is published. Held is not a finding.