Supported claims, page 5 of 8
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Supported claims (775)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 5 of 8: claims 401 to 500 of 775, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Gary Marcus Observation
“Two recent studies show nothing like “10x” improvement :”
Our reading: Two recent studies of generative AI coding tools show nothing like a 10x productivity improvement.
No horizon Evidence as of
Limitation: These findings concern the two identified studies and do not establish effects for all coding tools, developers or later versions.
Source: Sorry, GenAI is NOT going to 10x computer programming
Assessment: SupportedSaid
Gary Marcus Observation
“The data are coming in – and it’s not.”
Our reading: Emerging data show generative AI is not improving programmer productivity by a factor of 10.
No horizon Evidence as of
Limitation: The studies use different productivity measures and settings.
Source: Sorry, GenAI is NOT going to 10x computer programming
Assessment: SupportedSaid
Ed Zitron Observation
“In essence, Amazon is benefitting from AI in much the same way NVIDIA is.”
Our reading: In essence, Amazon is benefitting from AI in much the same way NVIDIA is, by supplying the compute people need while working out what AI is useful for.
No horizon Evidence as of
Limitation: No infrastructure revenue share is established.
Source: The Other Bubble
Assessment: SupportedSaid
Gary Marcus Observation
“many lawsuits pending”
Our reading: OpenAI faces many pending lawsuits.
No horizon Evidence as of
Limitation: Five is an evaluator-supplied threshold, not a number Marcus stated.
Source: OpenAI’s slow-motion train wreck
Assessment: SupportedSaid
Gary Marcus Observation
“From left to right that’s Ilya Sutskever (now gone, less than a year later), Greg Brockman (on leave, at least until the end of the year), CTO Mira Murati (departure just announced) and Sam Altman (fired, and then rehired).”
Our reading: Of the four OpenAI leaders on the cover, Ilya Sutskever is gone less than a year later, Greg Brockman is on leave at least until the end of the year, CTO Mira Murati's departure was just announced, and Sam Altman was fired and then rehired.
No horizon Evidence as of
Limitation: The eventual duration of Brockman's leave differed from the announced plan.
Source: OpenAI’s slow-motion train wreck
Assessment: SupportedSaid
Gary Marcus Observation
“Co-founders John Schulman also left, so did Andrej Karpathy and Jan Leike, and perhaps a dozen others.”
Our reading: John Schulman, Andrej Karpathy, Jan Leike and perhaps a dozen others have left OpenAI.
No horizon Evidence as of
Limitation: The additional count remains unverified and lacks a defined counting period.
Source: OpenAI’s slow-motion train wreck
Assessment: SupportedSaid
Ed Zitron Forecast
“I believe that, at the very least, Microsoft will begin reducing costs in other areas of its business as a means of helping sustain the AI boom.”
Our reading: Microsoft will begin reducing costs in other parts of its business to help sustain its AI spending.
Settled before its deadline Evidence as of
Limitation: The evidence establishes a contributing financial motive, not a quantified transfer of savings.
Source: The Subprime AI Crisis
Assessment: SupportedSaid
Gary Marcus Observation
“I rushed out a hot take on X, my usual cautions about how LLMs aren’t really AGI:”
Our reading: LLMs, including OpenAI's new o1, are not really AGI.
No horizon Evidence as of
Limitation: The system card tests several model checkpoints and configurations; it is not an exhaustive evaluation of all contemporary LLMs [e3].
Assessment: SupportedSaid
Gary Marcus Observation
“Almost exactly what I predicted we would see with GPT-4, back on Christmas Day 2022.”
Our reading: Altman's description of o1 as 'still flawed, still limited, seem more impressive on first use' is almost exactly what Marcus predicted we would see with GPT-4 on Christmas Day 2022.
No horizon Evidence as of
Limitation: The agreement is thematic, not verbatim.
Assessment: SupportedSaid
Gary Marcus Causal claim
“You should be watching to see what the candidates plan to do about fraud and disinformation, as AI contributes to escalation in both.”
Our reading: AI is contributing to an escalation in fraud and disinformation.
No horizon Evidence as of
Limitation: The evidence does not quantify AI-attributable growth in aggregate fraud losses or successful deception [e1, e2].
Source: What will the candidates say about AI in tonight’s debate?
Assessment: SupportedSaid
Gary Marcus Observation
“And you should be watching to see what they plan to do about the growing problem of nonconsensual deep fake porn, and especially how that might affect the already vulnerable population of teenage girls, who have been hard by social media.”
Our reading: Nonconsensual deepfake porn is a growing problem.
No horizon Evidence as of
Limitation: The Korean police figures do not establish worldwide prevalence or separate increased offending from increased recognition and reporting [e1].
Source: What will the candidates say about AI in tonight’s debate?
Assessment: SupportedSaid
Gary Marcus Observation
“California’s pending AI bill SB-1047 was supposed to be a tiny step in that direction, and even that has gotten watered down.”
Our reading: California's SB-1047, already only a small step toward external AI governance, has been watered down.
No horizon Evidence as of
Limitation: The evidence is the author's dated amendment explanation, not a complete comparison of all bill versions.
Source: Scoop: What former employees of OpenAI are worried about
Assessment: SupportedSaid
Gary Marcus Observation
“When Sam Altman stood next to me in the Senate, he told the Senate that he strongly favored AI regulation. But behind the scenes his team fought to water down the EU AI Act.”
Our reading: Sam Altman told the Senate, standing next to Marcus, that he strongly favored AI regulation, but behind the scenes his team fought to water down the EU AI Act.
No horizon Evidence as of
Limitation: The lobbying finding relies on reputable reporting because the available primary PDF extraction is corrupted.
Assessment: SupportedSaid
Gary Marcus Observation
“It also provides important whistleblower protections, which as many ex OpenAI employees have made clear, are critical.”
Our reading: SB-1047 provides important whistleblower protections, which, as many ex-OpenAI employees have made clear, are critical.
No horizon Evidence as of
Limitation: The assessment concerns a proposed bill, not protections already in force.
Assessment: SupportedSaid
Gary Marcus Observation
“Most or all of the major big tech companies joined a lobbying organization that fought SB-1047, tooth and nail, despite broad public support for the bill.”
Our reading: Most or all major big tech companies joined a lobbying group that fought SB-1047 hard despite broad public support for it.
No horizon Evidence as of
Limitation: The majority assessment uses the contract's five-company group: Alphabet/Google, Amazon, Apple, Meta, and Microsoft.
Assessment: SupportedSaid
Gary Marcus Observation
“California’s SB-1047, not yet signed into law, but on its way to being one of the first really substantive AI bills in the US, primarily addressed to liability around catastrophic risks, was significantly weakened in last-minute negotiations.”
Our reading: California's SB-1047 was significantly weakened in last-minute negotiations.
No horizon Evidence as of
Limitation: Significance is an evaluative judgment about the documented legal changes.
Assessment: SupportedSaid
Gary Marcus Causal claim
“As I emphasized last year in front of the Senate, there is not one risk but many: bias, disinformation, defamation, toxicity, invasion of privacy, and so much more, to say nothing of risks to employment, intellectual property, and the environment, precisely because recent techniques can be applied in so many ways, still poorly understood.”
Our reading: AI poses not one risk but many, including bias, disinformation, defamation, toxicity, invasion of privacy and so much more, plus risks to employment, intellectual property and the environment, precisely because recent techniques can be applied in so many ways and are still poorly understood.
No horizon Evidence as of
Limitation: The evidence includes scientific syntheses and a preprint, rather than one experiment testing the entire claim.
Assessment: SupportedSaid
Gary Marcus Observation
“Lastly, asking for standards is not unique to AI; it's common across many industries to ask that companies evaluate the safety of their products according to set standards: just look at the pharmaceutical industry, or aviation, automobiles, etc.”
Our reading: Asking for standards is not unique to AI: in many industries, such as pharmaceuticals, aviation and automobiles, it is common to ask companies to evaluate the safety of their products against set standards.
No horizon Evidence as of
Limitation: The evidence establishes the three named U.S. examples, not a numerical survey of every industry or jurisdiction.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Gary Marcus Observation
“And much of what would be required is already something companies committed to voluntarily, in discussions at the White House and in Seoul.”
Our reading: Much of what SB-1047 would require is something AI companies already voluntarily committed to at the White House and in Seoul.
No horizon Evidence as of
Limitation: The assessment is a qualitative comparison, not a count of statutory clauses.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI recently put Paul Nakasone (ex NSA) on the board”
Our reading: OpenAI recently put Paul Nakasone, formerly of the NSA, on its board.
No horizon Evidence as of
Limitation: This establishes the appointment and career background, not any implication about surveillance.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: SupportedSaid
Gary Marcus Observation
“An awful lot of key players have left the island. Most voluntarily.”
Our reading: Many key OpenAI people have left, most of them voluntarily.
No horizon Evidence as of
Limitation: The roster is a transparent group drawn from contemporaneous reporting, not a complete personnel register.
Source: The OpenAI Plot Thickens
Assessment: SupportedSaid
Gary Marcus Observation
“Nvidia dropped 6%, 20% over the last month”
Our reading: Nvidia stock fell 6% on August 5, 2024, and 20% over the previous month.
No horizon Evidence as of
Limitation: The result depends on reading the figures as rounded closing-price changes.
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI tempered expectations for its next event, and said we wouldn’t see GPT-5 then.”
Our reading: OpenAI tempered expectations for its next event and said we would not see GPT-5 at that event.
No horizon Evidence as of
Limitation: Evidence is a reporter's direct quotation, not an accessible original company announcement.
Assessment: SupportedSaid
Ed Zitron Observation
“That $32.5 is now $195. If it wanted to have the same margins, it would need to make $105 in profit from each user. So, $300. $300 a user for a product it hasn’t even built yet.”
Our reading: On his estimates, OpenAI's search product would need about $300 per user to match Google's search margins.
No horizon Evidence as of
Limitation: The evidence item contains an evaluator-derived calculation, not reported OpenAI financial results.
Source: Burst Damage
Assessment: SupportedSaid
Ed Zitron Observation
“For reasons I’ve alluded to at the start of this article, investors are getting impatient with the AI hype boom — and, more specifically, with the giant tech companies that are bankrolling it.”
Our reading: Investors are getting impatient with the AI boom and with the big tech companies funding it.
No horizon Evidence as of
Limitation: The evidence establishes a visible concern, not a measured percentage of investors.
Source: Burst Damage
Assessment: SupportedSaid
Ed Zitron Conditional forecast
“In the event that Anthropic or OpenAI is able to raise, we’ll see the cycle continue another quarter, at which point something crazy needs to happen to make any of this worth it. If these companies can raise, they’ll keep the bubble inflated, though not forever.”
Our reading: If Anthropic or OpenAI raises money, the bubble will last another quarter, but not forever.
Deadline passed Evidence as of
Only if: if Anthropic or OpenAI is able to raise
Source: Burst Damage
Assessment: SupportedSaid
Ed Zitron Observation
“Regardless, this article has put it very plainly: OpenAI only gets access to the latest technology as fast as Microsoft permits.”
Our reading: OpenAI gets access to the latest hardware only as fast as Microsoft allows.
No horizon Evidence as of
Limitation: The conclusion concerns advanced cloud compute, not every technology OpenAI might acquire.
Source: Burst Damage
Assessment: SupportedSaid
Ed Zitron Observation
“We’re starting to see a shift — or, more accurately, a centralization — in the point of failure for AI.”
Our reading: We are starting to see a shift, more accurately a centralization, in the point of failure for AI: away from a large, disparate group of startups and investors and onto a few companies, namely Microsoft, Google, Amazon, Facebook and, to a much lesser extent, Apple.
No horizon Evidence as of
Limitation: The result depends on reading point of failure as concentrated dependence and exposure.
Source: Burst Damage
Assessment: SupportedSaid
Gary Marcus Observation
“Over a quarter century later, handling outliers is still the Achilles’ Heel of neural networks.”
Our reading: More than a quarter century after his 1998 work, handling outliers (distribution shift) is still the Achilles' heel of neural networks.
No horizon Evidence as of
Limitation: The decisive study concerns multimodal language models rather than every neural architecture.
Source: This one important fact about current AI explains almost everything
Assessment: SupportedSaid
Gary Marcus Observation
“Another GenAI monetization scheme bites the dust.”
Our reading: Meta dropping a major GenAI project is one more GenAI monetization scheme that has failed.
No horizon Evidence as of
Limitation: Shutdown evidence comes from a reputable news report quoting Meta directly; no standalone Meta shutdown announcement was recovered.
Assessment: SupportedSaid
Gary Marcus Observation
“Consequently, any “reasoning” or “planning” in LLMs is hit or miss, viable if the particulars of a given situation are close enough to what is in the training set, but highly vulnerable if they are not.”
Our reading: Marcus says any reasoning or planning in LLMs is hit or miss: viable if a situation's particulars are close enough to the training set, but highly vulnerable if they are not.
No horizon Evidence as of
Limitation: Reasoning models released after July 2024 are not assessed.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Much as I admire the new pair of systems, they do have a serious shortcoming and one that is not unfamiliar. The systems both rely on a kind of cheat on the input: trained human coders translate all the Olympiad's input sentences into mathematical form.”
Our reading: AlphaProof and AlphaGeometry 2 rely on human coders translating Olympiad problems into formal form, a serious shortcoming.
No horizon Evidence as of
Limitation: The assessment concerns the July 2024 demonstration, not later systems.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Nonetheless, GDM's new work is powerful proof of concept (pun intended!). Neural networks and symbol systems can be profitably combined, contra Hinton and LeCun.”
Our reading: Google DeepMind's work shows neural networks and symbolic systems can be profitably combined.
No horizon Evidence as of
Limitation: Technical usefulness does not imply financial profitability.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Hallucinations and boneheaded errors of reasoning and what I like to call discomprehensions are, as far as I can tell, an inevitable byproduct of LLMs; certainly they have yet to disappear, despite many promises to the contrary.”
Our reading: Marcus says that, as far as he can tell, hallucinations, boneheaded reasoning errors and 'discomprehensions' are an inevitable byproduct of LLMs, and that they certainly have not yet disappeared despite many promises to the contrary.
No horizon Evidence as of
Limitation: Only the 'yet to disappear' part is scored.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Long-term adoption (vs mere experimentation) has been modest, there is no significantly better tech now than there was in January; GPT-5 is still a no show.”
Our reading: GPT-5 had not been released as of July 2024.
No horizon Evidence as of
Limitation: This establishes public release status, not the state of private development.
Assessment: SupportedSaid
Gary Marcus Causal claim
“As Ernie Davis and I pointed out in Rebooting AI, five years ago, part of the reason we are struggling with AI in complex AI systems is that we still lack adequate techniques for engineering complex systems. That hasn’t really changed.”
Our reading: Struggles with complex AI systems stem partly from a lack of adequate techniques for engineering complex systems, which has not really changed since 2019.
No horizon Evidence as of
Limitation: The causal assessment synthesizes technical explanations; it is not a controlled estimate of engineering's contribution to failures.
Source: Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call
Assessment: SupportedSaid
Ed Zitron Causal claim
“For Goldman to suddenly turn on the AI movement suggests that it’s extremely anxious about the future of generative AI, with almost everybody agreeing on one core point: that the longer this tech takes to make people money, the more money it's going to need to make.”
Our reading: The longer generative AI takes to make money, the more money it will need to make.
No horizon Evidence as of
Limitation: The conclusion assumes continued expenditure or a positive opportunity cost.
Source: Pop Culture
Assessment: SupportedSaid
Gary Marcus Forecast
“As of today, I am more confident than ever that GPT-5 won’t land this year.”
Our reading: OpenAI will not release GPT-5 in 2024.
Deadline passed Evidence as of
Limitation: This record assesses only the release timing of a model named GPT-5. It does not assess Marcus's reasons for the forecast or what GPT-5 could do.
Assessment: SupportedSaid
Gary Marcus Forecast
“As of today, I am more confident than ever that GPT-5 won’t land this year.”
Our reading: GPT-5 will not be released in 2024.
Deadline passed Evidence as of
Limitation: This assesses the named public release, not private prototypes or differently named models.
Assessment: SupportedSaid
Gary Marcus Observation
“Exhibit A: OpenAI’s CTO Mira Murati just publicly acknowledged what I long suspected: there is no mind blowing GPT-5 behind the scenes as of yet.”
Our reading: As of June 2024, OpenAI has no mind-blowing GPT-5 model waiting behind the scenes.
No horizon Evidence as of
Limitation: The decisive source is contemporaneous news reporting because the relevant primary interview transcript could not be recovered.
Assessment: SupportedSaid
Ed Zitron Conditional forecast
“As things get desperate, Altman will use the only approach he knows: sheer force of will. He will push OpenAI to grow and sell into as many industries as possible, a specious hype-man selling to other specious hype-men desperate to connect their company to an AI bubble inflated by Altman himself.”
Our reading: As things get desperate, Altman will use sheer force of will and push OpenAI to grow and sell into as many industries as possible.
Settled before its deadline Evidence as of
Only if: as things get desperate
Source: Silicon Valley's False Prophet
Assessment: SupportedSaid
Gary Marcus Observation
“In many ways, it is much weaker than what Altman and I talked about at the Senate last year.”
Our reading: SB-1047 is in many ways much weaker than the AI regulation Marcus and Sam Altman discussed at the Senate in 2023.
No horizon Evidence as of
Limitation: The result depends on reading strength through the licensing and review mechanisms highlighted in the surrounding passage.
Assessment: SupportedSaid
Gary Marcus Observation
“For one thing, the bad press about Sam Altman and OpenAI, who once seemingly could do no wrong, just keeps coming.”
Our reading: Negative press coverage of Sam Altman and OpenAI, once seen as able to do no wrong, keeps coming.
No horizon Evidence as of
Limitation: This is a finding about the existence of repeated coverage, not its prevalence or rate of increase.
Source: AI as Reality Television
Assessment: SupportedSaid
Ed Zitron Observation
“Despite being an inherently AI-powered feature, it can’t distinguish between ordinary computer usage and genuinely sensitive information — like passwords, health information, or trade secrets that must be protected at all costs.”
Our reading: Microsoft's AI-powered Recall feature cannot distinguish ordinary computer usage from sensitive information such as passwords or health data.
No horizon Evidence as of
Limitation: The primary wording survives here in a contemporary secondary source.
Source: The Rot-Com Bubble
Assessment: SupportedSaid
Gary Marcus Observation
“For all the daily claims of “exponential progress”, and for all the tens of billions that have been invested since, reliability is still a dream, and of course toxicity and hallucinations are still with us.”
Our reading: Despite tens of billions of dollars invested since 2022, deep learning systems are still unreliable and still produce toxicity and hallucinations.
No horizon Evidence as of
Limitation: The reliability phrase is rhetorical and supplies no threshold.
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI called itself open, and traded on the notion of being open, but even as early as May 2016 knew that the name was misleading.”
Our reading: OpenAI traded on being 'open' while knowing since May 2016 that the name was misleading.
No horizon Evidence as of
Limitation: The decisive May conversation comes from contemporaneous journalism, not a released internal email.
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: SupportedSaid
Gary Marcus Observation
“Altman once (in 2016) promised that outsiders would play an important role in the company’s governance; that key promise has not been kept.”
Our reading: Altman's 2016 promise that outsiders would play an important role in OpenAI governance has not been kept.
No horizon Evidence as of
Limitation: The finding concerns public electoral participation, not the absence of independent directors.
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI has trained on a massive amount of copyrighted material, without consent, and in many instances without compensation.”
Our reading: OpenAI has trained on a massive amount of copyrighted material without consent and often without compensation.
No horizon Evidence as of
Limitation: The conclusion combines company disclosures with a rights holder’s filing; the filing is not a court finding.
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: SupportedSaid
Gary Marcus Observation
“What the LLM does is more akin to what some high school students do when they plagiarize: change a few words here or there, while still sticking close to the original.”
Our reading: What the LLM in the glue-pizza example does is more akin to what some high school students do when they plagiarize: changing a few words while still sticking close to the original.
No horizon Evidence as of
Limitation: The supported scope is the supplied example, not LLM behavior generally.
Assessment: SupportedSaid
Gary Marcus Observation
“Many people are reaching essential the same perceptions.”
Our reading: Many people are reaching essentially the same perceptions of Sam Altman.
No horizon Evidence as of
Limitation: The undefined word 'many' makes the result depend on a qualitative reading.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Gary Marcus Observation
“In November, the narrative was about how everybody loved Sam. Things have changed. A lot.”
Our reading: The narrative about Sam Altman, which in November was that everybody loved him, has changed a lot.
No horizon Evidence as of
Limitation: The result depends on reading 'narrative' as prominent coverage and commentary, rather than measured population sentiment.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Gary Marcus Observation
“I shared some of my own thoughts about which Pandora’s Boxes had (LLMs) and had not (AGI) already been opened.”
Our reading: The Pandora's Box of LLMs has already been opened.
No horizon Evidence as of
Limitation: Public availability does not establish the scale of malicious use or prove that all safeguards can be defeated.
Source: Jane Goodall, AI Skeptic
Assessment: SupportedSaid
Ed Zitron Observation
“Yet these tiny little problems all lead to one overwhelming issue: that Sora isn’t so much a tool to make movies as it is a slot machine that spits out footage that may or may not be of any use at all.”
Our reading: Sora's problems add up to one overwhelming issue: it is less a tool for making movies than a slot machine that produces footage that may or may not be of any use at all.
No horizon Evidence as of
Limitation: Evidence concerns the early preview and one documented production.
Source: Expectations Versus Reality
Assessment: SupportedSaid
Ed Zitron Observation
“Like I said, useful. But it isn’t AI.”
Our reading: The Playwright scripting Rabbit uses is not AI.
No horizon Evidence as of
Limitation: The assessment follows the immediate antecedent and supplied normalized proposition.
Source: Expectations Versus Reality
Assessment: SupportedSaid
Ed Zitron Observation
“Putting aside the remarkable security concerns, the “Large Action Model” also looks like barely-functioning dogshit.”
Our reading: Setting aside its security concerns, Rabbit's Large Action Model looks like it barely functions.
No horizon Evidence as of
Limitation: The phrase has no numerical threshold.
Source: Expectations Versus Reality
Assessment: SupportedSaid
Ed Zitron Observation
“Funnily enough, this is exactly what the R1 is — an Android app shoved into a bright orange box.”
Our reading: The Rabbit R1 is essentially an Android app in a box.
No horizon Evidence as of
Limitation: The phone port does not establish complete feature equivalence or continuing service access on unofficial installations [e1, e2].
Source: Expectations Versus Reality
Assessment: SupportedSaid
Gary Marcus Observation
“You might think that if OpenAI said, hey, we don’t want our software used for military purposes, Microsoft would respect that and say, ok cool, but they haven’t.”
Our reading: Microsoft has not respected OpenAI's stated wish that its software not be used for military purposes.
No horizon Evidence as of
Limitation: The result depends on reading respect as consistency with a public position, rather than contractual compliance.
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: SupportedSaid
Gary Marcus Observation
“Playing the field with Mistral wasn’t enough; now he is building a really big model, big enough to compete directly with GPT-4, internally:”
Our reading: Beyond partnering with Mistral, Microsoft is building a large model in-house that is big enough to compete directly with GPT-4.
No horizon Evidence as of
Limitation: Support depends on reading compete as intended competitive positioning.
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: SupportedSaid
Gary Marcus Observation
“You might think that after Satya offered to rescue Sam after Sam got fired, that Sam wouldn’t want to compete directly with Microsoft, but they are.”
Our reading: Sam Altman's OpenAI competes directly with Microsoft, even though Satya Nadella offered to rescue Altman after he was fired.
No horizon Evidence as of
Limitation: The customer meetings are documented by reputable reporting based on attendees, rather than public meeting records.
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: SupportedSaid
Ed Zitron Observation
“And all of this is deeply relevant at a time when Rabbit’s R1 device has launched without several key features, such as “Teach Mode” and a stable, reliable version of its alleged “Large Action Model.””
Our reading: The Rabbit R1 launched without key features, including Teach Mode and a stable, reliable version of its Large Action Model.
No horizon Evidence as of
Limitation: The launch reviews do not establish a representative task-success rate across users and devices [e2, e3].
Source: Rabbit Holed
Assessment: SupportedSaid
Ed Zitron Observation
“The device — which costs $200 — has already started shipping and the reviews are not good.”
Our reading: The $200 Rabbit R1 AI device has started shipping and its reviews are not good.
No horizon Evidence as of
Limitation: This is a comparison of prominent launch reviews, not an exhaustive review census.
Source: Rabbit Holed
Assessment: SupportedSaid
Gary Marcus Forecast
“People who buy Altman’s narrative will continue to invest.”
Our reading: People who buy Altman's narrative will continue to invest.
Settled before its deadline Evidence as of
Limitation: Public endorsement is a proxy for belief.
Source: The Sam Altman Playbook
Assessment: SupportedSaid
Gary Marcus Observation
“At the same talk at Stanford, Altman grandoisely asserted, with no qualification, that “we’re making AGI”, without touching on the possibility that current approaches might not get us there — despite (a) the enormous known problems with reliability, reasoning, planning and hallucinations and (b) the fact that the entire field, despite being massively well-capitalized, has struggled for over year (and in fact for decades) to solve any of these problems.”
Our reading: Current AI approaches have enormous known problems with reliability, reasoning, planning and hallucinations, and the entire field, despite being massively well-capitalized, has struggled for over a year, and in fact for decades, to solve any of these problems.
No horizon Evidence as of
Limitation: The conclusion depends on reading solve as general resolution, rather than improvement or success on bounded tasks.
Source: The Sam Altman Playbook
Assessment: SupportedSaid
Gary Marcus Causal claim
“The very process that LLMs use to generalize also creates hallucinations.”
Our reading: The same process LLMs use to generalize is what creates their hallucinations.
No horizon Evidence as of
Limitation: The theory covers rare, arbitrary facts most clearly. Other kinds of hallucination may have other causes.
Assessment: SupportedSaid
Gary Marcus Forecast
“LLMs are well on their way to becoming a commodity; price wars will be intense.”
Our reading: LLMs are becoming a commodity and price wars among providers will be intense.
Settled before its deadline Evidence as of
Limitation: Commoditization is assessed as a direction of travel, consistent with well on their way.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: SupportedSaid
Gary Marcus Observation
“Adding visual data to LLMs hasn’t solved hallucinations.”
Our reading: Adding visual data to LLMs has not solved hallucinations.
No horizon Evidence as of
Limitation: 'Solved' is not defined. The claim is read as hallucinations remaining a material problem, not as adding vision bringing no improvement at all. The evidence packet notes that the GPT-4V system card reports fewer hallucinations during a beta, but that passage is not in the cited excerpt.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Gary Marcus Observation
“I also urged the world to focus on global AI governance, and am pleased to see some genuine progress in that direction, especially led by the UN and the Council of Europe.”
Our reading: There has been genuine progress on global AI governance, led especially by the UN and the Council of Europe.
No horizon Evidence as of
Limitation: Institutional progress is distinct from demonstrated practical effectiveness.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Gary Marcus Observation
“Yejin Choi worried that LLMs were not reliable. They still aren’t.”
Our reading: LLMs remained unreliable as of April 2024, a year after Choi's TED talk.
No horizon Evidence as of
Limitation: The finding concerns general factual dependability, not every task or application.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Gary Marcus Causal claim
“The Humane AI pin/virtual assistant just doesn’t work well, and a lot of the problem is the underlying AI, which I believe is powered by OpenAI.”
Our reading: The Humane AI pin just doesn't work well, and a lot of the problem is its underlying AI, which he believes is powered by OpenAI.
No horizon Evidence as of
Limitation: The causal conclusion is qualitative and rests on documented product behavior, not a controlled component analysis.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Gary Marcus Observation
“Eliezer Yudkowsky warned that AI was (inevitably!) going to kill us all. I am happy to report that so far it hasn’t.”
Our reading: So far, AI has not killed humanity as Yudkowsky warned.
No horizon Evidence as of
Limitation: This assesses neither future AI risk nor Marcus’s characterization of Yudkowsky’s position.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Ed Zitron Conditional forecast
“As I wrote (and broadcast) recently, LLMs are running out of training data, and the desperation to find more will lead to significant legal consequences when it's revealed exactly how much they stole and from whom they stole it from.”
Our reading: The hunt for more training data will bring AI companies significant legal consequences once it is revealed how much they stole and from whom.
Settled before its deadline Evidence as of
Only if: when it's revealed how much they stole and from whom
Source: They're Looting The Internet
Assessment: SupportedSaid
Gary Marcus Observation
“What it actually shows is some improvements, mostly modest, over the last few months for a bunch of different measures.”
Our reading: OpenAI's GPT-4 Turbo graph shows only mostly modest improvements over recent months.
No horizon Evidence as of
Limitation: Modest is a qualitative judgment, and relative changes can look larger.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“I think it’s fair to say that they have not. GPT-Turbo released this week still produces its share of groaners, like this interchange Phil Libin just sent:”
Our reading: Hallucinations and stupid errors in LLMs have not been solved; GPT-4 Turbo still makes them.
No horizon Evidence as of
Limitation: The coding test measures task completion, not hallucination frequency.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“Already in recent weeks, Inflection AI largely closed shop, Stability AI is struggling, and LLM-based autonomous vehicle company called Ghost closed shop, and a software engineer on YouTube raised quite serious questions about the wildly-hyped AI coding system Devin.”
Our reading: In recent weeks Inflection AI largely closed, Stability AI is struggling, Ghost closed, and serious questions were raised about Devin.
No horizon Evidence as of
Limitation: The Inflection finding depends on the meaning of 'largely'; the company continued operating.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns.”
Our reading: MMLU gains from GPT-4 to GPT-4 Turbo are small compared with earlier generational jumps, tentatively evidencing diminishing returns.
No horizon Evidence as of
Limitation: The records do not compare all four models under one protocol: GPT-2 was fine-tuned, GPT-3 used few-shot prompting, GPT-4 used five-shot prompting, and Turbo used zero-shot chain-of-thought prompting [e1, e2, e3].
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Forecast
“Skeptical of this, I offered Musk a $1 million bet that AGI superior to the smartest individual humans would not arrive by the end of 2025, reprinted here for those who missed it.”
Our reading: AGI superior to the smartest individual humans will not arrive by the end of 2025.
Deadline passed Evidence as of
Limitation: The evidence does not inventory undisclosed systems or every system and inference configuration available by the deadline.
Source: Superhuman AGI is not nigh
Assessment: SupportedSaid
Gary Marcus Observation
“Tesla FSD is getting better. But is getting enough better?”
Our reading: Tesla FSD is getting better, and he raises the question of whether it is getting enough better.
No horizon Evidence as of
Limitation: The tracker figures in e1 come from crowdsourced, self-reported data. e1 notes that data for later v12.3.x releases still needed weeks to be entered.
Source: 24 hours in AI news
Assessment: SupportedSaid
Gary Marcus Observation
“current models aren’t literally look up tables; they can generalize to some degree, but not enough.”
Our reading: Current AI models can generalize to some degree but not enough.
No horizon Evidence as of
Limitation: The outcome depends on interpreting 'not enough' through the surrounding discussion of reliability.
Assessment: SupportedSaid
Gary Marcus Observation
“Neural networks (at least in the configurations that have been dominant over the last three decades) have trouble generalizing beyond the multidimensional space that surrounds their training examples.”
Our reading: Neural networks in the configurations dominant for the last three decades have trouble generalizing beyond the space surrounding their training examples.
No horizon Evidence as of
Limitation: The outcome depends on reading the spatial language as unfamiliar conditions rather than a precise geometric boundary.
Assessment: SupportedSaid
Gary Marcus Observation
“The good news is that word is finally getting out more broadly.”
Our reading: Awareness of the danger generative AI poses to the information ecosystem is finally spreading more widely.
No horizon Evidence as of
Limitation: The trend ('more broadly') is inferred from high levels, not from a measured before-and-after comparison.
Assessment: SupportedSaid
Gary Marcus Observation
“Meanwhile X itself is promoting nonsense.”
Our reading: X itself is promoting nonsense.
No horizon Evidence as of
Limitation: The assessment relies on direct news reporting with screenshots and an independent fact-check; no separately retrievable original X display was obtained.
Assessment: SupportedSaid
Gary Marcus Causal claim
“Ironic that his own chatbot, Grok, is making the problem worse.”
Our reading: Musk's own chatbot, Grok, is making the information-pollution problem worse.
No horizon Evidence as of
Limitation: The false Iran story began with user posts; Grok's added effect on spread is not separately measured.
Assessment: SupportedSaid
Gary Marcus Observation
“The bad news, which outweighs the good, is that the problem is quickly getting worse.”
Our reading: Generative-AI pollution of the information ecosystem is getting worse quickly as of April 2024.
No horizon Evidence as of
Limitation: Tracker counts come from one commercial source and may partly reflect more searching rather than only more sites.
Assessment: SupportedSaid
Gary Marcus Observation
“The trouble is that the bigger the model, the higher the internal costs. And current-sized models, already expensive, aren’t reliable enough for many jobs.”
Our reading: Bigger models have higher costs, and current-sized models are already expensive and not reliable enough for many jobs.
No horizon Evidence as of
Limitation: The result depends on comparing otherwise similar implementations and interpreting reliability as suitability for unverified use.
Assessment: SupportedSaid
Gary Marcus Observation
“As noted here before, one study indicates that code quality is going down.”
Our reading: One study indicates code quality is going down amid AI-assisted coding.
No horizon Evidence as of
Limitation: The study uses indirect quality indicators.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Gary Marcus Observation
“On the other hand, the use cases on the negative side of the ledger are proliferating fast, literally faster than I can report them here, from fake books (undermining authors) to rings of fake websites designed to sell ads”
Our reading: Harmful uses of generative AI (fake books, ad-farm websites, election misinformation, polluted science) are proliferating rapidly.
No horizon Evidence as of
Limitation: Support depends on reading proliferation as broad expansion rather than proven rapid growth in every category.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Gary Marcus Observation
“Some core problems have lingered, particularly hallucinations, instability and occasional nonsensical errors.”
Our reading: Hallucinations, instability and occasional nonsensical errors remain unsolved in GPT-4 class LLMs.
No horizon Evidence as of
Limitation: The research tests selected tasks rather than representative everyday usage.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Gary Marcus Forecast
“We can expect price wars.”
Our reading: There will be price wars among LLM providers.
Settled before its deadline Evidence as of
Limitation: The decisive episode occurred in China; the claim supplied no geographic restriction.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Gary Marcus Observation
“There has however been lots of progress both in discovering potential applications and in putting GPT-4 class models into practice.”
Our reading: Over the past year there has been substantial progress in finding applications for and deploying GPT-4 class models.
No horizon Evidence as of
Limitation: The sources are reports from the deploying companies.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Gary Marcus Observation
“Translation: Using a well-known algorithm that we didn’t invent to decide which faces of the cube to turn, we taught a robot to do motor control on a specially instrumented Rubik’s cube with bluetooth sensors inside.”
Our reading: In practice, OpenAI's 2019 claim of 'solving a Rubik's cube with one hand' meant using a well-known algorithm OpenAI didn't invent to choose which faces to turn, and teaching a robot motor control on a specially instrumented Rubik's cube with Bluetooth sensors inside.
No horizon Evidence as of
Limitation: Support applies to the stated use of instrumented hardware, not exclusive reliance on it.
Source: OpenAI’s Lies and Half-Truths
Assessment: SupportedSaid
Gary Marcus Observation
“Translation: But (revealed a few weeks later) behind the scenes we will try hard to water down your regulations.”
Our reading: While publicly asking the Senate to regulate it, OpenAI worked behind the scenes to water down regulations.
No horizon Evidence as of
Limitation: The documented request concerned the EU AI Act; it does not establish efforts to weaken a U.S. Senate proposal [e2].
Source: OpenAI’s Lies and Half-Truths
Assessment: SupportedSaid
Gary Marcus Observation
“GenAI is messing with science.”
Our reading: Generative AI is currently damaging science.
No horizon Evidence as of
Limitation: The conclusion concerns an identifiable publication failure, not the frequency or aggregate consequences of such failures.
Source: The Year of the Big-Balled Rat
Assessment: SupportedSaid
Ed Zitron Observation
“Generative AI is a perfect tool for soullessly churning out content to match a particular set of instructions — such as those that an algorithm follows — and while an algorithm can theoretically be tuned to evaluate content as "human," so can scaled content be tweaked to make it seem more human.”
Our reading: Generative AI is a perfect tool for soullessly churning out content to match a set of instructions such as an algorithm's, and while an algorithm can theoretically be tuned to evaluate content as human, scaled content can likewise be tweaked to seem more human.
No horizon Evidence as of
Limitation: The result depends on reading perfect as rhetorical suitability.
Assessment: SupportedSaid
Ed Zitron Observation
“The algorithms are easily-tricked, and the tools used to trick them are becoming easier to use and scale.”
Our reading: Platform algorithms are easily tricked, and the tools used to trick them, including generative AI, are becoming easier to use and scale.
No horizon Evidence as of
Limitation: Support depends on reading the statement as a qualitative description of documented vulnerabilities.
Assessment: SupportedSaid
Ed Zitron Forecast
“Yet at some point these portals will generate an answer based off of the data they have (or have access to, in the case of Tumblr and Wordpress) rather than linking you to a place where you can find an answer by reading something created by another person.”
Our reading: At some point search portals will give AI-generated answers from their data instead of linking users to content made by people.
Settled before its deadline Evidence as of
Limitation: The result depends on interpreting replacement as replacing a necessary source visit, not abolishing links.
Assessment: SupportedSaid
Gary Marcus Observation
““Current deep-learning systems frequently succumb to stupid errors.” Still true.”
Our reading: Current deep-learning systems still often make stupid errors.
No horizon Evidence as of
Limitation: The finding depends on reading frequent as recurring in ordinary-task evaluations, rather than as a population-wide frequency threshold.
Source: Two years later, deep learning is still faced with the same fundamental challenges
Assessment: SupportedSaid
Gary Marcus Observation
““new effort by OpenAI to solve these problems wound [still] fabricat[ing] authoritative nonsense ” Still true.”
Our reading: OpenAI's new effort to solve these problems still ends up fabricating authoritative nonsense.
No horizon Evidence as of
Limitation: This example does not establish how frequently such errors occurred across OpenAI products.
Source: Two years later, deep learning is still faced with the same fundamental challenges
Assessment: SupportedSaid
Gary Marcus Observation
“And the apparent copyright infringements that Reid Southen and I discussed in early January continue apace two months later, as I found in a quick experiment this morning:”
Our reading: The apparent copyright infringements that Marcus and Reid Southen discussed in early January 2024 were continuing apace two months later, based on a quick experiment Marcus ran on the morning of the post.
No horizon Evidence as of
Limitation: The result depends on reading 'apace' as continued ready production rather than a quantitative rate claim.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Gary Marcus Observation
“Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has trained on.”
Our reading: Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has been trained on.
No horizon Evidence as of
Limitation: Official nondisclosure does not prove that outsiders knew nothing.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Gary Marcus Observation
“They knew they were misleading people and failed to take an opportunity to correct.”
Our reading: OpenAI knew it was misleading people into believing it meant open-sourced AI and failed to take an opportunity, at a May 2016 meeting, to correct that belief.
No horizon Evidence as of
Limitation: The supplied evidence consists of one reported account; no independent recording, transcript, or meeting minutes are supplied.
Source: OpenAI’s “Own Goal”