Supported claims, page 4 of 8
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Supported claims (775)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 4 of 8: claims 301 to 400 of 775, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Gary Marcus Observation
“• AGI should have reliable ability to reason over abstractions. (Ongoing unpredictable errors in LLMs reflect this.)”
Our reading: LLMs still lack a reliable ability to reason over abstractions, as their ongoing unpredictable errors show.
No horizon Evidence as of
Limitation: The February studies are preprints and do not provide complete execution timestamps.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“Most parts of the US still don’t have functioning driverless cars systems, for example, even after over a decade of lavish promises.”
Our reading: Most parts of the US still lack functioning driverless car systems after more than a decade of lavish promises.
No horizon Evidence as of
Limitation: Geographic coverage is assessed qualitatively rather than through a national area or population calculation.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“But it is absolutely absurd to pretend “AGI imminent” hypothesis isn’t getting play in high places.”
Our reading: The hypothesis that AGI is imminent is getting play in high places, and it is absurd to pretend otherwise.
No horizon Evidence as of
Limitation: This supports the general observation, not every example or institutional explanation in the surrounding post.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“Every red cell in this graph (e.g, multiplying 20 digit numbers by 19 digit numbers2) is failure to generalize, even after immense amounts of training data:”
Our reading: LLMs including o1-mini fail to generalize on large multi-digit multiplication despite immense training data.
No horizon Evidence as of
Limitation: The retrieved text does not permit a cell-by-cell audit, including the specific 20-by-19-digit example.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“A new study of the latest system, GPT 4.5 showed that by on one standardized measure over 1/3 of the outputs are still hallucinations.”
Our reading: On one standardized measure, over a third of GPT-4.5's outputs are hallucinations.
No horizon Evidence as of
Limitation: The exact percentage is supported by Computerworld's report of OpenAI's evaluation, not a readable primary numeric table.
Assessment: SupportedSaid
Gary Marcus Observation
“Already, of course, DeepSeek has shown that our once strong lead is diminishing.”
Our reading: DeepSeek has shown that the US's once-strong lead is shrinking.
No horizon Evidence as of
Limitation: The report was published after the claim, although its measurements cover the relevant earlier period.
Assessment: SupportedSaid
Gary Marcus Observation
“Here Roose has displayed the cardinal sin of looking only at regurgitation rather than generalization. History has told us over and over that systems can be good at the former and still face enormous challenges at the latter.”
Our reading: AI systems can be good at regurgitating known code while still facing enormous challenges generalizing to new problems.
No horizon Evidence as of
Limitation: Training similarity and robustness failures do not by themselves prove verbatim memorization.
Assessment: SupportedSaid
Gary Marcus Observation
“Scaling hasn’t gotten them —or anyone else — to AGI.”
Our reading: Scaling has not brought OpenAI or anyone else to AGI.
No horizon Evidence as of
Limitation: The result depends on the stated meaning of AGI.
Source: OpenAI, in deep trouble
Assessment: SupportedSaid
Gary Marcus Observation
“And of course, there is still no GPT-5, even after two years, of hype, bullshit, and cryptic hints.”
Our reading: OpenAI still has not released GPT-5 after two years of hype.
No horizon Evidence as of
Limitation: The assessment concerns public availability, not internal prototypes.
Source: OpenAI, in deep trouble
Assessment: SupportedSaid
Gary Marcus Observation
“Some have started serious competitors with similar IP.”
Our reading: Some departed OpenAI staff have founded serious competitors with similar IP.
No horizon Evidence as of
Limitation: The interpretation of similar IP materially affects the result.
Source: OpenAI, in deep trouble
Assessment: SupportedSaid
Gary Marcus Observation
“GPT 4.5 is hugely expensive.”
Our reading: GPT-4.5 is very expensive.
No horizon Evidence as of
Limitation: The conclusion concerns customer API prices, not OpenAI's internal costs.
Source: OpenAI, in deep trouble
Assessment: SupportedSaid
Gary Marcus Observation
“So now GPT 4.5 has dropped, and — breaking news — I was right. Hallucinations didn’t disappear today, and nor did stupid errors.”
Our reading: GPT-4.5 did not eliminate hallucinations or stupid errors.
No horizon Evidence as of
Limitation: The evidence is developer-reported.
Assessment: SupportedSaid
Gary Marcus Forecast
“Actually, at any moment, we will finally get GPT 4.5, which most certainly wouldn’t live up to expectations around GPT-5 — hence its rather tentative name.”
Our reading: GPT 4.5 will not live up to the expectations that had been set for GPT-5.
Settled before its deadline Evidence as of
Limitation: The decisive measurements are company-reported.
Source: GPT 4.5 is no GPT-5
Assessment: SupportedSaid
Gary Marcus Observation
“In “Deep learning is hitting a wall”, I warned that problems with reasoning and hallucination wouldn’t go away simply as a function of scaling, and they haven’t.”
Our reading: Problems with reasoning and hallucination in deep learning systems have not gone away as a function of scaling.
No horizon Evidence as of
Limitation: Humanity's Last Exam combines knowledge and reasoning; not every wrong answer identifies a reasoning failure.
Source: GPT 4.5 is no GPT-5
Assessment: SupportedSaid
Gary Marcus Observation
“Hallucinations haven’t been magically solved, etc.”
Our reading: Grok 3 has not solved hallucinations.
No horizon Evidence as of
Limitation: The decisive evidence is an identified researcher's informal testing, not a controlled benchmark.
Source: Grok 3 Hot Take
Assessment: SupportedSaid
Gary Marcus Observation
“That wasn’t tested (presumably they don’t have access), and only a handful of benchmarks were actually reported.”
Our reading: The Grok 3 launch did not test against full o3 and reported only a handful of benchmarks.
No horizon Evidence as of
Limitation: The complete primary launch charts were not readable, so the evidence does not establish an exhaustive benchmark count.
Source: Grok 3 Hot Take
Assessment: SupportedSaid
Gary Marcus Observation
“That said, OpenAI’s moat keeps diminishing, so price wars will continue and profits will continue to be elusive for everyone except Nvidia.”
Our reading: OpenAI's competitive moat keeps shrinking.
No horizon Evidence as of
Limitation: The outcome depends on interpreting moat as the capability advantage discussed in the post.
Source: Grok 3 Hot Take
Assessment: SupportedSaid
Gary Marcus Forecast
“That said, OpenAI’s moat keeps diminishing, so price wars will continue and profits will continue to be elusive for everyone except Nvidia.”
Our reading: Because OpenAI's moat keeps shrinking, price wars among AI model providers will continue.
Settled before its deadline Evidence as of
Limitation: The causal evidence is reporting and analyst assessment, not a controlled economic study.
Source: Grok 3 Hot Take
Assessment: SupportedSaid
Gary Marcus Observation
“Simply false, in view of the many basic aspects of geometry that AlphaGeometry2 does not cover, as discussed above.”
Our reading: The claim that AlphaGeometry2 'just figured out geometry' is false because it does not cover many basic aspects of geometry.
No horizon Evidence as of
Limitation: Support depends on reading figured out geometry as general mastery.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: SupportedSaid
Gary Marcus Observation
“Unlike DeepMind’s AlphaZero system which learned to play multiple board games, or OpenAI’s ChatGPT, AlphaGeometry2 is nowhere close to being a single general-purpose learning architecture.”
Our reading: AlphaGeometry2 is a specialized system, nowhere close to a single general-purpose learning architecture.
No horizon Evidence as of
Limitation: The documentation became available after Marcus’s February 17 post.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: SupportedSaid
Ed Zitron Observation
“OpenAI burned more than $5 billion last year.”
Our reading: OpenAI burned more than $5 billion in 2024.
No horizon Evidence as of
Limitation: The result depends on reading burn as net loss, as suggested by the surrounding discussion.
Source: The Generative AI Con
Assessment: SupportedSaid
Ed Zitron Observation
“User numbers alone tell you nothing about the sustainability or profitability of a business, or how those people use the product.”
Our reading: User numbers alone tell you nothing about a business's sustainability or profitability, or about how its users use the product.
No horizon Evidence as of
Limitation: The result depends on reading 'nothing' as insufficient to establish the conclusions, rather than literally no information.
Source: The Generative AI Con
Assessment: SupportedSaid
Gary Marcus Observation
“About year ago the City of New York tried this kind of thing, with less than spectacular results:”
Our reading: About a year ago the City of New York tried this kind of thing (deploying problematic, unreliable AI to the public), with less than spectacular results.
No horizon Evidence as of
Limitation: The documented tests do not establish a representative overall error rate.
Assessment: SupportedSaid
Gary Marcus Forecast
“Second, pure scaling will no longer be the means of attack.”
Our reading: OpenAI will no longer rely on pure scaling as its means of building GPT-5.
Settled before its deadline Evidence as of
Limitation: The disclosure establishes the additional mechanisms, not their precise share of capability gains.
Source: Breaking: OpenAI's efforts at pure scaling have hit a wall.
Assessment: SupportedSaid
Gary Marcus Observation
“Even the latest systems like Deep Research are still struggling in a few ways – and those ways pretty much correspond exactly to the places that I warned would be LLM’s Achilles’ Heels: hallucinations and reasoning errors.”
Our reading: The latest systems such as OpenAI's Deep Research still suffer from hallucinations and reasoning errors.
No horizon Evidence as of
Limitation: Supports the existence of both error classes, not any particular frequency or severity.
Assessment: SupportedSaid
Gary Marcus Observation
“To some extent newer models are doing that.”
Our reading: Newer models such as DeepSeek R1, with a rule-based reward system, partly use neurosymbolic techniques.
No horizon Evidence as of
Limitation: The outcome depends on the broad, training-inclusive meaning supplied by the speaker's example.
Assessment: SupportedSaid
Gary Marcus Observation
“Me, December 2022: The problem with LLMs is that they hallucinate, and their errors can be hard to catch.”
Our reading: As of December 2022, LLMs hallucinated and their errors could be hard to catch.
No horizon Evidence as of
Limitation: This single ChatGPT demonstration does not establish hallucination prevalence or average detection difficulty across models and users.
Source: Shame on Google, twice
Assessment: SupportedSaid
Gary Marcus Observation
“And that’s just not realistic given the current state of AI.”
Our reading: Current AI cannot reliably get synthesis of online information and multi-step research right.
No horizon Evidence as of
Limitation: The evidence directly concerns Deep Research, not every contemporary AI system.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Gary Marcus Observation
“The basic conjecture then — which has already been confirmed to some degree – was that LLMs would be used to write garbage content undermining the value of the internet itself.”
Our reading: His 2023 conjecture that LLMs would be used to write garbage content undermining the internet's value has already been confirmed to some degree.
No horizon Evidence as of
Limitation: The Wikipedia study was a preprint and relied on imperfect detectors.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Gary Marcus Observation
“Sam Altman just released a fancier remake Deep Research, which, like Meta’s predecessor, can write science-sounding articles on demand, on any topic of your choice.”
Our reading: OpenAI's Deep Research is essentially a fancier remake of Meta's Galactica that can write science-sounding articles on any topic on demand.
No horizon Evidence as of
Limitation: The evidence supports a shared writing function, not technical equivalence or equal performance.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Gary Marcus Observation
“The real lesson of DeepSeek is that we can still make vast improvements in efficency.”
Our reading: DeepSeek shows that vast efficiency improvements in AI are still possible.
No horizon Evidence as of
Limitation: The result depends on interpreting efficiency as computing resources at comparable benchmark capability.
Assessment: SupportedSaid
Ed Zitron Observation
“DeepSeek's models don't require massive new data centers (they run on the GPUs currently used to run services like ChatGPT, and can even work on more austere hardware), nor do they require an endless supply of bigger, faster NVIDIA GPUs every year to progress.”
Our reading: DeepSeek's models do not require massive new data centers or ever-bigger NVIDIA GPUs to progress.
No horizon Evidence as of
Limitation: V3's training hardware is company-reported; the report does not establish the facility's construction history [e8].
Source: Deep Impact
Assessment: SupportedSaid
Ed Zitron Forecast
“One thing we do know is that OpenAI and Anthropic will now have to either drop the price of accessing their models, and potentially even the cost of their subscriptions.”
Our reading: OpenAI and Anthropic will have to cut API prices, and possibly subscription prices.
Settled before its deadline Evidence as of
Limitation: Support depends on including lower-priced successors within a model family.
Source: Deep Impact
Assessment: SupportedSaid
Ed Zitron Forecast
“Whether or not someone chooses to pay DeepSeek is largely irrelevant — someone else will take what it’s created and build their own, or people will start running their own DeepSeek instances renting GPUs from one of the various cloud computing firms.”
Our reading: Others will build their own models from DeepSeek's work or run their own DeepSeek instances on rented cloud GPUs.
Settled before its deadline Evidence as of
Limitation: The derivative starts from DeepSeek's distilled Qwen models rather than the full R1 model.
Source: Deep Impact
Assessment: SupportedSaid
Gary Marcus Observation
“DeepSeek r1 is not smarter than earlier models, just trained more cheaply”
Our reading: DeepSeek r1 is not smarter than earlier models.
No horizon Evidence as of
Limitation: The result depends on interpreting earlier models as the preceding frontier.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: SupportedSaid
Gary Marcus Observation
“It doesn't solve hallucinations or problems with reliability.”
Our reading: DeepSeek r1 does not solve hallucinations or reliability problems.
No horizon Evidence as of
Limitation: The hallucination example comes from a reported reasoning trace, rather than a systematic release-period benchmark.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: SupportedSaid
Gary Marcus Observation
“This morning, Nvidia is down more than 10% in pretrading.”
Our reading: On the morning of 2025-01-27, Nvidia stock was down more than 10% in pre-market trading.
No horizon Evidence as of
Limitation: The report gives a rounded percentage rather than underlying trade prices.
Assessment: SupportedSaid
Gary Marcus Forecast
“None of this means however that China won the AI race or even took the lead. American companies will incorporate these new results and continue to produce new results of their own.”
Our reading: The recent Chinese results do not mean China has won the AI race or even taken the lead; American companies will incorporate these results and keep producing new results of their own.
Settled before its deadline Evidence as of
Limitation: The decisive examples involve model checkpoints and generated training data, rather than proof that every named technique was adopted.
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: SupportedSaid
Gary Marcus Observation
“On LLMs China is already catching up.”
Our reading: China is already catching up with the U.S. on LLMs.
No horizon Evidence as of
Limitation: Benchmark performance does not capture every aspect of real-world reliability.
Source: Hot take on an AI catfight
Assessment: SupportedSaid
Gary Marcus Observation
“Today he walked it back.”
Our reading: On 2025-01-20 Sam Altman walked back his earlier hint that OpenAI was close to building AGI.
No horizon Evidence as of
Limitation: Relies on news reports quoting the X post, not the post itself.
Assessment: SupportedSaid
Gary Marcus Observation
“That is in part because you can’t build reliable agents unless you can solve the hallucination problem, and there is still no principled trustworthy answer to that problem.”
Our reading: As of January 2025 there is no principled, trustworthy solution to the hallucination problem.
No horizon Evidence as of
Limitation: 'Principled trustworthy answer' has no stated threshold. The assessment uses the contract's reading: a method that reliably prevents hallucinations in general use.
Assessment: SupportedSaid
Gary Marcus Forecast
“The thing about scheduling is that it needs to work reliably. And it won’t.”
Our reading: OpenAI's ChatGPT task-scheduling agent will not work reliably.
Settled before its deadline Evidence as of
Limitation: The evidence is anecdotal. There is no measured failure rate.
Assessment: SupportedSaid
Gary Marcus Observation
“Despite the daily shouts of “exponential progress” for nearly two years, GPT-5 hasn’t come.”
Our reading: As of January 2025, GPT-5 has not been released despite nearly two years of claims of exponential progress.
No horizon Evidence as of
Limitation: The in-period evidence is a news summary of a paywalled WSJ report.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Gary Marcus Observation
“But it’s becoming increasingly clear that they are struggling in building a model worthy of the GPT-5 name.”
Our reading: OpenAI is struggling to build a model worthy of the GPT-5 name.
No horizon Evidence as of
Limitation: Rests on anonymous-source news reports summarized by secondary outlets.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Gary Marcus Observation
“The hypothesis I have been floating since 2022 that scaling data alone would not be enough for robust AI has become increasingly prevalent.”
Our reading: Marcus's hypothesis, which he has been floating since 2022, that scaling data alone would not be enough for robust AI has become increasingly prevalent.
No horizon Evidence as of
Limitation: The evidence does not measure prevalence in a defined population or establish majority agreement.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Gary Marcus Forecast
“No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”
Our reading: The next generation of AI systems will have greater abilities and broader applications and will somewhat mitigate current weaknesses.
Settled before its deadline Evidence as of
Limitation: Support uses the ordinary directional reading, not a claim that every successor or application improved.
Source: The Five Stages of AGI Grief
Assessment: SupportedSaid
Gary Marcus Observation
“While such a milestone would be important economically and socially, it is also a departure from, and even a retrenchment from, the kind of cognitive criteria around flexibility and resourcefulness that people in the field had generally talked about it.”
Our reading: A milestone of a system doing 80% of economically valuable human work, which Marcus recalls (not with certainty) OpenAI suggesting as a definition of AGI, would be important economically and socially, but it departs from, and even retreats from, the cognitive criteria of flexibility and resourcefulness that people in the field had generally discussed.
No horizon Evidence as of
Limitation: Support applies to the comparison of stated requirements.
Source: The Five Stages of AGI Grief
Assessment: SupportedSaid
Gary Marcus Observation
“The problem of distribution shift, well-tested in Apple’s 2024 reasoning paper, which goes back to my own 1998 work, continues.”
Our reading: The problem of distribution shift in neural models continues to be unsolved.
No horizon Evidence as of
Limitation: Support concerns continued empirical failures, not a proof that neural models cannot generalize.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Gary Marcus Observation
“Commonsense reasoning is still dodgy, as it has long been, as Ernie Davis and I just reviewed here..”
Our reading: Commonsense reasoning in current AI systems remains unreliable, as it long has been.
No horizon Evidence as of
Limitation: Commonsense is broader than the tasks tested.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Gary Marcus Observation
“LLMs, like their predecessors generalize well to similar items but struggle with reliability in unfamiliar regimes, even minor variations sometimes break them.”
Our reading: LLMs generalize well to similar items but are unreliable in unfamiliar regimes, and minor variations sometimes break them.
No horizon Evidence as of
Limitation: The cited evidence concerns mathematical word problems, not every application of LLMs.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Gary Marcus Observation
“Many leading figures in the field have acknowledged that we may have reached a period of diminishing returns of pure LLM scaling, much as I anticipated in 2022.”
Our reading: Many leading figures in the field have acknowledged that pure LLM scaling may have reached a period of diminishing returns, much as Marcus anticipated in 2022.
No horizon Evidence as of
Limitation: The quantity 'many' remains interpretive.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Gary Marcus Forecast
“They will be a major focus of attention in the field in 2025.”
Our reading: AI agents will be a major focus of attention in the AI field in 2025.
Deadline passed Evidence as of
Limitation: 'Major focus' has no numeric threshold. The evidence shows agents were prominent but does not rank them against other themes in the field.
Assessment: SupportedSaid
Gary Marcus Observation
“But what was most striking to us was not the unsurprising fact that these systems are not close to ready for prime time (an employee that bad would probably be immediately fired), but the examples from the list of what went wrong:”
Our reading: Current LLM agents (Claude, Gemini, GPT-4 etc.) are not close to ready for prime time on real-world workplace tasks.
No horizon Evidence as of
Limitation: 'Ready for prime time' has no explicit threshold and is a judgment.
Assessment: SupportedSaid
Gary Marcus Observation
“Ten months later, Sora is finally available to the public, and it’s apparent that it too is struggling with physical reasoning.”
Our reading: Sora, now publicly available, struggles with physical reasoning.
No horizon Evidence as of
Limitation: The announcement does not quantify the physical-error rate or independently reproduce the examples in the post.
Assessment: SupportedSaid
Gary Marcus Observation
“But one important component of physical and conceptual understanding is reasoning about entities and their properties, and current approaches have consistently fallen short on this.”
Our reading: Current AI approaches have consistently fallen short on reasoning about entities and their properties.
No horizon Evidence as of
Limitation: This outcome depends on reading consistently as recurring gaps, not failure on every task.
Assessment: SupportedSaid
Gary Marcus Observation
“Depressingly, despite all of the apparent progress in the last few years, much of what we raised in 2015 and re-emphasized in 2019 remains unsolved.”
Our reading: Much of the commonsense reasoning problem the authors raised in 2015 and 2019 remains unsolved in AI as of January 2025.
No horizon Evidence as of
Limitation: The result depends on interpreting 'unsolved' as lacking broad reliability.
Assessment: SupportedSaid
Gary Marcus Observation
“If general purpose domestic robots are a long way away, “AI agents” —AI systems that act in the world on user’s behalf—are already here to a limited degree.”
Our reading: AI agents, meaning AI systems that act in the world on users' behalf, are already here to a limited degree, even though general-purpose domestic robots are a long way away.
No horizon Evidence as of
Limitation: This establishes limited availability and functionality, not general reliability or widespread adoption.
Assessment: SupportedSaid
Gary Marcus Observation
“Hoping that reliable commonsense will simply “emerge” from large databases and video has proven unrealistic, again and again.”
Our reading: The hope that reliable commonsense will emerge from large text and video datasets has repeatedly proven unrealistic.
No horizon Evidence as of
Limitation: Reliable commonsense remains an interpretive standard.
Assessment: SupportedSaid
Gary Marcus Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: Throughout 2025, AI agents will be endlessly hyped but far from reliable, except possibly in very narrow use cases.
Deadline passed Evidence as of
Limitation: The promotion shown comes from one company, OpenAI. It does not measure how much agents were promoted across the industry.
Assessment: SupportedSaid
Gary Marcus Forecast
“Copyright lawsuits over generative AI will continue throughout the year.”
Our reading: Copyright lawsuits over generative AI will continue throughout 2025.
Deadline passed Evidence as of
Limitation: The records establish continuing litigation, not the merits of infringement allegations.
Assessment: SupportedSaid
Gary Marcus Forecast
“Reasoning flubs will continue to haunt generative AI.”
Our reading: Through 2025 reasoning errors will continue to plague generative AI.
Deadline passed Evidence as of
Limitation: 'Flubs' and 'haunt' set no threshold, so the verdict rests on a plain reading.
Assessment: SupportedSaid
Gary Marcus Observation
“Immediately after the February announcement of Sora, I noted Sora’s troubles with physics and predicted that they wouldn’t go away. They haven’t.”
Our reading: Sora's troubles with physics have not gone away since its February 2024 announcement.
No horizon Evidence as of
Limitation: The disclosures do not quantify failure rates or establish that physics performance had not improved [e1, e2].
Assessment: SupportedSaid
Gary Marcus Observation
“My prediction that 2024 would be seen as the year of AI disillusionment was overstated but not entirely off the mark, holding true for some folks but not all, and as such perhaps deserve partial credit.”
Our reading: 2024 was a year of AI disillusionment for some but not all; his prediction was overstated.
No horizon Evidence as of
Limitation: The surveys cover business leaders and employees rather than a representative global public.
Assessment: SupportedSaid
Gary Marcus Observation
“There is still no robust solution to hallucinations.”
Our reading: There is still no robust solution to LLM hallucinations.
No horizon Evidence as of
Limitation: The main source is a lab paper describing its own models' limits. Independent leaderboard figures for late 2024 could not be read.
Assessment: SupportedSaid
Gary Marcus Forecast
“Hallucinations (which should really be called confabulations) will continue to haunt generative AI.”
Our reading: Through 2025 hallucinations will continue to plague generative AI.
Deadline passed Evidence as of
Limitation: 'Haunt' has no threshold. The verdict rests on the plain reading that the problem stayed material and recognized.
Assessment: SupportedSaid
Gary Marcus Observation
“There are many such models now; there is a price war. There is very little moat.”
Our reading: There are many GPT-4 level models, a price war, and very little moat among LLM providers.
No horizon Evidence as of
Limitation: 'Moat' is read as a lasting performance lead. Brand, distribution or user base, which are other possible moats, were not measured.
Assessment: SupportedSaid
Gary Marcus Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: AI agents will be heavily hyped in 2025 but far from reliable except possibly in very narrow use cases.
Deadline passed Evidence as of
Limitation: The result partly depends on reading coding agents as a 'narrow use case'. A reader who treats coding as broad could call this mixed.
Assessment: SupportedSaid
Gary Marcus Forecast
“OpenAI will continue to preview products months and perhaps years before they are solid and widely available at an accessible price.”
Our reading: By the end of 2025 OpenAI will have continued to preview products months, and perhaps years, before they are solid and widely available at an accessible price.
Deadline passed Evidence as of
Limitation: The result depends on treating premium Pro-only previews as short of ordinary broad consumer access.
Assessment: SupportedSaid
Gary Marcus Observation
“At *most* the test is necessary for AGI; it certainly isn’t sufficient.”
Our reading: Passing ARC is at most necessary for AGI and certainly not sufficient.
No horizon Evidence as of
Limitation: The contract notes that AGI is not defined, so 'sufficient' is judged by the designers' stated view.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Gary Marcus Observation
“What was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.”
Our reading: What was done for o3 on ARC, pretraining on what Marcus believes was hundreds of public examples, is not comparable to what humans require.
No horizon Evidence as of
Limitation: The conclusion depends on reading comparable as referring to ARC-specific preparation.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Gary Marcus Observation
“We still don’t have a solid test of what o3 does without the pretraining, in the case that would be more comparable to humans”
Our reading: There is no solid test yet of o3's ARC performance without pretraining.
No horizon Evidence as of
Limitation: The evaluator page has later updates, and the training note's precise insertion date is not separately archived in this record.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Gary Marcus Observation
“I am saying that many basic problems haven’t been solved.”
Our reading: Many basic problems on the path to AGI remain unsolved.
No horizon Evidence as of
Limitation: The scope and meaning of 'many' are inferred and materially affect the assessment.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Gary Marcus Observation
“The product was not yet released to the general public; it’s not been vetted by the scientific community.”
Our reading: As of the demo, o3 had not been released to the public nor vetted by the scientific community.
No horizon Evidence as of
Limitation: 'Vetted by the scientific community' has no precise test. ARC Prize's evaluation is a partial independent check, which he acknowledged.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Gary Marcus Observation
“The best results were truly impressive, such as major advances on Francois Chollet’s ARC task and on an important mathematical benchmark FrontierMath.”
Our reading: o3's best demo results showed truly impressive major advances on ARC and FrontierMath.
No horizon Evidence as of
Limitation: 'Truly impressive' is a judgment and was not scored.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Gary Marcus Observation
“(It certainly was not the long-awaited GPT-5).”
Our reading: OpenAI's o3 is not the long-awaited GPT-5.
No horizon Evidence as of
Limitation: The core evidence for the demo day is a news report, not an OpenAI page from that day.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Gary Marcus Observation
“As in: Passing ARC may be necessary but certainly not sufficient for AGI.”
Our reading: Passing the ARC test is certainly not sufficient for AGI, though it may be necessary.
No horizon Evidence as of
Limitation: The verdict concerns what the benchmark result establishes, not a universal theorem about every possible AGI definition.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Ed Zitron Observation
“Be the person who tells people that Sam Altman burns $5 billion a year on unsustainable software that destroys the environment and is built upon the large-scale larceny of creative works because he's desperate for power.”
Our reading: OpenAI's software is built upon the large-scale theft of creative works.
No horizon Evidence as of
Limitation: The interpretation of larceny determines the result.
Source: Never Forgive Them
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI’s problems with text and copyright started to become especially clear a year ago, when the New York Times sued them, and showed how close some of the outputs could be to some of the stories on their website.”
Our reading: OpenAI's problems with text and copyright started to become especially clear a year ago, when the New York Times sued OpenAI and showed how close some of the outputs could be to some of the stories on its website.
No horizon Evidence as of
Limitation: A complaint establishes allegations and presented evidence, not a judgment of infringement.
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: SupportedSaid
Gary Marcus Observation
“They can easily tell you one thing and (especially if hooked up as agents) do another — possibly quite contrary to what they have alleged they are doing.”
Our reading: LLMs can easily say one thing and do another, especially when deployed as agents.
No horizon Evidence as of
Limitation: Follow-up results concern trials where the relevant actions had already occurred; they do not measure how often those actions occur in ordinary use [e1].
Source: Humanity’s “Oh shit!” AI moment?
Assessment: SupportedSaid
Gary Marcus Observation
“In 2022 I warned that there would be chatbot related deaths such as suicides. This began to happen in 2023 and continued in 2024.”
Our reading: Chatbot-related deaths such as suicides occurred in 2023 and 2024.
No horizon Evidence as of
Limitation: Causation is not established. The Belgian link rests on the widow's account and chat logs she supplied. The Setzer link rests on lawsuit allegations.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Gary Marcus Observation
“Part of the value in skepticism is correctly anticipating road blocks, and suggesting alternative directions (as I have for decades with neurosymbolic AI, which is at last coming into vogue, as noted in Fortune earlier this week).”
Our reading: Neurosymbolic AI is finally coming into vogue.
No horizon Evidence as of
Limitation: The result depends on reading vogue as growing attention.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Gary Marcus Observation
“That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”
Our reading: Sora's failures of object permanence and spatiotemporal continuity are general rather than isolated.
No horizon Evidence as of
Limitation: Support rests on contemporaneous developer documentation rather than a representative independent audit.
Assessment: SupportedSaid
Ed Zitron Observation
“OpenAI currently spends $2.35 to make $1, and Microsoft CFO Amy Hood said that OpenAI would cut into Microsoft's profits this October, losing it a remarkable $1.5 billion, "mainly because of an expected loss from OpenAI" according to CNBC.”
Our reading: OpenAI currently spends $2.35 for every $1 of revenue it makes.
No horizon Evidence as of
Limitation: The figures were projections when published.
Source: Godot Isn't Making it
Assessment: SupportedSaid
Gary Marcus Observation
“Two years later Google’s search share hasn’t diminished all that much.”
Our reading: Two years after ChatGPT's launch, Google's search share has not diminished much.
No horizon Evidence as of
Limitation: The figures come from secondary reporting of Statcounter because historical primary data were not retrievable.
Source: ChatGPT, at Age Two
Assessment: SupportedSaid
Gary Marcus Observation
“GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.”
Our reading: GPT-4 was better than GPT-3 in almost every way, whereas o1 is a far less uniform improvement.
No horizon Evidence as of
Limitation: The result depends on reading the comparison qualitatively across capability categories.
Assessment: SupportedSaid
Gary Marcus Observation
“(In some cases o1 doesn’t even do better than GPT-4).”
Our reading: In some cases o1 does not perform better than GPT-4.
No horizon Evidence as of
Limitation: The supported comparison is principally against GPT-4o, not specifically the original GPT-4.
Assessment: SupportedSaid
Gary Marcus Observation
“Many other Tech CEOs and others, including Marc Andreeseen (A16Z), Alexandr Wang (ScaleAI), have also used the phrase “hitting a wall” and the idea of diminishing returns in the last couple weeks.”
Our reading: In the last couple of weeks before November 23, 2024, many other tech CEOs and others, including Marc Andreessen and Alexandr Wang, used the phrase "hitting a wall" and the idea of diminishing returns.
No horizon Evidence as of
Limitation: Andreessen did not use the exact phrase 'hitting a wall'; the claim is scored on the group reading.
Source: Satya Nadella and the three stages of scientific truth
Assessment: SupportedSaid
Gary Marcus Observation
“Scaling was always a hypothesis.”
Our reading: Scaling (improving AI by adding more data and compute) was always a hypothesis.
No horizon Evidence as of
Limitation: Hypothesis does not mean unsupported speculation.
Assessment: SupportedSaid
Gary Marcus Observation
“They are merely empirical generalizations that held for a certain period time, when there was enough fresh data and compute.”
Our reading: Scaling laws are merely empirical generalizations that held for a limited period while enough fresh data and compute were available.
No horizon Evidence as of
Limitation: The result depends on reading the sentence in light of the surrounding hedges about scaling having stopped.
Assessment: SupportedSaid
Gary Marcus Observation
“Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs.”
Our reading: Marcus and many others, including Melanie Mitchell, Subbarao Kambhampati, Emily Bender and Ernie Davis, have long emphasized that there are principled limits with LLMs.
No horizon Evidence as of
Limitation: 'For ages' is colloquial, and the evidence does not establish an identical duration for every researcher.
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: SupportedSaid
Gary Marcus Observation
“People aren’t ignoring GenAI; they are waiting to see if it will work.”
Our reading: People are not ignoring generative AI; they are waiting to see whether it will work.
No horizon Evidence as of
Limitation: 'People' is undefined; the strongest evidence covers businesses.
Source: AI influencer hype gone wild
Assessment: SupportedSaid
Gary Marcus Observation
“(is OpenAI’s o1 100 times better than GPT4? Certainly not by their own data)”
Our reading: By OpenAI's own data, o1 is certainly not 100 times better than GPT-4.
No horizon Evidence as of
Limitation: OpenAI's baseline is GPT-4o, not the original GPT-4.
Assessment: SupportedSaid
Gary Marcus Observation
“Jailbreaks aren’t new, but even after years of them, the tech industry has nothing like a robust response.”
Our reading: Despite years of jailbreaks, the tech industry has no robust defense against them.
No horizon Evidence as of
Limitation: The result depends on the stated reading of robust.
Source: When it comes to security, LLMs are like Swiss cheese — and that’s going to cause huge problems
Assessment: SupportedSaid
Ed Zitron Observation
“It isn’t. You saw a prototype and heard a speech.”
Our reading: Tesla's robotaxi has not arrived; the We, Robot event showed only a prototype and a speech.
No horizon Evidence as of
Limitation: This assesses availability at the event and publication, not subsequent deployment.
Assessment: SupportedSaid
Gary Marcus Observation
“Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.”
Our reading: Both older and newer LLMs have repeatedly been observed to get worse as multiplication problems get larger.
No horizon Evidence as of
Limitation: The Deng results are a researcher's social-media post plus a news report, not a peer-reviewed paper.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Gary Marcus Observation
“Performance is ok on small problems, but quickly tails off.”
Our reading: o1 does acceptably on small planning problems, but its performance drops quickly as problems get larger.
No horizon Evidence as of
Limitation: The model tested was o1-preview.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Gary Marcus Observation
“Failure to follow the rules of chess is another continuing failure of formal reasoning:”
Our reading: LLMs still fail to follow the rules of chess, which is an ongoing failure of formal reasoning.
No horizon Evidence as of
Limitation: The evidence is one puzzle benchmark with a single prompt format. Other formats or full games may give different illegal-move rates.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Ed Zitron Observation
“All of this is happening at a time when an astronomical amount of talent is leaving the company — the talent that OpenAI desperately needs to build products that people will actually pay for.”
Our reading: OpenAI is losing a huge amount of the talent it needs to build products people will pay for.
No horizon Evidence as of
Limitation: The size description is qualitative.
Source: OpenAI Is A Bad Business
Assessment: SupportedSaid
Ed Zitron Observation
“To be abundantly clear, as it stands, OpenAI currently spends $2.35 to make $1.”
Our reading: OpenAI currently spends $2.35 for every $1 of revenue.
No horizon Evidence as of
Limitation: The result depends on reading the sentence in its immediate numerical context.
Source: OpenAI Is A Bad Business
Assessment: SupportedSaid
Ed Zitron Observation
“OpenAI's products are increasingly-commoditized, with Google, Meta, Amazon and even Microsoft building generative AI models to compete.”
Our reading: OpenAI's products are increasingly commoditized as big tech firms build competing models.
No horizon Evidence as of
Limitation: The result depends on reading commoditization as increasing substitutability for common tasks.
Source: OpenAI Is A Bad Business