Supported claims, page 3 of 6
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Supported claims (531)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 3 of 6: claims 201 to 300 of 531, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Gary Marcus Observation
“GenAI is an approach that is enormously popular, but one that is neither reliable nor particularly well-grounded in truth.”
Our reading: GenAI is an enormously popular approach, but one that is neither reliable nor particularly well-grounded in truth.
No horizon Evidence as of
Limitation: The adoption figure is company-reported through a news interview, not independently audited.
Assessment: SupportedSaid
Gary Marcus Observation
“Many other authors are discovering the same thing; social media is full of reports like mine.”
Our reading: Many other authors are discovering that Meta used their works from the pirated database.
No horizon Evidence as of
Limitation: The evidence establishes widespread reports qualitatively, not a measured share of social media.
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: SupportedSaid
Gary Marcus Observation
“He worked hard on every one of those books and articles and does not appreciate them being used without permission by AI systems with an established tendency towards regurgitation.”
Our reading: AI systems such as Meta's have an established tendency to regurgitate their training data.
No horizon Evidence as of
Limitation: The studies do not establish how often reproduction occurs in representative everyday conversations or whether Marcus's particular works were reproduced [e1, e2].
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: SupportedSaid
Gary Marcus Observation
“I don’t want spoil all the fun, but … things didn’t go well:”
Our reading: In The Guardian's test, people using AI tools without coding knowledge did not succeed well at building a Pac-Man clone.
No horizon Evidence as of
Limitation: The report does not establish universal failure or a population failure rate [e1, e2].
Source: AI Coding Fantasy meets Pac-Man
Assessment: SupportedSaid
Gary Marcus Forecast
“To be sure, most or all trained coders will (if they are not already) soon use AI as a part of their workflow.”
Our reading: Most or all trained coders will soon use AI as part of their workflow, if they are not already.
Settled before its deadline Evidence as of
Limitation: These are self-reported surveys, not a census.
Source: AI Coding Fantasy meets Pac-Man
Assessment: SupportedSaid
Gary Marcus Observation
“Musk’s robotaxi promise hasn’t fared much better.”
Our reading: Elon Musk's 2019 promise of over a million Tesla robotaxis has largely failed to come true.
No horizon Evidence as of
Limitation: The filing establishes launch status rather than an exact worldwide robotaxi count.
Assessment: SupportedSaid
Gary Marcus Observation
“Even in the best work, like Google’s new AI co-scientist, humans scientists are still choosing the research goals.”
Our reading: Even in the best AI-for-science work, such as Google's AI co-scientist, human scientists still choose the research goals.
No horizon Evidence as of
Limitation: Support is limited to the concrete named example.
Assessment: SupportedSaid
Gary Marcus Observation
“Hallucinations haven’t solved, despite constant assurances.”
Our reading: Hallucinations have not been solved despite constant assurances that they would be.
No horizon Evidence as of
Limitation: PersonQA is a company-reported, task-specific evaluation; its rates do not establish rates across everyday use.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“Hallucinations simply haven’t gone away, even in the latest models.”
Our reading: Hallucinations have not gone away, even in the latest models.
No horizon Evidence as of
Limitation: PersonQA concerns factual questions about people; its rate is not an estimate for all uses.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“• AGI should have reliable ability to reason over abstractions. (Ongoing unpredictable errors in LLMs reflect this.)”
Our reading: LLMs still lack a reliable ability to reason over abstractions, as their ongoing unpredictable errors show.
No horizon Evidence as of
Limitation: The February studies are preprints and do not provide complete execution timestamps.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“Most parts of the US still don’t have functioning driverless cars systems, for example, even after over a decade of lavish promises.”
Our reading: Most parts of the US still lack functioning driverless car systems after more than a decade of lavish promises.
No horizon Evidence as of
Limitation: Geographic coverage is assessed qualitatively rather than through a national area or population calculation.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“But it is absolutely absurd to pretend “AGI imminent” hypothesis isn’t getting play in high places.”
Our reading: The hypothesis that AGI is imminent is getting play in high places, and it is absurd to pretend otherwise.
No horizon Evidence as of
Limitation: This supports the general observation, not every example or institutional explanation in the surrounding post.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“Every red cell in this graph (e.g, multiplying 20 digit numbers by 19 digit numbers2) is failure to generalize, even after immense amounts of training data:”
Our reading: LLMs including o1-mini fail to generalize on large multi-digit multiplication despite immense training data.
No horizon Evidence as of
Limitation: The retrieved text does not permit a cell-by-cell audit, including the specific 20-by-19-digit example.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Gary Marcus Observation
“A new study of the latest system, GPT 4.5 showed that by on one standardized measure over 1/3 of the outputs are still hallucinations.”
Our reading: On one standardized measure, over a third of GPT-4.5's outputs are hallucinations.
No horizon Evidence as of
Limitation: The exact percentage is supported by Computerworld's report of OpenAI's evaluation, not a readable primary numeric table.
Assessment: SupportedSaid
Gary Marcus Observation
“Already, of course, DeepSeek has shown that our once strong lead is diminishing.”
Our reading: DeepSeek has shown that the US's once-strong lead is shrinking.
No horizon Evidence as of
Limitation: The report was published after the claim, although its measurements cover the relevant earlier period.
Assessment: SupportedSaid
Gary Marcus Observation
“Here Roose has displayed the cardinal sin of looking only at regurgitation rather than generalization. History has told us over and over that systems can be good at the former and still face enormous challenges at the latter.”
Our reading: AI systems can be good at regurgitating known code while still facing enormous challenges generalizing to new problems.
No horizon Evidence as of
Limitation: Training similarity and robustness failures do not by themselves prove verbatim memorization.
Assessment: SupportedSaid
Gary Marcus Observation
“And of course, there is still no GPT-5, even after two years, of hype, bullshit, and cryptic hints.”
Our reading: OpenAI still has not released GPT-5 after two years of hype.
No horizon Evidence as of
Limitation: The assessment concerns public availability, not internal prototypes.
Source: OpenAI, in deep trouble
Assessment: SupportedSaid
Gary Marcus Observation
“So now GPT 4.5 has dropped, and — breaking news — I was right. Hallucinations didn’t disappear today, and nor did stupid errors.”
Our reading: GPT-4.5 did not eliminate hallucinations or stupid errors.
No horizon Evidence as of
Limitation: The evidence is developer-reported.
Assessment: SupportedSaid
Gary Marcus Forecast
“Actually, at any moment, we will finally get GPT 4.5, which most certainly wouldn’t live up to expectations around GPT-5 — hence its rather tentative name.”
Our reading: GPT 4.5 will not live up to the expectations that had been set for GPT-5.
Settled before its deadline Evidence as of
Limitation: The decisive measurements are company-reported.
Source: GPT 4.5 is no GPT-5
Assessment: SupportedSaid
Gary Marcus Observation
“In “Deep learning is hitting a wall”, I warned that problems with reasoning and hallucination wouldn’t go away simply as a function of scaling, and they haven’t.”
Our reading: Problems with reasoning and hallucination in deep learning systems have not gone away as a function of scaling.
No horizon Evidence as of
Limitation: Humanity's Last Exam combines knowledge and reasoning; not every wrong answer identifies a reasoning failure.
Source: GPT 4.5 is no GPT-5
Assessment: SupportedSaid
Gary Marcus Observation
“Simply false, in view of the many basic aspects of geometry that AlphaGeometry2 does not cover, as discussed above.”
Our reading: The claim that AlphaGeometry2 'just figured out geometry' is false because it does not cover many basic aspects of geometry.
No horizon Evidence as of
Limitation: Support depends on reading figured out geometry as general mastery.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: SupportedSaid
Gary Marcus Observation
“Unlike DeepMind’s AlphaZero system which learned to play multiple board games, or OpenAI’s ChatGPT, AlphaGeometry2 is nowhere close to being a single general-purpose learning architecture.”
Our reading: AlphaGeometry2 is a specialized system, nowhere close to a single general-purpose learning architecture.
No horizon Evidence as of
Limitation: The documentation became available after Marcus’s February 17 post.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: SupportedSaid
Gary Marcus Observation
“About year ago the City of New York tried this kind of thing, with less than spectacular results:”
Our reading: About a year ago the City of New York tried this kind of thing (deploying problematic, unreliable AI to the public), with less than spectacular results.
No horizon Evidence as of
Limitation: The documented tests do not establish a representative overall error rate.
Assessment: SupportedSaid
Gary Marcus Forecast
“Second, pure scaling will no longer be the means of attack.”
Our reading: OpenAI will no longer rely on pure scaling as its means of building GPT-5.
Settled before its deadline Evidence as of
Limitation: The disclosure establishes the additional mechanisms, not their precise share of capability gains.
Source: Breaking: OpenAI's efforts at pure scaling have hit a wall.
Assessment: SupportedSaid
Gary Marcus Observation
“Even the latest systems like Deep Research are still struggling in a few ways – and those ways pretty much correspond exactly to the places that I warned would be LLM’s Achilles’ Heels: hallucinations and reasoning errors.”
Our reading: The latest systems such as OpenAI's Deep Research still suffer from hallucinations and reasoning errors.
No horizon Evidence as of
Limitation: Supports the existence of both error classes, not any particular frequency or severity.
Assessment: SupportedSaid
Gary Marcus Observation
“To some extent newer models are doing that.”
Our reading: Newer models such as DeepSeek R1, with a rule-based reward system, partly use neurosymbolic techniques.
No horizon Evidence as of
Limitation: The outcome depends on the broad, training-inclusive meaning supplied by the speaker's example.
Assessment: SupportedSaid
Gary Marcus Observation
“And that’s just not realistic given the current state of AI.”
Our reading: Current AI cannot reliably get synthesis of online information and multi-step research right.
No horizon Evidence as of
Limitation: The evidence directly concerns Deep Research, not every contemporary AI system.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Gary Marcus Observation
“The basic conjecture then — which has already been confirmed to some degree – was that LLMs would be used to write garbage content undermining the value of the internet itself.”
Our reading: His 2023 conjecture that LLMs would be used to write garbage content undermining the internet's value has already been confirmed to some degree.
No horizon Evidence as of
Limitation: The Wikipedia study was a preprint and relied on imperfect detectors.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Gary Marcus Observation
“Sam Altman just released a fancier remake Deep Research, which, like Meta’s predecessor, can write science-sounding articles on demand, on any topic of your choice.”
Our reading: OpenAI's Deep Research is essentially a fancier remake of Meta's Galactica that can write science-sounding articles on any topic on demand.
No horizon Evidence as of
Limitation: The evidence supports a shared writing function, not technical equivalence or equal performance.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Gary Marcus Observation
“DeepSeek r1 is not smarter than earlier models, just trained more cheaply”
Our reading: DeepSeek r1 is not smarter than earlier models.
No horizon Evidence as of
Limitation: The result depends on interpreting earlier models as the preceding frontier.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: SupportedSaid
Gary Marcus Observation
“It doesn't solve hallucinations or problems with reliability.”
Our reading: DeepSeek r1 does not solve hallucinations or reliability problems.
No horizon Evidence as of
Limitation: The hallucination example comes from a reported reasoning trace, rather than a systematic release-period benchmark.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: SupportedSaid
Gary Marcus Forecast
“None of this means however that China won the AI race or even took the lead. American companies will incorporate these new results and continue to produce new results of their own.”
Our reading: The recent Chinese results do not mean China has won the AI race or even taken the lead; American companies will incorporate these results and keep producing new results of their own.
Settled before its deadline Evidence as of
Limitation: The decisive examples involve model checkpoints and generated training data, rather than proof that every named technique was adopted.
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: SupportedSaid
Gary Marcus Observation
“Today he walked it back.”
Our reading: On 2025-01-20 Sam Altman walked back his earlier hint that OpenAI was close to building AGI.
No horizon Evidence as of
Limitation: Relies on news reports quoting the X post, not the post itself.
Assessment: SupportedSaid
Gary Marcus Observation
“That is in part because you can’t build reliable agents unless you can solve the hallucination problem, and there is still no principled trustworthy answer to that problem.”
Our reading: As of January 2025 there is no principled, trustworthy solution to the hallucination problem.
No horizon Evidence as of
Limitation: 'Principled trustworthy answer' has no stated threshold. The assessment uses the contract's reading: a method that reliably prevents hallucinations in general use.
Assessment: SupportedSaid
Gary Marcus Forecast
“The thing about scheduling is that it needs to work reliably. And it won’t.”
Our reading: OpenAI's ChatGPT task-scheduling agent will not work reliably.
Settled before its deadline Evidence as of
Limitation: The evidence is anecdotal. There is no measured failure rate.
Assessment: SupportedSaid
Gary Marcus Observation
“Despite the daily shouts of “exponential progress” for nearly two years, GPT-5 hasn’t come.”
Our reading: As of January 2025, GPT-5 has not been released despite nearly two years of claims of exponential progress.
No horizon Evidence as of
Limitation: The in-period evidence is a news summary of a paywalled WSJ report.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Gary Marcus Observation
“But it’s becoming increasingly clear that they are struggling in building a model worthy of the GPT-5 name.”
Our reading: OpenAI is struggling to build a model worthy of the GPT-5 name.
No horizon Evidence as of
Limitation: Rests on anonymous-source news reports summarized by secondary outlets.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Gary Marcus Observation
“The hypothesis I have been floating since 2022 that scaling data alone would not be enough for robust AI has become increasingly prevalent.”
Our reading: Marcus's hypothesis, which he has been floating since 2022, that scaling data alone would not be enough for robust AI has become increasingly prevalent.
No horizon Evidence as of
Limitation: The evidence does not measure prevalence in a defined population or establish majority agreement.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Gary Marcus Forecast
“No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”
Our reading: The next generation of AI systems will have greater abilities and broader applications and will somewhat mitigate current weaknesses.
Settled before its deadline Evidence as of
Limitation: Support uses the ordinary directional reading, not a claim that every successor or application improved.
Source: The Five Stages of AGI Grief
Assessment: SupportedSaid
Gary Marcus Forecast
“They will be a major focus of attention in the field in 2025.”
Our reading: AI agents will be a major focus of attention in the AI field in 2025.
Deadline passed Evidence as of
Limitation: 'Major focus' has no numeric threshold. The evidence shows agents were prominent but does not rank them against other themes in the field.
Assessment: SupportedSaid
Gary Marcus Observation
“But what was most striking to us was not the unsurprising fact that these systems are not close to ready for prime time (an employee that bad would probably be immediately fired), but the examples from the list of what went wrong:”
Our reading: Current LLM agents (Claude, Gemini, GPT-4 etc.) are not close to ready for prime time on real-world workplace tasks.
No horizon Evidence as of
Limitation: 'Ready for prime time' has no explicit threshold and is a judgment.
Assessment: SupportedSaid
Gary Marcus Observation
“Ten months later, Sora is finally available to the public, and it’s apparent that it too is struggling with physical reasoning.”
Our reading: Sora, now publicly available, struggles with physical reasoning.
No horizon Evidence as of
Limitation: The announcement does not quantify the physical-error rate or independently reproduce the examples in the post.
Assessment: SupportedSaid
Gary Marcus Observation
“But one important component of physical and conceptual understanding is reasoning about entities and their properties, and current approaches have consistently fallen short on this.”
Our reading: Current AI approaches have consistently fallen short on reasoning about entities and their properties.
No horizon Evidence as of
Limitation: This outcome depends on reading consistently as recurring gaps, not failure on every task.
Assessment: SupportedSaid
Gary Marcus Observation
“Depressingly, despite all of the apparent progress in the last few years, much of what we raised in 2015 and re-emphasized in 2019 remains unsolved.”
Our reading: Much of the commonsense reasoning problem the authors raised in 2015 and 2019 remains unsolved in AI as of January 2025.
No horizon Evidence as of
Limitation: The result depends on interpreting 'unsolved' as lacking broad reliability.
Assessment: SupportedSaid
Gary Marcus Observation
“If general purpose domestic robots are a long way away, “AI agents” —AI systems that act in the world on user’s behalf—are already here to a limited degree.”
Our reading: AI agents, meaning AI systems that act in the world on users' behalf, are already here to a limited degree, even though general-purpose domestic robots are a long way away.
No horizon Evidence as of
Limitation: This establishes limited availability and functionality, not general reliability or widespread adoption.
Assessment: SupportedSaid
Gary Marcus Observation
“Hoping that reliable commonsense will simply “emerge” from large databases and video has proven unrealistic, again and again.”
Our reading: The hope that reliable commonsense will emerge from large text and video datasets has repeatedly proven unrealistic.
No horizon Evidence as of
Limitation: Reliable commonsense remains an interpretive standard.
Assessment: SupportedSaid
Gary Marcus Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: Throughout 2025, AI agents will be endlessly hyped but far from reliable, except possibly in very narrow use cases.
Deadline passed Evidence as of
Limitation: The promotion shown comes from one company, OpenAI. It does not measure how much agents were promoted across the industry.
Assessment: SupportedSaid
Gary Marcus Forecast
“Copyright lawsuits over generative AI will continue throughout the year.”
Our reading: Copyright lawsuits over generative AI will continue throughout 2025.
Deadline passed Evidence as of
Limitation: The records establish continuing litigation, not the merits of infringement allegations.
Assessment: SupportedSaid
Gary Marcus Forecast
“Reasoning flubs will continue to haunt generative AI.”
Our reading: Through 2025 reasoning errors will continue to plague generative AI.
Deadline passed Evidence as of
Limitation: 'Flubs' and 'haunt' set no threshold, so the verdict rests on a plain reading.
Assessment: SupportedSaid
Gary Marcus Observation
“Immediately after the February announcement of Sora, I noted Sora’s troubles with physics and predicted that they wouldn’t go away. They haven’t.”
Our reading: Sora's troubles with physics have not gone away since its February 2024 announcement.
No horizon Evidence as of
Limitation: The disclosures do not quantify failure rates or establish that physics performance had not improved [e1, e2].
Assessment: SupportedSaid
Gary Marcus Observation
“My prediction that 2024 would be seen as the year of AI disillusionment was overstated but not entirely off the mark, holding true for some folks but not all, and as such perhaps deserve partial credit.”
Our reading: 2024 was a year of AI disillusionment for some but not all; his prediction was overstated.
No horizon Evidence as of
Limitation: The surveys cover business leaders and employees rather than a representative global public.
Assessment: SupportedSaid
Gary Marcus Observation
“There is still no robust solution to hallucinations.”
Our reading: There is still no robust solution to LLM hallucinations.
No horizon Evidence as of
Limitation: The main source is a lab paper describing its own models' limits. Independent leaderboard figures for late 2024 could not be read.
Assessment: SupportedSaid
Gary Marcus Forecast
“Hallucinations (which should really be called confabulations) will continue to haunt generative AI.”
Our reading: Through 2025 hallucinations will continue to plague generative AI.
Deadline passed Evidence as of
Limitation: 'Haunt' has no threshold. The verdict rests on the plain reading that the problem stayed material and recognized.
Assessment: SupportedSaid
Gary Marcus Observation
“There are many such models now; there is a price war. There is very little moat.”
Our reading: There are many GPT-4 level models, a price war, and very little moat among LLM providers.
No horizon Evidence as of
Limitation: 'Moat' is read as a lasting performance lead. Brand, distribution or user base, which are other possible moats, were not measured.
Assessment: SupportedSaid
Gary Marcus Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: AI agents will be heavily hyped in 2025 but far from reliable except possibly in very narrow use cases.
Deadline passed Evidence as of
Limitation: The result partly depends on reading coding agents as a 'narrow use case'. A reader who treats coding as broad could call this mixed.
Assessment: SupportedSaid
Gary Marcus Forecast
“OpenAI will continue to preview products months and perhaps years before they are solid and widely available at an accessible price.”
Our reading: By the end of 2025 OpenAI will have continued to preview products months, and perhaps years, before they are solid and widely available at an accessible price.
Deadline passed Evidence as of
Limitation: The result depends on treating premium Pro-only previews as short of ordinary broad consumer access.
Assessment: SupportedSaid
Gary Marcus Observation
“At *most* the test is necessary for AGI; it certainly isn’t sufficient.”
Our reading: Passing ARC is at most necessary for AGI and certainly not sufficient.
No horizon Evidence as of
Limitation: The contract notes that AGI is not defined, so 'sufficient' is judged by the designers' stated view.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Gary Marcus Observation
“What was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.”
Our reading: What was done for o3 on ARC, pretraining on what Marcus believes was hundreds of public examples, is not comparable to what humans require.
No horizon Evidence as of
Limitation: The conclusion depends on reading comparable as referring to ARC-specific preparation.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Gary Marcus Observation
“We still don’t have a solid test of what o3 does without the pretraining, in the case that would be more comparable to humans”
Our reading: There is no solid test yet of o3's ARC performance without pretraining.
No horizon Evidence as of
Limitation: The evaluator page has later updates, and the training note's precise insertion date is not separately archived in this record.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Gary Marcus Observation
“I am saying that many basic problems haven’t been solved.”
Our reading: Many basic problems on the path to AGI remain unsolved.
No horizon Evidence as of
Limitation: The scope and meaning of 'many' are inferred and materially affect the assessment.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Gary Marcus Observation
“The product was not yet released to the general public; it’s not been vetted by the scientific community.”
Our reading: As of the demo, o3 had not been released to the public nor vetted by the scientific community.
No horizon Evidence as of
Limitation: 'Vetted by the scientific community' has no precise test. ARC Prize's evaluation is a partial independent check, which he acknowledged.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Gary Marcus Observation
“The best results were truly impressive, such as major advances on Francois Chollet’s ARC task and on an important mathematical benchmark FrontierMath.”
Our reading: o3's best demo results showed truly impressive major advances on ARC and FrontierMath.
No horizon Evidence as of
Limitation: 'Truly impressive' is a judgment and was not scored.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Gary Marcus Observation
“(It certainly was not the long-awaited GPT-5).”
Our reading: OpenAI's o3 is not the long-awaited GPT-5.
No horizon Evidence as of
Limitation: The core evidence for the demo day is a news report, not an OpenAI page from that day.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI’s problems with text and copyright started to become especially clear a year ago, when the New York Times sued them, and showed how close some of the outputs could be to some of the stories on their website.”
Our reading: OpenAI's problems with text and copyright started to become especially clear a year ago, when the New York Times sued OpenAI and showed how close some of the outputs could be to some of the stories on its website.
No horizon Evidence as of
Limitation: A complaint establishes allegations and presented evidence, not a judgment of infringement.
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: SupportedSaid
Gary Marcus Observation
“In 2022 I warned that there would be chatbot related deaths such as suicides. This began to happen in 2023 and continued in 2024.”
Our reading: Chatbot-related deaths such as suicides occurred in 2023 and 2024.
No horizon Evidence as of
Limitation: Causation is not established. The Belgian link rests on the widow's account and chat logs she supplied. The Setzer link rests on lawsuit allegations.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Gary Marcus Observation
“Part of the value in skepticism is correctly anticipating road blocks, and suggesting alternative directions (as I have for decades with neurosymbolic AI, which is at last coming into vogue, as noted in Fortune earlier this week).”
Our reading: Neurosymbolic AI is finally coming into vogue.
No horizon Evidence as of
Limitation: The result depends on reading vogue as growing attention.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Gary Marcus Observation
“That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”
Our reading: Sora's failures of object permanence and spatiotemporal continuity are general rather than isolated.
No horizon Evidence as of
Limitation: Support rests on contemporaneous developer documentation rather than a representative independent audit.
Assessment: SupportedSaid
Gary Marcus Observation
“Two years later Google’s search share hasn’t diminished all that much.”
Our reading: Two years after ChatGPT's launch, Google's search share has not diminished much.
No horizon Evidence as of
Limitation: The figures come from secondary reporting of Statcounter because historical primary data were not retrievable.
Source: ChatGPT, at Age Two
Assessment: SupportedSaid
Gary Marcus Observation
“GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.”
Our reading: GPT-4 was better than GPT-3 in almost every way, whereas o1 is a far less uniform improvement.
No horizon Evidence as of
Limitation: The result depends on reading the comparison qualitatively across capability categories.
Assessment: SupportedSaid
Gary Marcus Observation
“(In some cases o1 doesn’t even do better than GPT-4).”
Our reading: In some cases o1 does not perform better than GPT-4.
No horizon Evidence as of
Limitation: The supported comparison is principally against GPT-4o, not specifically the original GPT-4.
Assessment: SupportedSaid
Gary Marcus Observation
“Many other Tech CEOs and others, including Marc Andreeseen (A16Z), Alexandr Wang (ScaleAI), have also used the phrase “hitting a wall” and the idea of diminishing returns in the last couple weeks.”
Our reading: In the last couple of weeks before November 23, 2024, many other tech CEOs and others, including Marc Andreessen and Alexandr Wang, used the phrase "hitting a wall" and the idea of diminishing returns.
No horizon Evidence as of
Limitation: Andreessen did not use the exact phrase 'hitting a wall'; the claim is scored on the group reading.
Source: Satya Nadella and the three stages of scientific truth
Assessment: SupportedSaid
Gary Marcus Observation
“Scaling was always a hypothesis.”
Our reading: Scaling (improving AI by adding more data and compute) was always a hypothesis.
No horizon Evidence as of
Limitation: Hypothesis does not mean unsupported speculation.
Assessment: SupportedSaid
Gary Marcus Observation
“They are merely empirical generalizations that held for a certain period time, when there was enough fresh data and compute.”
Our reading: Scaling laws are merely empirical generalizations that held for a limited period while enough fresh data and compute were available.
No horizon Evidence as of
Limitation: The result depends on reading the sentence in light of the surrounding hedges about scaling having stopped.
Assessment: SupportedSaid
Gary Marcus Observation
“Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs.”
Our reading: Marcus and many others, including Melanie Mitchell, Subbarao Kambhampati, Emily Bender and Ernie Davis, have long emphasized that there are principled limits with LLMs.
No horizon Evidence as of
Limitation: 'For ages' is colloquial, and the evidence does not establish an identical duration for every researcher.
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: SupportedSaid
Gary Marcus Observation
“People aren’t ignoring GenAI; they are waiting to see if it will work.”
Our reading: People are not ignoring generative AI; they are waiting to see whether it will work.
No horizon Evidence as of
Limitation: 'People' is undefined; the strongest evidence covers businesses.
Source: AI influencer hype gone wild
Assessment: SupportedSaid
Gary Marcus Observation
“(is OpenAI’s o1 100 times better than GPT4? Certainly not by their own data)”
Our reading: By OpenAI's own data, o1 is certainly not 100 times better than GPT-4.
No horizon Evidence as of
Limitation: OpenAI's baseline is GPT-4o, not the original GPT-4.
Assessment: SupportedSaid
Gary Marcus Observation
“Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.”
Our reading: Both older and newer LLMs have repeatedly been observed to get worse as multiplication problems get larger.
No horizon Evidence as of
Limitation: The Deng results are a researcher's social-media post plus a news report, not a peer-reviewed paper.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Gary Marcus Observation
“Performance is ok on small problems, but quickly tails off.”
Our reading: o1 does acceptably on small planning problems, but its performance drops quickly as problems get larger.
No horizon Evidence as of
Limitation: The model tested was o1-preview.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Gary Marcus Observation
“Failure to follow the rules of chess is another continuing failure of formal reasoning:”
Our reading: LLMs still fail to follow the rules of chess, which is an ongoing failure of formal reasoning.
No horizon Evidence as of
Limitation: The evidence is one puzzle benchmark with a single prompt format. Other formats or full games may give different illegal-move rates.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Gary Marcus Observation
“I rushed out a hot take on X, my usual cautions about how LLMs aren’t really AGI:”
Our reading: LLMs, including OpenAI's new o1, are not really AGI.
No horizon Evidence as of
Limitation: The system card tests several model checkpoints and configurations; it is not an exhaustive evaluation of all contemporary LLMs [e3].
Assessment: SupportedSaid
Gary Marcus Observation
“Almost exactly what I predicted we would see with GPT-4, back on Christmas Day 2022.”
Our reading: Altman's description of o1 as 'still flawed, still limited, seem more impressive on first use' is almost exactly what Marcus predicted we would see with GPT-4 on Christmas Day 2022.
No horizon Evidence as of
Limitation: The agreement is thematic, not verbatim.
Assessment: SupportedSaid
Gary Marcus Observation
“California’s pending AI bill SB-1047 was supposed to be a tiny step in that direction, and even that has gotten watered down.”
Our reading: California's SB-1047, already only a small step toward external AI governance, has been watered down.
No horizon Evidence as of
Limitation: The evidence is the author's dated amendment explanation, not a complete comparison of all bill versions.
Source: Scoop: What former employees of OpenAI are worried about
Assessment: SupportedSaid
Gary Marcus Observation
“When Sam Altman stood next to me in the Senate, he told the Senate that he strongly favored AI regulation. But behind the scenes his team fought to water down the EU AI Act.”
Our reading: Sam Altman told the Senate, standing next to Marcus, that he strongly favored AI regulation, but behind the scenes his team fought to water down the EU AI Act.
No horizon Evidence as of
Limitation: The lobbying finding relies on reputable reporting because the available primary PDF extraction is corrupted.
Assessment: SupportedSaid
Gary Marcus Observation
“It also provides important whistleblower protections, which as many ex OpenAI employees have made clear, are critical.”
Our reading: SB-1047 provides important whistleblower protections, which, as many ex-OpenAI employees have made clear, are critical.
No horizon Evidence as of
Limitation: The assessment concerns a proposed bill, not protections already in force.
Assessment: SupportedSaid
Gary Marcus Observation
“Most or all of the major big tech companies joined a lobbying organization that fought SB-1047, tooth and nail, despite broad public support for the bill.”
Our reading: Most or all major big tech companies joined a lobbying group that fought SB-1047 hard despite broad public support for it.
No horizon Evidence as of
Limitation: The majority assessment uses the contract's five-company group: Alphabet/Google, Amazon, Apple, Meta, and Microsoft.
Assessment: SupportedSaid
Gary Marcus Observation
“Lastly, asking for standards is not unique to AI; it's common across many industries to ask that companies evaluate the safety of their products according to set standards: just look at the pharmaceutical industry, or aviation, automobiles, etc.”
Our reading: Asking for standards is not unique to AI: in many industries, such as pharmaceuticals, aviation and automobiles, it is common to ask companies to evaluate the safety of their products against set standards.
No horizon Evidence as of
Limitation: The evidence establishes the three named U.S. examples, not a numerical survey of every industry or jurisdiction.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Gary Marcus Observation
“And much of what would be required is already something companies committed to voluntarily, in discussions at the White House and in Seoul.”
Our reading: Much of what SB-1047 would require is something AI companies already voluntarily committed to at the White House and in Seoul.
No horizon Evidence as of
Limitation: The assessment is a qualitative comparison, not a count of statutory clauses.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI recently put Paul Nakasone (ex NSA) on the board”
Our reading: OpenAI recently put Paul Nakasone, formerly of the NSA, on its board.
No horizon Evidence as of
Limitation: This establishes the appointment and career background, not any implication about surveillance.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: SupportedSaid
Gary Marcus Observation
“An awful lot of key players have left the island. Most voluntarily.”
Our reading: Many key OpenAI people have left, most of them voluntarily.
No horizon Evidence as of
Limitation: The roster is a transparent group drawn from contemporaneous reporting, not a complete personnel register.
Source: The OpenAI Plot Thickens
Assessment: SupportedSaid
Gary Marcus Observation
“Nvidia dropped 6%, 20% over the last month”
Our reading: Nvidia stock fell 6% on August 5, 2024, and 20% over the previous month.
No horizon Evidence as of
Limitation: The result depends on reading the figures as rounded closing-price changes.
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI tempered expectations for its next event, and said we wouldn’t see GPT-5 then.”
Our reading: OpenAI tempered expectations for its next event and said we would not see GPT-5 at that event.
No horizon Evidence as of
Limitation: Evidence is a reporter's direct quotation, not an accessible original company announcement.
Assessment: SupportedSaid
Gary Marcus Observation
“Another GenAI monetization scheme bites the dust.”
Our reading: Meta dropping a major GenAI project is one more GenAI monetization scheme that has failed.
No horizon Evidence as of
Limitation: Shutdown evidence comes from a reputable news report quoting Meta directly; no standalone Meta shutdown announcement was recovered.
Assessment: SupportedSaid
Gary Marcus Observation
“Consequently, any “reasoning” or “planning” in LLMs is hit or miss, viable if the particulars of a given situation are close enough to what is in the training set, but highly vulnerable if they are not.”
Our reading: Marcus says any reasoning or planning in LLMs is hit or miss: viable if a situation's particulars are close enough to the training set, but highly vulnerable if they are not.
No horizon Evidence as of
Limitation: Reasoning models released after July 2024 are not assessed.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Much as I admire the new pair of systems, they do have a serious shortcoming and one that is not unfamiliar. The systems both rely on a kind of cheat on the input: trained human coders translate all the Olympiad's input sentences into mathematical form.”
Our reading: AlphaProof and AlphaGeometry 2 rely on human coders translating Olympiad problems into formal form, a serious shortcoming.
No horizon Evidence as of
Limitation: The assessment concerns the July 2024 demonstration, not later systems.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Nonetheless, GDM's new work is powerful proof of concept (pun intended!). Neural networks and symbol systems can be profitably combined, contra Hinton and LeCun.”
Our reading: Google DeepMind's work shows neural networks and symbolic systems can be profitably combined.
No horizon Evidence as of
Limitation: Technical usefulness does not imply financial profitability.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Hallucinations and boneheaded errors of reasoning and what I like to call discomprehensions are, as far as I can tell, an inevitable byproduct of LLMs; certainly they have yet to disappear, despite many promises to the contrary.”
Our reading: Marcus says that, as far as he can tell, hallucinations, boneheaded reasoning errors and 'discomprehensions' are an inevitable byproduct of LLMs, and that they certainly have not yet disappeared despite many promises to the contrary.
No horizon Evidence as of
Limitation: Only the 'yet to disappear' part is scored.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Causal claim
“As Ernie Davis and I pointed out in Rebooting AI, five years ago, part of the reason we are struggling with AI in complex AI systems is that we still lack adequate techniques for engineering complex systems. That hasn’t really changed.”
Our reading: Struggles with complex AI systems stem partly from a lack of adequate techniques for engineering complex systems, which has not really changed since 2019.
No horizon Evidence as of
Limitation: The causal assessment synthesizes technical explanations; it is not a controlled estimate of engineering's contribution to failures.
Source: Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call
Assessment: SupportedSaid
Gary Marcus Forecast
“As of today, I am more confident than ever that GPT-5 won’t land this year.”
Our reading: OpenAI will not release GPT-5 in 2024.
Deadline passed Evidence as of
Limitation: This record assesses only the release timing of a model named GPT-5. It does not assess Marcus's reasons for the forecast or what GPT-5 could do.
Assessment: SupportedSaid
Gary Marcus Observation
“Exhibit A: OpenAI’s CTO Mira Murati just publicly acknowledged what I long suspected: there is no mind blowing GPT-5 behind the scenes as of yet.”
Our reading: As of June 2024, OpenAI has no mind-blowing GPT-5 model waiting behind the scenes.
No horizon Evidence as of
Limitation: The decisive source is contemporaneous news reporting because the relevant primary interview transcript could not be recovered.
Assessment: SupportedSaid
Gary Marcus Observation
“In many ways, it is much weaker than what Altman and I talked about at the Senate last year.”
Our reading: SB-1047 is in many ways much weaker than the AI regulation Marcus and Sam Altman discussed at the Senate in 2023.
No horizon Evidence as of
Limitation: The result depends on reading strength through the licensing and review mechanisms highlighted in the surrounding passage.