Supported claims by Gary Marcus, page 3 of 6
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Supported claims by Gary Marcus (558)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 3 of 6: claims 201 to 300 of 558, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Observation
“It certainly took zero responsibility for what appears to be an obvious infringement.”
Our reading: Sam Altman's response at TED took no responsibility for what appears to be an obvious copyright infringement by an OpenAI system.
No horizon Evidence as of
Limitation: The finding concerns the content of the reply, not the legality of the image.
Source: Sam Altman’s attitude problem
Assessment: SupportedSaid
Causal claim
“GenAI is pitched as (near) AGI, but reliability problems persist, limiting how well it can be used in practice.”
Our reading: Although GenAI is marketed as near-AGI, its reliability problems persist and limit how well it can be used in practice.
No horizon Evidence as of
Limitation: The CEO's statement does not establish that existing products were AGI or that every supplier used the same framing.
Source: Poor ROI for GenAI
Assessment: SupportedSaid
Observation
“My sources assure me that this is NOT a clerical error and that Papan’s office confirms that the change is real.”
Our reading: The change replacing AB-501's content with unrelated provisions is real and not a clerical error, according to the author's sources.
No horizon Evidence as of
Limitation: The evidence does not identify Marcus's private sources or verify his communications with them.
Assessment: SupportedSaid
Observation
“First, the AI community is in fact pretty disappointed with LLama 4’s performance:”
Our reading: The AI community is disappointed with Llama 4's performance.
No horizon Evidence as of
Limitation: Public commentary is self-selected.
Source: Deep Learning, Deep Scandal
Assessment: SupportedSaid
Observation
“models still don’t reason reliably (as anticipated in What to Expect When Your Expecting GPT-4)”
Our reading: LLMs still do not reason reliably as of April 2025.
No horizon Evidence as of
Limitation: The cited versions are preprints.
Source: Scaling is over, the bubble may be deflating, LLMs still can’t reason, and you can’t trust Sam
Assessment: SupportedSaid
Observation
“In Taming Silicon Valley, I warned that LLMs would be used for dumb things that would affect lots of people.”
Our reading: Marcus says that in Taming Silicon Valley he warned that LLMs would be used for dumb things that would affect lots of people, and he presents the tariff formula, which he says was probably recommended by an LLM, as confirming that warning.
No horizon Evidence as of
Limitation: The complete book was not inspected; the assessment rests on an identified excerpt and an independent review.
Assessment: SupportedSaid
Causal claim
“To do well the system had to be coached to adopt the persona of “a young person who is introverted, knowledgeable about internet culture, and uses slang”; it’s still in no small part about the tricks.”
Our reading: To do well in the new study, the system had to be coached to adopt the persona of a young, introverted, slang-using person knowledgeable about internet culture, so its success is still in no small part about tricks.
No horizon Evidence as of
Limitation: The comparison does not establish that this particular persona was necessary among all possible prompts [e1].
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: SupportedSaid
Observation
“Especially since DeepSeek there has been a massive price war.”
Our reading: There has been a massive LLM price war, especially since DeepSeek.
No horizon Evidence as of
Limitation: Historical primary announcements were not recovered for the decisive reports; reputable contemporaneous Reuters reporting is used.
Assessment: SupportedSaid
Observation
“GenAI is an approach that is enormously popular, but one that is neither reliable nor particularly well-grounded in truth.”
Our reading: GenAI is an enormously popular approach, but one that is neither reliable nor particularly well-grounded in truth.
No horizon Evidence as of
Limitation: The adoption figure is company-reported through a news interview, not independently audited.
Assessment: SupportedSaid
Observation
“Many other authors are discovering the same thing; social media is full of reports like mine.”
Our reading: Many other authors are discovering that Meta used their works from the pirated database.
No horizon Evidence as of
Limitation: The evidence establishes widespread reports qualitatively, not a measured share of social media.
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: SupportedSaid
Observation
“He worked hard on every one of those books and articles and does not appreciate them being used without permission by AI systems with an established tendency towards regurgitation.”
Our reading: AI systems such as Meta's have an established tendency to regurgitate their training data.
No horizon Evidence as of
Limitation: The studies do not establish how often reproduction occurs in representative everyday conversations or whether Marcus's particular works were reproduced [e1, e2].
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: SupportedSaid
Observation
“I don’t want spoil all the fun, but … things didn’t go well:”
Our reading: In The Guardian's test, people using AI tools without coding knowledge did not succeed well at building a Pac-Man clone.
No horizon Evidence as of
Limitation: The report does not establish universal failure or a population failure rate [e1, e2].
Source: AI Coding Fantasy meets Pac-Man
Assessment: SupportedSaid
Forecast
“To be sure, most or all trained coders will (if they are not already) soon use AI as a part of their workflow.”
Our reading: Most or all trained coders will soon use AI as part of their workflow, if they are not already.
Settled before its deadline Evidence as of
Limitation: These are self-reported surveys, not a census.
Source: AI Coding Fantasy meets Pac-Man
Assessment: SupportedSaid
Observation
“Musk’s robotaxi promise hasn’t fared much better.”
Our reading: Elon Musk's 2019 promise of over a million Tesla robotaxis has largely failed to come true.
No horizon Evidence as of
Limitation: The filing establishes launch status rather than an exact worldwide robotaxi count.
Assessment: SupportedSaid
Observation
“Even in the best work, like Google’s new AI co-scientist, humans scientists are still choosing the research goals.”
Our reading: Even in the best AI-for-science work, such as Google's AI co-scientist, human scientists still choose the research goals.
No horizon Evidence as of
Limitation: Support is limited to the concrete named example.
Assessment: SupportedSaid
Observation
“Hallucinations haven’t solved, despite constant assurances.”
Our reading: Hallucinations have not been solved despite constant assurances that they would be.
No horizon Evidence as of
Limitation: PersonQA is a company-reported, task-specific evaluation; its rates do not establish rates across everyday use.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“Hallucinations simply haven’t gone away, even in the latest models.”
Our reading: Hallucinations have not gone away, even in the latest models.
No horizon Evidence as of
Limitation: PersonQA concerns factual questions about people; its rate is not an estimate for all uses.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“• AGI should have reliable ability to reason over abstractions. (Ongoing unpredictable errors in LLMs reflect this.)”
Our reading: LLMs still lack a reliable ability to reason over abstractions, as their ongoing unpredictable errors show.
No horizon Evidence as of
Limitation: The February studies are preprints and do not provide complete execution timestamps.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“Most parts of the US still don’t have functioning driverless cars systems, for example, even after over a decade of lavish promises.”
Our reading: Most parts of the US still lack functioning driverless car systems after more than a decade of lavish promises.
No horizon Evidence as of
Limitation: Geographic coverage is assessed qualitatively rather than through a national area or population calculation.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“But it is absolutely absurd to pretend “AGI imminent” hypothesis isn’t getting play in high places.”
Our reading: The hypothesis that AGI is imminent is getting play in high places, and it is absurd to pretend otherwise.
No horizon Evidence as of
Limitation: This supports the general observation, not every example or institutional explanation in the surrounding post.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“Every red cell in this graph (e.g, multiplying 20 digit numbers by 19 digit numbers2) is failure to generalize, even after immense amounts of training data:”
Our reading: LLMs including o1-mini fail to generalize on large multi-digit multiplication despite immense training data.
No horizon Evidence as of
Limitation: The retrieved text does not permit a cell-by-cell audit, including the specific 20-by-19-digit example.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“A new study of the latest system, GPT 4.5 showed that by on one standardized measure over 1/3 of the outputs are still hallucinations.”
Our reading: On one standardized measure, over a third of GPT-4.5's outputs are hallucinations.
No horizon Evidence as of
Limitation: The exact percentage is supported by Computerworld's report of OpenAI's evaluation, not a readable primary numeric table.
Assessment: SupportedSaid
Observation
“Already, of course, DeepSeek has shown that our once strong lead is diminishing.”
Our reading: DeepSeek has shown that the US's once-strong lead is shrinking.
No horizon Evidence as of
Limitation: The report was published after the claim, although its measurements cover the relevant earlier period.
Assessment: SupportedSaid
Observation
“Here Roose has displayed the cardinal sin of looking only at regurgitation rather than generalization. History has told us over and over that systems can be good at the former and still face enormous challenges at the latter.”
Our reading: AI systems can be good at regurgitating known code while still facing enormous challenges generalizing to new problems.
No horizon Evidence as of
Limitation: Training similarity and robustness failures do not by themselves prove verbatim memorization.
Assessment: SupportedSaid
Observation
“And of course, there is still no GPT-5, even after two years, of hype, bullshit, and cryptic hints.”
Our reading: OpenAI still has not released GPT-5 after two years of hype.
No horizon Evidence as of
Limitation: The assessment concerns public availability, not internal prototypes.
Source: OpenAI, in deep trouble
Assessment: SupportedSaid
Observation
“So now GPT 4.5 has dropped, and — breaking news — I was right. Hallucinations didn’t disappear today, and nor did stupid errors.”
Our reading: GPT-4.5 did not eliminate hallucinations or stupid errors.
No horizon Evidence as of
Limitation: The evidence is developer-reported.
Assessment: SupportedSaid
Forecast
“Actually, at any moment, we will finally get GPT 4.5, which most certainly wouldn’t live up to expectations around GPT-5 — hence its rather tentative name.”
Our reading: GPT 4.5 will not live up to the expectations that had been set for GPT-5.
Settled before its deadline Evidence as of
Limitation: The decisive measurements are company-reported.
Source: GPT 4.5 is no GPT-5
Assessment: SupportedSaid
Observation
“In “Deep learning is hitting a wall”, I warned that problems with reasoning and hallucination wouldn’t go away simply as a function of scaling, and they haven’t.”
Our reading: Problems with reasoning and hallucination in deep learning systems have not gone away as a function of scaling.
No horizon Evidence as of
Limitation: Humanity's Last Exam combines knowledge and reasoning; not every wrong answer identifies a reasoning failure.
Source: GPT 4.5 is no GPT-5
Assessment: SupportedSaid
Observation
“Hallucinations haven’t been magically solved, etc.”
Our reading: Grok 3 has not solved hallucinations.
No horizon Evidence as of
Limitation: The decisive evidence is an identified researcher's informal testing, not a controlled benchmark.
Source: Grok 3 Hot Take
Assessment: SupportedSaid
Observation
“That wasn’t tested (presumably they don’t have access), and only a handful of benchmarks were actually reported.”
Our reading: The Grok 3 launch did not test against full o3 and reported only a handful of benchmarks.
No horizon Evidence as of
Limitation: The complete primary launch charts were not readable, so the evidence does not establish an exhaustive benchmark count.
Source: Grok 3 Hot Take
Assessment: SupportedSaid
Observation
“That said, OpenAI’s moat keeps diminishing, so price wars will continue and profits will continue to be elusive for everyone except Nvidia.”
Our reading: OpenAI's competitive moat keeps shrinking.
No horizon Evidence as of
Limitation: The outcome depends on interpreting moat as the capability advantage discussed in the post.
Source: Grok 3 Hot Take
Assessment: SupportedSaid
Forecast
“That said, OpenAI’s moat keeps diminishing, so price wars will continue and profits will continue to be elusive for everyone except Nvidia.”
Our reading: Because OpenAI's moat keeps shrinking, price wars among AI model providers will continue.
Settled before its deadline Evidence as of
Limitation: The causal evidence is reporting and analyst assessment, not a controlled economic study.
Source: Grok 3 Hot Take
Assessment: SupportedSaid
Observation
“Simply false, in view of the many basic aspects of geometry that AlphaGeometry2 does not cover, as discussed above.”
Our reading: The claim that AlphaGeometry2 'just figured out geometry' is false because it does not cover many basic aspects of geometry.
No horizon Evidence as of
Limitation: Support depends on reading figured out geometry as general mastery.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: SupportedSaid
Observation
“Unlike DeepMind’s AlphaZero system which learned to play multiple board games, or OpenAI’s ChatGPT, AlphaGeometry2 is nowhere close to being a single general-purpose learning architecture.”
Our reading: AlphaGeometry2 is a specialized system, nowhere close to a single general-purpose learning architecture.
No horizon Evidence as of
Limitation: The documentation became available after Marcus’s February 17 post.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: SupportedSaid
Observation
“About year ago the City of New York tried this kind of thing, with less than spectacular results:”
Our reading: About a year ago the City of New York tried this kind of thing (deploying problematic, unreliable AI to the public), with less than spectacular results.
No horizon Evidence as of
Limitation: The documented tests do not establish a representative overall error rate.
Assessment: SupportedSaid
Forecast
“Second, pure scaling will no longer be the means of attack.”
Our reading: OpenAI will no longer rely on pure scaling as its means of building GPT-5.
Settled before its deadline Evidence as of
Limitation: The disclosure establishes the additional mechanisms, not their precise share of capability gains.
Source: Breaking: OpenAI's efforts at pure scaling have hit a wall.
Assessment: SupportedSaid
Observation
“Even the latest systems like Deep Research are still struggling in a few ways – and those ways pretty much correspond exactly to the places that I warned would be LLM’s Achilles’ Heels: hallucinations and reasoning errors.”
Our reading: The latest systems such as OpenAI's Deep Research still suffer from hallucinations and reasoning errors.
No horizon Evidence as of
Limitation: Supports the existence of both error classes, not any particular frequency or severity.
Assessment: SupportedSaid
Observation
“To some extent newer models are doing that.”
Our reading: Newer models such as DeepSeek R1, with a rule-based reward system, partly use neurosymbolic techniques.
No horizon Evidence as of
Limitation: The outcome depends on the broad, training-inclusive meaning supplied by the speaker's example.
Assessment: SupportedSaid
Observation
“And that’s just not realistic given the current state of AI.”
Our reading: Current AI cannot reliably get synthesis of online information and multi-step research right.
No horizon Evidence as of
Limitation: The evidence directly concerns Deep Research, not every contemporary AI system.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Observation
“The basic conjecture then — which has already been confirmed to some degree – was that LLMs would be used to write garbage content undermining the value of the internet itself.”
Our reading: His 2023 conjecture that LLMs would be used to write garbage content undermining the internet's value has already been confirmed to some degree.
No horizon Evidence as of
Limitation: The Wikipedia study was a preprint and relied on imperfect detectors.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Observation
“Sam Altman just released a fancier remake Deep Research, which, like Meta’s predecessor, can write science-sounding articles on demand, on any topic of your choice.”
Our reading: OpenAI's Deep Research is essentially a fancier remake of Meta's Galactica that can write science-sounding articles on any topic on demand.
No horizon Evidence as of
Limitation: The evidence supports a shared writing function, not technical equivalence or equal performance.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Observation
“DeepSeek r1 is not smarter than earlier models, just trained more cheaply”
Our reading: DeepSeek r1 is not smarter than earlier models.
No horizon Evidence as of
Limitation: The result depends on interpreting earlier models as the preceding frontier.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: SupportedSaid
Observation
“It doesn't solve hallucinations or problems with reliability.”
Our reading: DeepSeek r1 does not solve hallucinations or reliability problems.
No horizon Evidence as of
Limitation: The hallucination example comes from a reported reasoning trace, rather than a systematic release-period benchmark.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: SupportedSaid
Forecast
“None of this means however that China won the AI race or even took the lead. American companies will incorporate these new results and continue to produce new results of their own.”
Our reading: The recent Chinese results do not mean China has won the AI race or even taken the lead; American companies will incorporate these results and keep producing new results of their own.
Settled before its deadline Evidence as of
Limitation: The decisive examples involve model checkpoints and generated training data, rather than proof that every named technique was adopted.
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: SupportedSaid
Observation
“Today he walked it back.”
Our reading: On 2025-01-20 Sam Altman walked back his earlier hint that OpenAI was close to building AGI.
No horizon Evidence as of
Limitation: Relies on news reports quoting the X post, not the post itself.
Assessment: SupportedSaid
Observation
“That is in part because you can’t build reliable agents unless you can solve the hallucination problem, and there is still no principled trustworthy answer to that problem.”
Our reading: As of January 2025 there is no principled, trustworthy solution to the hallucination problem.
No horizon Evidence as of
Limitation: 'Principled trustworthy answer' has no stated threshold. The assessment uses the contract's reading: a method that reliably prevents hallucinations in general use.
Assessment: SupportedSaid
Forecast
“The thing about scheduling is that it needs to work reliably. And it won’t.”
Our reading: OpenAI's ChatGPT task-scheduling agent will not work reliably.
Settled before its deadline Evidence as of
Limitation: The evidence is anecdotal. There is no measured failure rate.
Assessment: SupportedSaid
Observation
“Despite the daily shouts of “exponential progress” for nearly two years, GPT-5 hasn’t come.”
Our reading: As of January 2025, GPT-5 has not been released despite nearly two years of claims of exponential progress.
No horizon Evidence as of
Limitation: The in-period evidence is a news summary of a paywalled WSJ report.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Observation
“But it’s becoming increasingly clear that they are struggling in building a model worthy of the GPT-5 name.”
Our reading: OpenAI is struggling to build a model worthy of the GPT-5 name.
No horizon Evidence as of
Limitation: Rests on anonymous-source news reports summarized by secondary outlets.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Observation
“The hypothesis I have been floating since 2022 that scaling data alone would not be enough for robust AI has become increasingly prevalent.”
Our reading: Marcus's hypothesis, which he has been floating since 2022, that scaling data alone would not be enough for robust AI has become increasingly prevalent.
No horizon Evidence as of
Limitation: The evidence does not measure prevalence in a defined population or establish majority agreement.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Forecast
“No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”
Our reading: The next generation of AI systems will have greater abilities and broader applications and will somewhat mitigate current weaknesses.
Settled before its deadline Evidence as of
Limitation: Support uses the ordinary directional reading, not a claim that every successor or application improved.
Source: The Five Stages of AGI Grief
Assessment: SupportedSaid
Forecast
“They will be a major focus of attention in the field in 2025.”
Our reading: AI agents will be a major focus of attention in the AI field in 2025.
Deadline passed Evidence as of
Limitation: 'Major focus' has no numeric threshold. The evidence shows agents were prominent but does not rank them against other themes in the field.
Assessment: SupportedSaid
Observation
“But what was most striking to us was not the unsurprising fact that these systems are not close to ready for prime time (an employee that bad would probably be immediately fired), but the examples from the list of what went wrong:”
Our reading: Current LLM agents (Claude, Gemini, GPT-4 etc.) are not close to ready for prime time on real-world workplace tasks.
No horizon Evidence as of
Limitation: 'Ready for prime time' has no explicit threshold and is a judgment.
Assessment: SupportedSaid
Observation
“Ten months later, Sora is finally available to the public, and it’s apparent that it too is struggling with physical reasoning.”
Our reading: Sora, now publicly available, struggles with physical reasoning.
No horizon Evidence as of
Limitation: The announcement does not quantify the physical-error rate or independently reproduce the examples in the post.
Assessment: SupportedSaid
Observation
“But one important component of physical and conceptual understanding is reasoning about entities and their properties, and current approaches have consistently fallen short on this.”
Our reading: Current AI approaches have consistently fallen short on reasoning about entities and their properties.
No horizon Evidence as of
Limitation: This outcome depends on reading consistently as recurring gaps, not failure on every task.
Assessment: SupportedSaid
Observation
“Depressingly, despite all of the apparent progress in the last few years, much of what we raised in 2015 and re-emphasized in 2019 remains unsolved.”
Our reading: Much of the commonsense reasoning problem the authors raised in 2015 and 2019 remains unsolved in AI as of January 2025.
No horizon Evidence as of
Limitation: The result depends on interpreting 'unsolved' as lacking broad reliability.
Assessment: SupportedSaid
Observation
“If general purpose domestic robots are a long way away, “AI agents” —AI systems that act in the world on user’s behalf—are already here to a limited degree.”
Our reading: AI agents, meaning AI systems that act in the world on users' behalf, are already here to a limited degree, even though general-purpose domestic robots are a long way away.
No horizon Evidence as of
Limitation: This establishes limited availability and functionality, not general reliability or widespread adoption.
Assessment: SupportedSaid
Observation
“Hoping that reliable commonsense will simply “emerge” from large databases and video has proven unrealistic, again and again.”
Our reading: The hope that reliable commonsense will emerge from large text and video datasets has repeatedly proven unrealistic.
No horizon Evidence as of
Limitation: Reliable commonsense remains an interpretive standard.
Assessment: SupportedSaid
Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: Throughout 2025, AI agents will be endlessly hyped but far from reliable, except possibly in very narrow use cases.
Deadline passed Evidence as of
Limitation: The promotion shown comes from one company, OpenAI. It does not measure how much agents were promoted across the industry.
Assessment: SupportedSaid
Forecast
“Copyright lawsuits over generative AI will continue throughout the year.”
Our reading: Copyright lawsuits over generative AI will continue throughout 2025.
Deadline passed Evidence as of
Limitation: The records establish continuing litigation, not the merits of infringement allegations.
Assessment: SupportedSaid
Forecast
“Reasoning flubs will continue to haunt generative AI.”
Our reading: Through 2025 reasoning errors will continue to plague generative AI.
Deadline passed Evidence as of
Limitation: 'Flubs' and 'haunt' set no threshold, so the verdict rests on a plain reading.
Assessment: SupportedSaid
Observation
“Immediately after the February announcement of Sora, I noted Sora’s troubles with physics and predicted that they wouldn’t go away. They haven’t.”
Our reading: Sora's troubles with physics have not gone away since its February 2024 announcement.
No horizon Evidence as of
Limitation: The disclosures do not quantify failure rates or establish that physics performance had not improved [e1, e2].
Assessment: SupportedSaid
Observation
“My prediction that 2024 would be seen as the year of AI disillusionment was overstated but not entirely off the mark, holding true for some folks but not all, and as such perhaps deserve partial credit.”
Our reading: 2024 was a year of AI disillusionment for some but not all; his prediction was overstated.
No horizon Evidence as of
Limitation: The surveys cover business leaders and employees rather than a representative global public.
Assessment: SupportedSaid
Observation
“There is still no robust solution to hallucinations.”
Our reading: There is still no robust solution to LLM hallucinations.
No horizon Evidence as of
Limitation: The main source is a lab paper describing its own models' limits. Independent leaderboard figures for late 2024 could not be read.
Assessment: SupportedSaid
Forecast
“Hallucinations (which should really be called confabulations) will continue to haunt generative AI.”
Our reading: Through 2025 hallucinations will continue to plague generative AI.
Deadline passed Evidence as of
Limitation: 'Haunt' has no threshold. The verdict rests on the plain reading that the problem stayed material and recognized.
Assessment: SupportedSaid
Observation
“There are many such models now; there is a price war. There is very little moat.”
Our reading: There are many GPT-4 level models, a price war, and very little moat among LLM providers.
No horizon Evidence as of
Limitation: 'Moat' is read as a lasting performance lead. Brand, distribution or user base, which are other possible moats, were not measured.
Assessment: SupportedSaid
Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: AI agents will be heavily hyped in 2025 but far from reliable except possibly in very narrow use cases.
Deadline passed Evidence as of
Limitation: The result partly depends on reading coding agents as a 'narrow use case'. A reader who treats coding as broad could call this mixed.
Assessment: SupportedSaid
Forecast
“OpenAI will continue to preview products months and perhaps years before they are solid and widely available at an accessible price.”
Our reading: By the end of 2025 OpenAI will have continued to preview products months, and perhaps years, before they are solid and widely available at an accessible price.
Deadline passed Evidence as of
Limitation: The result depends on treating premium Pro-only previews as short of ordinary broad consumer access.
Assessment: SupportedSaid
Observation
“At *most* the test is necessary for AGI; it certainly isn’t sufficient.”
Our reading: Passing ARC is at most necessary for AGI and certainly not sufficient.
No horizon Evidence as of
Limitation: The contract notes that AGI is not defined, so 'sufficient' is judged by the designers' stated view.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Observation
“What was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.”
Our reading: What was done for o3 on ARC, pretraining on what Marcus believes was hundreds of public examples, is not comparable to what humans require.
No horizon Evidence as of
Limitation: The conclusion depends on reading comparable as referring to ARC-specific preparation.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Observation
“We still don’t have a solid test of what o3 does without the pretraining, in the case that would be more comparable to humans”
Our reading: There is no solid test yet of o3's ARC performance without pretraining.
No horizon Evidence as of
Limitation: The evaluator page has later updates, and the training note's precise insertion date is not separately archived in this record.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Observation
“I am saying that many basic problems haven’t been solved.”
Our reading: Many basic problems on the path to AGI remain unsolved.
No horizon Evidence as of
Limitation: The scope and meaning of 'many' are inferred and materially affect the assessment.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“The product was not yet released to the general public; it’s not been vetted by the scientific community.”
Our reading: As of the demo, o3 had not been released to the public nor vetted by the scientific community.
No horizon Evidence as of
Limitation: 'Vetted by the scientific community' has no precise test. ARC Prize's evaluation is a partial independent check, which he acknowledged.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“The best results were truly impressive, such as major advances on Francois Chollet’s ARC task and on an important mathematical benchmark FrontierMath.”
Our reading: o3's best demo results showed truly impressive major advances on ARC and FrontierMath.
No horizon Evidence as of
Limitation: 'Truly impressive' is a judgment and was not scored.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“(It certainly was not the long-awaited GPT-5).”
Our reading: OpenAI's o3 is not the long-awaited GPT-5.
No horizon Evidence as of
Limitation: The core evidence for the demo day is a news report, not an OpenAI page from that day.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“OpenAI’s problems with text and copyright started to become especially clear a year ago, when the New York Times sued them, and showed how close some of the outputs could be to some of the stories on their website.”
Our reading: OpenAI's problems with text and copyright started to become especially clear a year ago, when the New York Times sued OpenAI and showed how close some of the outputs could be to some of the stories on its website.
No horizon Evidence as of
Limitation: A complaint establishes allegations and presented evidence, not a judgment of infringement.
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: SupportedSaid
Observation
“In 2022 I warned that there would be chatbot related deaths such as suicides. This began to happen in 2023 and continued in 2024.”
Our reading: Chatbot-related deaths such as suicides occurred in 2023 and 2024.
No horizon Evidence as of
Limitation: Causation is not established. The Belgian link rests on the widow's account and chat logs she supplied. The Setzer link rests on lawsuit allegations.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Observation
“Part of the value in skepticism is correctly anticipating road blocks, and suggesting alternative directions (as I have for decades with neurosymbolic AI, which is at last coming into vogue, as noted in Fortune earlier this week).”
Our reading: Neurosymbolic AI is finally coming into vogue.
No horizon Evidence as of
Limitation: The result depends on reading vogue as growing attention.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Observation
“That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”
Our reading: Sora's failures of object permanence and spatiotemporal continuity are general rather than isolated.
No horizon Evidence as of
Limitation: Support rests on contemporaneous developer documentation rather than a representative independent audit.
Assessment: SupportedSaid
Observation
“Two years later Google’s search share hasn’t diminished all that much.”
Our reading: Two years after ChatGPT's launch, Google's search share has not diminished much.
No horizon Evidence as of
Limitation: The figures come from secondary reporting of Statcounter because historical primary data were not retrievable.
Source: ChatGPT, at Age Two
Assessment: SupportedSaid
Observation
“GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.”
Our reading: GPT-4 was better than GPT-3 in almost every way, whereas o1 is a far less uniform improvement.
No horizon Evidence as of
Limitation: The result depends on reading the comparison qualitatively across capability categories.
Assessment: SupportedSaid
Observation
“(In some cases o1 doesn’t even do better than GPT-4).”
Our reading: In some cases o1 does not perform better than GPT-4.
No horizon Evidence as of
Limitation: The supported comparison is principally against GPT-4o, not specifically the original GPT-4.
Assessment: SupportedSaid
Observation
“Many other Tech CEOs and others, including Marc Andreeseen (A16Z), Alexandr Wang (ScaleAI), have also used the phrase “hitting a wall” and the idea of diminishing returns in the last couple weeks.”
Our reading: In the last couple of weeks before November 23, 2024, many other tech CEOs and others, including Marc Andreessen and Alexandr Wang, used the phrase "hitting a wall" and the idea of diminishing returns.
No horizon Evidence as of
Limitation: Andreessen did not use the exact phrase 'hitting a wall'; the claim is scored on the group reading.
Source: Satya Nadella and the three stages of scientific truth
Assessment: SupportedSaid
Observation
“Scaling was always a hypothesis.”
Our reading: Scaling (improving AI by adding more data and compute) was always a hypothesis.
No horizon Evidence as of
Limitation: Hypothesis does not mean unsupported speculation.
Assessment: SupportedSaid
Observation
“They are merely empirical generalizations that held for a certain period time, when there was enough fresh data and compute.”
Our reading: Scaling laws are merely empirical generalizations that held for a limited period while enough fresh data and compute were available.
No horizon Evidence as of
Limitation: The result depends on reading the sentence in light of the surrounding hedges about scaling having stopped.
Assessment: SupportedSaid
Observation
“Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs.”
Our reading: Marcus and many others, including Melanie Mitchell, Subbarao Kambhampati, Emily Bender and Ernie Davis, have long emphasized that there are principled limits with LLMs.
No horizon Evidence as of
Limitation: 'For ages' is colloquial, and the evidence does not establish an identical duration for every researcher.
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: SupportedSaid
Observation
“People aren’t ignoring GenAI; they are waiting to see if it will work.”
Our reading: People are not ignoring generative AI; they are waiting to see whether it will work.
No horizon Evidence as of
Limitation: 'People' is undefined; the strongest evidence covers businesses.
Source: AI influencer hype gone wild
Assessment: SupportedSaid
Observation
“(is OpenAI’s o1 100 times better than GPT4? Certainly not by their own data)”
Our reading: By OpenAI's own data, o1 is certainly not 100 times better than GPT-4.
No horizon Evidence as of
Limitation: OpenAI's baseline is GPT-4o, not the original GPT-4.
Assessment: SupportedSaid
Observation
“Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.”
Our reading: Both older and newer LLMs have repeatedly been observed to get worse as multiplication problems get larger.
No horizon Evidence as of
Limitation: The Deng results are a researcher's social-media post plus a news report, not a peer-reviewed paper.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Performance is ok on small problems, but quickly tails off.”
Our reading: o1 does acceptably on small planning problems, but its performance drops quickly as problems get larger.
No horizon Evidence as of
Limitation: The model tested was o1-preview.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Failure to follow the rules of chess is another continuing failure of formal reasoning:”
Our reading: LLMs still fail to follow the rules of chess, which is an ongoing failure of formal reasoning.
No horizon Evidence as of
Limitation: The evidence is one puzzle benchmark with a single prompt format. Other formats or full games may give different illegal-move rates.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“I rushed out a hot take on X, my usual cautions about how LLMs aren’t really AGI:”
Our reading: LLMs, including OpenAI's new o1, are not really AGI.
No horizon Evidence as of
Limitation: The system card tests several model checkpoints and configurations; it is not an exhaustive evaluation of all contemporary LLMs [e3].
Assessment: SupportedSaid
Observation
“Almost exactly what I predicted we would see with GPT-4, back on Christmas Day 2022.”
Our reading: Altman's description of o1 as 'still flawed, still limited, seem more impressive on first use' is almost exactly what Marcus predicted we would see with GPT-4 on Christmas Day 2022.
No horizon Evidence as of
Limitation: The agreement is thematic, not verbatim.
Assessment: SupportedSaid
Observation
“California’s pending AI bill SB-1047 was supposed to be a tiny step in that direction, and even that has gotten watered down.”
Our reading: California's SB-1047, already only a small step toward external AI governance, has been watered down.
No horizon Evidence as of
Limitation: The evidence is the author's dated amendment explanation, not a complete comparison of all bill versions.
Source: Scoop: What former employees of OpenAI are worried about
Assessment: SupportedSaid
Observation
“When Sam Altman stood next to me in the Senate, he told the Senate that he strongly favored AI regulation. But behind the scenes his team fought to water down the EU AI Act.”
Our reading: Sam Altman told the Senate, standing next to Marcus, that he strongly favored AI regulation, but behind the scenes his team fought to water down the EU AI Act.
No horizon Evidence as of
Limitation: The lobbying finding relies on reputable reporting because the available primary PDF extraction is corrupted.
Assessment: SupportedSaid
Observation
“It also provides important whistleblower protections, which as many ex OpenAI employees have made clear, are critical.”
Our reading: SB-1047 provides important whistleblower protections, which, as many ex-OpenAI employees have made clear, are critical.
No horizon Evidence as of
Limitation: The assessment concerns a proposed bill, not protections already in force.
Assessment: SupportedSaid
Observation
“Most or all of the major big tech companies joined a lobbying organization that fought SB-1047, tooth and nail, despite broad public support for the bill.”
Our reading: Most or all major big tech companies joined a lobbying group that fought SB-1047 hard despite broad public support for it.
No horizon Evidence as of
Limitation: The majority assessment uses the contract's five-company group: Alphabet/Google, Amazon, Apple, Meta, and Microsoft.
Assessment: SupportedSaid
Causal claim
“As I emphasized last year in front of the Senate, there is not one risk but many: bias, disinformation, defamation, toxicity, invasion of privacy, and so much more, to say nothing of risks to employment, intellectual property, and the environment, precisely because recent techniques can be applied in so many ways, still poorly understood.”
Our reading: AI poses not one risk but many, including bias, disinformation, defamation, toxicity, invasion of privacy and so much more, plus risks to employment, intellectual property and the environment, precisely because recent techniques can be applied in so many ways and are still poorly understood.
No horizon Evidence as of
Limitation: The evidence includes scientific syntheses and a preprint, rather than one experiment testing the entire claim.
Assessment: SupportedSaid
Observation
“Lastly, asking for standards is not unique to AI; it's common across many industries to ask that companies evaluate the safety of their products according to set standards: just look at the pharmaceutical industry, or aviation, automobiles, etc.”
Our reading: Asking for standards is not unique to AI: in many industries, such as pharmaceuticals, aviation and automobiles, it is common to ask companies to evaluate the safety of their products against set standards.
No horizon Evidence as of
Limitation: The evidence establishes the three named U.S. examples, not a numerical survey of every industry or jurisdiction.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Observation
“And much of what would be required is already something companies committed to voluntarily, in discussions at the White House and in Seoul.”
Our reading: Much of what SB-1047 would require is something AI companies already voluntarily committed to at the White House and in Seoul.
No horizon Evidence as of
Limitation: The assessment is a qualitative comparison, not a count of statutory clauses.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047