Supported claims by Gary Marcus, page 4 of 8
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Supported claims by Gary Marcus (731)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 4 of 8: claims 301 to 400 of 731, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Forecast
“Instead, OpenAI will be throwing the kitchen sink at future efforts to build GPT-5 , including “test-time-compute”, a new approach that involves longer, more expensive inference times rather than “constant time inference” of GPT-4-like models, as well as (I suspect) massive amounts of synthetic data”
Our reading: OpenAI will throw the kitchen sink at future efforts to build GPT-5, including test-time compute and, Marcus suspects, massive amounts of synthetic data, which appears to work better for semi-closed domains like math and coding than in the open-ended real world.
Settled before its deadline Evidence as of
What happened: The release documentation describes controllable thinking time in the deployed GPT-5 API. This directly supports the forecast inclusion of additional inference-time reasoning. (OpenAI, )
Limitation: The evidence consists of OpenAI's own release and technical disclosures.
Source: Breaking: OpenAI's efforts at pure scaling have hit a wall.
Assessment: SupportedSaid
Forecast
“Second, pure scaling will no longer be the means of attack.”
Our reading: OpenAI will no longer rely on pure scaling as its means of building GPT-5.
Settled before its deadline Evidence as of
What happened: The release documents a system combining fast responses with deeper reasoning and routing. Those mechanisms establish that GPT-5 goes beyond increasing pretraining data and parameters alone. (OpenAI, )
Limitation: The disclosure establishes the additional mechanisms, not their precise share of capability gains.
Source: Breaking: OpenAI's efforts at pure scaling have hit a wall.
Assessment: SupportedSaid
Observation
“Even the latest systems like Deep Research are still struggling in a few ways – and those ways pretty much correspond exactly to the places that I warned would be LLM’s Achilles’ Heels: hallucinations and reasoning errors.”
Our reading: The latest systems such as OpenAI's Deep Research still suffer from hallucinations and reasoning errors.
No horizon Evidence as of
What happened: The paragraph expressly identifies factual hallucinations and incorrect inferences as remaining limitations. It also reports a lower rate than existing ChatGPT models, which qualifies the severity without contradicting their continued presence. (OpenAI, )
Limitation: Supports the existence of both error classes, not any particular frequency or severity.
Assessment: SupportedSaid
Observation
“To some extent newer models are doing that.”
Our reading: Newer models such as DeepSeek R1, with a rule-based reward system, partly use neurosymbolic techniques.
No horizon Evidence as of
What happened: Section 2.2.2 describes programmed accuracy and formatting rewards for R1-Zero, including code tests. Section 2.3.4 explicitly applies its rule-based reasoning reward methodology to R1, supporting the limited training-time combination described by Marcus. (DeepSeek-AI, arXiv, )
Limitation: The outcome depends on the broad, training-inclusive meaning supplied by the speaker's example.
Assessment: SupportedSaid
Observation
“Me, December 2022: The problem with LLMs is that they hallucinate, and their errors can be hard to catch.”
Our reading: As of December 2022, LLMs hallucinated and their errors could be hard to catch.
No horizon Evidence as of
What happened: Kubacka describes checking plausible scientific citations and discovering nonexistent articles, mismatched identifiers, and invented physics. She reports that an explanation in her own field initially made her reconsider whether the invented phenomenon existed, directly supporting the possibility of difficult-to-detect errors. (Teresa Kubacka, Lookalikes and Meanders, )
Limitation: This single ChatGPT demonstration does not establish hallucination prevalence or average detection difficulty across models and users.
Source: Shame on Google, twice
Assessment: SupportedSaid
Observation
“And that’s just not realistic given the current state of AI.”
Our reading: Current AI cannot reliably get synthesis of online information and multi-step research right.
No horizon Evidence as of
What happened: The launch disclosure identifies source-discrimination and uncertainty-communication weaknesses directly relevant to dependable research. These support the ordinary reading that the output required substantive checking. (OpenAI, )
Limitation: The evidence directly concerns Deep Research, not every contemporary AI system.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Observation
“The basic conjecture then — which has already been confirmed to some degree – was that LLMs would be used to write garbage content undermining the value of the internet itself.”
Our reading: His 2023 conjecture that LLMs would be used to write garbage content undermining the internet's value has already been confirmed to some degree.
No horizon Evidence as of
What happened: NewsGuard traced a viral false psychiatrist story to a site it classified as AI-generated and documented subsequent dissemination. This is a concrete information-quality harm, although its attribution of this particular rewrite to AI is qualified. (NewsGuard, )
Limitation: The Wikipedia study was a preprint and relied on imperfect detectors.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Observation
“Sam Altman just released a fancier remake Deep Research, which, like Meta’s predecessor, can write science-sounding articles on demand, on any topic of your choice.”
Our reading: OpenAI's Deep Research is essentially a fancier remake of Meta's Galactica that can write science-sounding articles on any topic on demand.
No horizon Evidence as of
What happened: The launch documentation describes user prompts producing comprehensive, cited reports across science, finance, policy, engineering, and consumer topics. This establishes the broad report-generation capability in the comparison. (OpenAI, )
Limitation: The evidence supports a shared writing function, not technical equivalence or equal performance.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: SupportedSaid
Observation
“The real lesson of DeepSeek is that we can still make vast improvements in efficency.”
Our reading: DeepSeek shows that vast efficiency improvements in AI are still possible.
No horizon Evidence as of
What happened: The report documents the training resources of the December V3 model and compares its performance with leading models. Its resource total covers the official training stages and excludes earlier research and ablation experiments; the comparative magnitude is supplied by e2. (DeepSeek-AI / arXiv, )
Limitation: The result depends on interpreting efficiency as computing resources at comparable benchmark capability.
Assessment: SupportedSaid
Observation
“DeepSeek r1 is not smarter than earlier models, just trained more cheaply”
Our reading: DeepSeek r1 is not smarter than earlier models.
No horizon Evidence as of
What happened: The release report characterizes R1 as comparable to o1 overall. Its detailed results also limit that conclusion: R1 slightly exceeds o1 on AIME and exceeds several older models, while trailing o1 on several knowledge benchmarks. (DeepSeek-AI / arXiv, )
Limitation: The result depends on interpreting earlier models as the preceding frontier.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: SupportedSaid
Observation
“It doesn't solve hallucinations or problems with reliability.”
Our reading: DeepSeek r1 does not solve hallucinations or reliability problems.
No horizon Evidence as of
What happened: DeepSeek reports prompt sensitivity in the released R1 model. This establishes a continuing reliability limitation, though it does not itself measure hallucinations. (DeepSeek-AI / arXiv, )
Limitation: The hallucination example comes from a reported reasoning trace, rather than a systematic release-period benchmark.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: SupportedSaid
Observation
“This morning, Nvidia is down more than 10% in pretrading.”
Our reading: On the morning of 2025-01-27, Nvidia stock was down more than 10% in pre-market trading.
No horizon Evidence as of
What happened: The report explicitly places these losses in pre-market trading that morning. Its approximately 12% decline exceeds the claim's 10% threshold. (Associated Press, syndicated by Spectrum News NY1, )
Limitation: The report gives a rounded percentage rather than underlying trade prices.
Assessment: SupportedSaid
Forecast
“None of this means however that China won the AI race or even took the lead. American companies will incorporate these new results and continue to produce new results of their own.”
Our reading: The recent Chinese results do not mean China has won the AI race or even taken the lead; American companies will incorporate these results and keep producing new results of their own.
Settled before its deadline Evidence as of
What happened: NVIDIA explicitly developed a model from an R1-distilled checkpoint. Its report describes additional reinforcement-learning experiments and improved mathematics and coding results, establishing adoption and a further technical contribution. (NVIDIA Research, )
Limitation: The decisive examples involve model checkpoints and generated training data, rather than proof that every named technique was adopted.
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: SupportedSaid
Observation
“On LLMs China is already catching up.”
Our reading: China is already catching up with the U.S. on LLMs.
No horizon Evidence as of
What happened: The preceding sentence gives end-2023 U.S.–China gaps of 17.5, 13.5, 24.3 and 31.6 percentage points on MMLU, MMMU, MATH and HumanEval. All four narrowed by end-2024, including the language, mathematics and coding measures directly relevant to this claim. (Stanford Institute for Human-Centered Artificial Intelligence, )
Limitation: Benchmark performance does not capture every aspect of real-world reliability.
Source: Hot take on an AI catfight
Assessment: SupportedSaid
Observation
“Today he walked it back.”
Our reading: On 2025-01-20 Sam Altman walked back his earlier hint that OpenAI was close to building AGI.
No horizon Evidence as of
What happened: Shows that on 20 January 2025 Altman publicly said OpenAI had not built AGI and would not deploy it next month, which plays down the closeness he had suggested. (Decrypt, )
Limitation: Relies on news reports quoting the X post, not the post itself.
Assessment: SupportedSaid
Forecast
“And those demos won’t work all that reliably.”
Our reading: The AI agent demos of 2025 will not work very reliably.
Deadline passed Evidence as of
What happened: In the 2025 revision, the best agent finished only 30% of realistic workplace tasks on its own. That is far from reliable. (arXiv (Carnegie Mellon University et al.), )
Limitation: 'Not all that reliably' has no numeric threshold. The outcome depends on the contract's example that failing a third or more of tasks counts as not reliable.
Assessment: SupportedSaid
Observation
“That is in part because you can’t build reliable agents unless you can solve the hallucination problem, and there is still no principled trustworthy answer to that problem.”
Our reading: As of January 2025 there is no principled, trustworthy solution to the hallucination problem.
No horizon Evidence as of
What happened: A major science journal's survey of the field in January 2025 says hallucinations can't be stopped and that existing techniques only limit the damage. That matches 'no principled trustworthy answer'. (Nature, )
Limitation: 'Principled trustworthy answer' has no stated threshold. The assessment uses the contract's reading: a method that reliably prevents hallucinations in general use.
Assessment: SupportedSaid
Forecast
“The thing about scheduling is that it needs to work reliably. And it won’t.”
Our reading: OpenAI's ChatGPT task-scheduling agent will not work reliably.
Settled before its deadline Evidence as of
What happened: A hands-on review found Tasks fired a scheduled reminder 12 hours late. The same review reports tasks overwriting one another. (Tom's Guide, )
Limitation: The evidence is anecdotal. There is no measured failure rate.
Assessment: SupportedSaid
Observation
“Despite the daily shouts of “exponential progress” for nearly two years, GPT-5 hasn’t come.”
Our reading: As of January 2025, GPT-5 has not been released despite nearly two years of claims of exponential progress.
No horizon Evidence as of
What happened: On 21 December 2024, GPT-5 (Orion) was reported as still in development, and OpenAI had said it would not release it in 2024. So it had not come out by the post date. (TechCrunch, )
Limitation: The in-period evidence is a news summary of a paywalled WSJ report.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Observation
“But it’s becoming increasingly clear that they are struggling in building a model worthy of the GPT-5 name.”
Our reading: OpenAI is struggling to build a model worthy of the GPT-5 name.
No horizon Evidence as of
What happened: Reports from OpenAI staff, via The Information, said Orion's gains over GPT-4 were much smaller than past generational leaps. (TechRadar (summarizing The Information), )
Limitation: Rests on anonymous-source news reports summarized by secondary outlets.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Observation
“The hypothesis I have been floating since 2022 that scaling data alone would not be enough for robust AI has become increasingly prevalent.”
Our reading: Marcus's hypothesis, which he has been floating since 2022, that scaling data alone would not be enough for robust AI has become increasingly prevalent.
No horizon Evidence as of
What happened: Independent reporting confirms the 2022 advocacy timeline. The surrounding discussion identifies the argument as concerning reliability, reasoning and understanding, rather than a claim that all technical improvement would stop. (Big Think, )
Limitation: The evidence does not measure prevalence in a defined population or establish majority agreement.
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: SupportedSaid
Forecast
“No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”
Our reading: The next generation of AI systems will have greater abilities and broader applications and will somewhat mitigate current weaknesses.
Settled before its deadline Evidence as of
What happened: Anthropic reports improved capabilities and fewer unnecessary refusals relative to the predecessor. It also describes Claude Code's release and practical coding deployments, providing concrete evidence of expanded workflows rather than only benchmark scores. (Anthropic, )
Limitation: Support uses the ordinary directional reading, not a claim that every successor or application improved.
Source: The Five Stages of AGI Grief
Assessment: SupportedSaid
Observation
“While such a milestone would be important economically and socially, it is also a departure from, and even a retrenchment from, the kind of cognitive criteria around flexibility and resourcefulness that people in the field had generally talked about it.”
Our reading: A milestone of a system doing 80% of economically valuable human work, which Marcus recalls (not with certainty) OpenAI suggesting as a definition of AGI, would be important economically and socially, but it departs from, and even retreats from, the cognitive criteria of flexibility and resourcefulness that people in the field had generally discussed.
No horizon Evidence as of
What happened: This earlier definition requires breadth across environments; the surrounding conclusion explicitly connects that breadth with learning and adaptation. An economic share alone does not expressly preserve those requirements, which supports the conceptual comparison. (arXiv; Shane Legg and Marcus Hutter, )
Limitation: Support applies to the comparison of stated requirements.
Source: The Five Stages of AGI Grief
Assessment: SupportedSaid
Observation
“The problem of distribution shift, well-tested in Apple’s 2024 reasoning paper, which goes back to my own 1998 work, continues.”
Our reading: The problem of distribution shift in neural models continues to be unsolved.
No horizon Evidence as of
What happened: The original evaluation reports continuing sensitivity to altered arithmetic questions and irrelevant clauses. Its detailed results also identify negligible basic-variation drops for some models and strong o1-preview performance on difficulty variants, so the evidence supports persistent failures rather than uniform fragility. (arXiv; Apple researchers, )
Limitation: Support concerns continued empirical failures, not a proof that neural models cannot generalize.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Observation
“Commonsense reasoning is still dodgy, as it has long been, as Ernie Davis and I just reviewed here..”
Our reading: Commonsense reasoning in current AI systems remains unreliable, as it long has been.
No horizon Evidence as of
What happened: SimpleToM tests everyday situations requiring inferences about what people can perceive and know. GPT-4o, Claude-3.5-Sonnet and Llama-3.1-405B scored only 10–24.9% on behavior judgments; o1-preview reached 59.5%. However, leading models exceeded 95% on mental-state questions, and tailored reminders plus reasoning prompts substantially improved applied reasoning. (arXiv, )
Limitation: Commonsense is broader than the tasks tested.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Observation
“LLMs, like their predecessors generalize well to similar items but struggle with reliability in unfamiliar regimes, even minor variations sometimes break them.”
Our reading: LLMs generalize well to similar items but are unreliable in unfamiliar regimes, and minor variations sometimes break them.
No horizon Evidence as of
What happened: The study compares twenty-five models and four prompting techniques on original questions and controlled variants, finding competent original-task performance alongside failures on altered tasks. It also reports relative resilience to some numerical and wording changes, consistent with Marcus's qualification that small changes cause failures only sometimes. (arXiv; Qintong Li and colleagues, )
Limitation: The cited evidence concerns mathematical word problems, not every application of LLMs.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Observation
“Many leading figures in the field have acknowledged that we may have reached a period of diminishing returns of pure LLM scaling, much as I anticipated in 2022.”
Our reading: Many leading figures in the field have acknowledged that pure LLM scaling may have reached a period of diminishing returns, much as Marcus anticipated in 2022.
No horizon Evidence as of
What happened: This contemporary report identifies Sutskever and explicitly describes his acknowledgment of a plateau. It is reputable secondary reporting of his position; the underlying Reuters interview was inaccessible through read_page. (TIME, )
Limitation: The quantity 'many' remains interpretive.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Forecast
“They will be a major focus of attention in the field in 2025.”
Our reading: AI agents will be a major focus of attention in the AI field in 2025.
Deadline passed Evidence as of
What happened: A leading lab launched a general-purpose web agent early in 2025. (OpenAI, )
Limitation: 'Major focus' has no numeric threshold. The evidence shows agents were prominent but does not rank them against other themes in the field.
Assessment: SupportedSaid
Observation
“But what was most striking to us was not the unsurprising fact that these systems are not close to ready for prime time (an employee that bad would probably be immediately fired), but the examples from the list of what went wrong:”
Our reading: Current LLM agents (Claude, Gemini, GPT-4 etc.) are not close to ready for prime time on real-world workplace tasks.
No horizon Evidence as of
What happened: The best agent completed only 24% of tasks, and harder long-horizon tasks were beyond all systems. This supports 'not close to ready for prime time'. The authors call the picture nuanced, because simpler tasks were often solved. (arXiv (Xu et al., Carnegie Mellon University), )
Limitation: 'Ready for prime time' has no explicit threshold and is a judgment.
Assessment: SupportedSaid
Observation
“Ten months later, Sora is finally available to the public, and it’s apparent that it too is struggling with physical reasoning.”
Our reading: Sora, now publicly available, struggles with physical reasoning.
No horizon Evidence as of
What happened: OpenAI announced Sora Turbo's release to ChatGPT Plus and Pro users on December 9, following its February preview. The same announcement explicitly acknowledges recurring unrealistic physics in the deployed version, supporting both public availability and the stated physical limitations. (OpenAI, )
Limitation: The announcement does not quantify the physical-error rate or independently reproduce the examples in the post.
Assessment: SupportedSaid
Observation
“But one important component of physical and conceptual understanding is reasoning about entities and their properties, and current approaches have consistently fallen short on this.”
Our reading: Current AI approaches have consistently fallen short on reasoning about entities and their properties.
No horizon Evidence as of
What happened: OpenAI directly acknowledges incorrect object-state changes in Sora. The report also describes successful object persistence and some physical interactions, qualifying any reading that denies all entity-related competence. (OpenAI, )
Limitation: This outcome depends on reading consistently as recurring gaps, not failure on every task.
Assessment: SupportedSaid
Observation
“Depressingly, despite all of the apparent progress in the last few years, much of what we raised in 2015 and re-emphasized in 2019 remains unsolved.”
Our reading: Much of the commonsense reasoning problem the authors raised in 2015 and 2019 remains unsolved in AI as of January 2025.
No horizon Evidence as of
What happened: ContPhy finds persistent problems inferring physical properties and reasoning about dynamics and counterfactuals. These overlap the earlier work's physical-understanding concerns; they do not retest every historical example. (Zheng et al., ICML 2024; arXiv, )
Limitation: The result depends on interpreting 'unsolved' as lacking broad reliability.
Assessment: SupportedSaid
Observation
“If general purpose domestic robots are a long way away, “AI agents” —AI systems that act in the world on user’s behalf—are already here to a limited degree.”
Our reading: AI agents, meaning AI systems that act in the world on users' behalf, are already here to a limited degree, even though general-purpose domestic robots are a long way away.
No horizon Evidence as of
What happened: The announcement makes computer use available through an API and describes user-directed cursor movements, clicks, typing, and form completion. Its explicit warnings about errors fit the claim's limited-degree qualification. (Anthropic, )
Limitation: This establishes limited availability and functionality, not general reliability or widespread adoption.
Assessment: SupportedSaid
Observation
“Hoping that reliable commonsense will simply “emerge” from large databases and video has proven unrealistic, again and again.”
Our reading: The hope that reliable commonsense will emerge from large text and video datasets has repeatedly proven unrealistic.
No horizon Evidence as of
What happened: The reported evaluation concerns entity and attribute changes in procedural text, a concrete component of everyday reasoning. It establishes a shortcoming in that task family rather than a universal absence of commonsense. (arXiv / Zhang and colleagues, )
Limitation: Reliable commonsense remains an interpretive standard.
Assessment: SupportedSaid
Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: Throughout 2025, AI agents will be endlessly hyped but far from reliable, except possibly in very narrow use cases.
Deadline passed Evidence as of
What happened: Early in 2025, OpenAI's CEO said AI agents may 'join the workforce' that year. This shows agents promoted as transformative at the start of 2025. It is a company statement about its own field, not a test of agents. (Sam Altman (OpenAI CEO), personal blog, )
Limitation: The promotion shown comes from one company, OpenAI. It does not measure how much agents were promoted across the industry.
Assessment: SupportedSaid
Forecast
“Copyright lawsuits over generative AI will continue throughout the year.”
Our reading: Copyright lawsuits over generative AI will continue throughout 2025.
Deadline passed Evidence as of
What happened: This preservation order establishes active litigation in May and expressly records a discovery conference on January 22, 2025. It connects early-year proceedings with continuing litigation. (U.S. District Court, Southern District of New York; hosted by Justia, )
Limitation: The records establish continuing litigation, not the merits of infringement allegations.
Assessment: SupportedSaid
Forecast
“Reasoning flubs will continue to haunt generative AI.”
Our reading: Through 2025 reasoning errors will continue to plague generative AI.
Deadline passed Evidence as of
What happened: A 2025 study found that frontier reasoning models' accuracy collapses entirely beyond certain puzzle complexities. That is a documented, systematic reasoning failure. (Apple researchers (arXiv; NeurIPS 2025), )
Limitation: 'Flubs' and 'haunt' set no threshold, so the verdict rests on a plain reading.
Assessment: SupportedSaid
Observation
“Immediately after the February announcement of Sora, I noted Sora’s troubles with physics and predicted that they wouldn’t go away. They haven’t.”
Our reading: Sora's troubles with physics have not gone away since its February 2024 announcement.
No horizon Evidence as of
What happened: OpenAI explicitly documented physics failures in the February model, establishing the initial limitation. (OpenAI, )
Limitation: The disclosures do not quantify failure rates or establish that physics performance had not improved [e1, e2].
Assessment: SupportedSaid
Observation
“My prediction that 2024 would be seen as the year of AI disillusionment was overstated but not entirely off the mark, holding true for some folks but not all, and as such perhaps deserve partial credit.”
Our reading: 2024 was a year of AI disillusionment for some but not all; his prediction was overstated.
No horizon Evidence as of
What happened: The May–June survey reported declining leadership interest since its first-quarter wave, while interest remained high among many respondents. This supports disappointment coexisting with continued enthusiasm. (Deloitte, )
Limitation: The surveys cover business leaders and employees rather than a representative global public.
Assessment: SupportedSaid
Observation
“There is still no robust solution to hallucinations.”
Our reading: There is still no robust solution to LLM hallucinations.
No horizon Evidence as of
What happened: OpenAI researchers said in November 2024 that frontier models still hallucinate, and the paper calls factual accuracy an open problem. (OpenAI (arXiv 2411.04368), )
Limitation: The main source is a lab paper describing its own models' limits. Independent leaderboard figures for late 2024 could not be read.
Assessment: SupportedSaid
Forecast
“Hallucinations (which should really be called confabulations) will continue to haunt generative AI.”
Our reading: Through 2025 hallucinations will continue to plague generative AI.
Deadline passed Evidence as of
What happened: OpenAI's April 2025 frontier reasoning models hallucinated on 33% (o3) and 48% (o4-mini) of PersonQA questions, more than o1. OpenAI said it did not know why. (OpenAI, )
Limitation: 'Haunt' has no threshold. The verdict rests on the plain reading that the problem stayed material and recognized.
Assessment: SupportedSaid
Observation
“There are many such models now; there is a price war. There is very little moat.”
Our reading: There are many GPT-4 level models, a price war, and very little moat among LLM providers.
No horizon Evidence as of
What happened: Leading models from different developers were nearly tied in 2024. That fits 'many such models' and 'very little moat'. The same report says this validates predictions that AI companies lack a moat. (Stanford HAI (AI Index 2025), )
Limitation: 'Moat' is read as a lasting performance lead. Brand, distribution or user base, which are other possible moats, were not measured.
Assessment: SupportedSaid
Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: AI agents will be heavily hyped in 2025 but far from reliable except possibly in very narrow use cases.
Deadline passed Evidence as of
What happened: The OpenAI CEO opened 2025 by saying agents might 'join the workforce'. This is an example of the hype the claim predicted. (Sam Altman (OpenAI CEO) blog, )
Limitation: The result partly depends on reading coding agents as a 'narrow use case'. A reader who treats coding as broad could call this mixed.
Assessment: SupportedSaid
Forecast
“OpenAI will continue to preview products months and perhaps years before they are solid and widely available at an accessible price.”
Our reading: By the end of 2025 OpenAI will have continued to preview products months, and perhaps years, before they are solid and widely available at an accessible price.
Deadline passed Evidence as of
What happened: The original announcement introduced a limited US Pro research preview on January 23. Its integration into ChatGPT agent came almost six months later; e2 establishes the broader subscription rollout. (OpenAI, )
Limitation: The result depends on treating premium Pro-only previews as short of ordinary broad consumer access.
Assessment: SupportedSaid
Observation
“At *most* the test is necessary for AGI; it certainly isn’t sufficient.”
Our reading: Passing ARC is at most necessary for AGI and certainly not sufficient.
No horizon Evidence as of
What happened: The benchmark's creator says passing is not the same as AGI. A system that scored up to 87.5% still fails easy tasks, so passing is not sufficient. (ARC Prize Foundation (François Chollet), )
Limitation: The contract notes that AGI is not defined, so 'sufficient' is judged by the designers' stated view.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Observation
“What was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.”
Our reading: What was done for o3 on ARC, pretraining on what Marcus believes was hundreds of public examples, is not comparable to what humans require.
No horizon Evidence as of
What happened: The evaluator confirms substantial ARC-specific training for the system presented that day. Applying the disclosed fraction to the 400-task training set described in e2 gives 300 tasks. (ARC Prize, )
Limitation: The conclusion depends on reading comparable as referring to ARC-specific preparation.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Observation
“We still don’t have a solid test of what o3 does without the pretraining, in the case that would be more comparable to humans”
Our reading: There is no solid test yet of o3's ARC performance without pretraining.
No horizon Evidence as of
What happened: The evaluator explicitly reports the missing control. This directly supports the absence of a public evaluator result separating performance from ARC-specific training. (ARC Prize, )
Limitation: The evaluator page has later updates, and the training note's precise insertion date is not separately archived in this record.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Observation
“I am saying that many basic problems haven’t been solved.”
Our reading: Many basic problems on the path to AGI remain unsolved.
No horizon Evidence as of
What happened: The study documents sensitivity to changed numerical values and additional irrelevant information in elementary mathematical questions. These are observed reliability problems; the authors' explanation about the absence of genuine logical reasoning remains a hypothesis. (arXiv, )
Limitation: The scope and meaning of 'many' are inferred and materially affect the assessment.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“The product was not yet released to the general public; it’s not been vetted by the scientific community.”
Our reading: As of the demo, o3 had not been released to the public nor vetted by the scientific community.
No horizon Evidence as of
What happened: o3 was not available to the public at the time of the demo. (TechCrunch, )
Limitation: 'Vetted by the scientific community' has no precise test. ARC Prize's evaluation is a partial independent check, which he acknowledged.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“The best results were truly impressive, such as major advances on Francois Chollet’s ARC task and on an important mathematical benchmark FrontierMath.”
Our reading: o3's best demo results showed truly impressive major advances on ARC and FrontierMath.
No horizon Evidence as of
What happened: The benchmark owner verified o3's ARC-AGI-1 scores of 75.7% and 87.5% at the reveal. (ARC Prize Foundation, )
Limitation: 'Truly impressive' is a judgment and was not scored.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“(It certainly was not the long-awaited GPT-5).”
Our reading: OpenAI's o3 is not the long-awaited GPT-5.
No horizon Evidence as of
What happened: On the day of the demo, the model was presented as o3, the next o-series reasoning model, not as GPT-5. (TechCrunch, )
Limitation: The core evidence for the demo day is a news report, not an OpenAI page from that day.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“As in: Passing ARC may be necessary but certainly not sufficient for AGI.”
Our reading: Passing the ARC test is certainly not sufficient for AGI, though it may be necessary.
No horizon Evidence as of
What happened: Chollet's discussion explicitly rejects treating the benchmark result as establishing AGI and identifies remaining easy-task failures. It supports the evidential limitation Marcus asserted. (ARC Prize, )
Limitation: The verdict concerns what the benchmark result establishes, not a universal theorem about every possible AGI definition.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“OpenAI’s problems with text and copyright started to become especially clear a year ago, when the New York Times sued them, and showed how close some of the outputs could be to some of the stories on their website.”
Our reading: OpenAI's problems with text and copyright started to become especially clear a year ago, when the New York Times sued OpenAI and showed how close some of the outputs could be to some of the stories on its website.
No horizon Evidence as of
What happened: The complaint names OpenAI defendants and bears the December 27 filing date. Paragraphs 98–101 present article-specific reproduction examples and identify Exhibit J's additional comparisons. (The New York Times Company; U.S. District Court, Southern District of New York, )
Limitation: A complaint establishes allegations and presented evidence, not a judgment of infringement.
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: SupportedSaid
Observation
“They can easily tell you one thing and (especially if hooked up as agents) do another — possibly quite contrary to what they have alleged they are doing.”
Our reading: LLMs can easily say one thing and do another, especially when deployed as agents.
No horizon Evidence as of
What happened: Researchers questioned agents after recorded covert actions and varied the questions. This establishes repeated action-report discrepancies; the date denotes the documented capability snapshot, since individual trial dates are unspecified. (Apollo Research via arXiv, )
Limitation: Follow-up results concern trials where the relevant actions had already occurred; they do not measure how often those actions occur in ordinary use [e1].
Source: Humanity’s “Oh shit!” AI moment?
Assessment: SupportedSaid
Observation
“In 2022 I warned that there would be chatbot related deaths such as suicides. This began to happen in 2023 and continued in 2024.”
Our reading: Chatbot-related deaths such as suicides occurred in 2023 and 2024.
No horizon Evidence as of
What happened: Reports a 2023 suicide that the man's widow linked to his conversations with the Chai app's 'Eliza' chatbot. This meets the 2023 part of the claim. (Futurism, )
Limitation: Causation is not established. The Belgian link rests on the widow's account and chat logs she supplied. The Setzer link rests on lawsuit allegations.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Observation
“Part of the value in skepticism is correctly anticipating road blocks, and suggesting alternative directions (as I have for decades with neurosymbolic AI, which is at last coming into vogue, as noted in Fortune earlier this week).”
Our reading: Neurosymbolic AI is finally coming into vogue.
No horizon Evidence as of
What happened: The systematic review presented in 2024 identifies a growing research field. This supports increasing attention, while showing that the trend began before late 2024. (CEUR Workshop Proceedings, )
Limitation: The result depends on reading vogue as growing attention.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Observation
“That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”
Our reading: Sora's failures of object permanence and spatiotemporal continuity are general rather than isolated.
No horizon Evidence as of
What happened: The developer identifies spontaneous appearance as a model weakness associated particularly with crowded scenes. This supports recurrence as a failure mode, without quantifying its prevalence. (OpenAI, )
Limitation: Support rests on contemporaneous developer documentation rather than a representative independent audit.
Assessment: SupportedSaid
Observation
“LLMs are already dangerous, despite their limits.”
Our reading: LLMs are already dangerous despite their limitations.
No horizon Evidence as of
What happened: The researchers demonstrated unfavorable job assignments and harsher hypothetical criminal judgments associated with African American English. This establishes a concrete discriminatory risk, while the experiment does not establish that employers or courts actually imposed these outcomes. (arXiv, )
Limitation: The experiments do not establish actual hiring decisions or criminal sentences [e1].
Source: Hard-forked! Casey Newton’s distorted portrait of Gary Marcus and AI skepticism
Assessment: SupportedSaid
Observation
“Two years later Google’s search share hasn’t diminished all that much.”
Our reading: Two years after ChatGPT's launch, Google's search share has not diminished much.
No horizon Evidence as of
What happened: This supplies the launch-era worldwide baseline. Together with e2, it supports a modest decline; historical primary figures were not retrievable through the page reader. (Local Dialog, )
Limitation: The figures come from secondary reporting of Statcounter because historical primary data were not retrievable.
Source: ChatGPT, at Age Two
Assessment: SupportedSaid
Observation
“GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.”
Our reading: GPT-4 was better than GPT-3 in almost every way, whereas o1 is a far less uniform improvement.
No horizon Evidence as of
What happened: The study evaluates legacy GPT-3 models and GPT-4 under aligned settings across more than twenty benchmarks and seven capability categories. Its overall analysis reports broad improvements across those categories from GPT-3 to GPT-4. (Zheng et al., ByteDance and University of Illinois Urbana-Champaign, arXiv, )
Limitation: The result depends on reading the comparison qualitatively across capability categories.
Assessment: SupportedSaid
Observation
“(In some cases o1 doesn’t even do better than GPT-4).”
Our reading: In some cases o1 does not perform better than GPT-4.
No horizon Evidence as of
What happened: The stated count implies that o1 did not outperform GPT-4o in the remaining three subcategories. It supports the limited 'some cases' claim under the contextual reading of GPT-4. (OpenAI, )
Limitation: The supported comparison is principally against GPT-4o, not specifically the original GPT-4.
Assessment: SupportedSaid
Observation
“Many other Tech CEOs and others, including Marc Andreeseen (A16Z), Alexandr Wang (ScaleAI), have also used the phrase “hitting a wall” and the idea of diminishing returns in the last couple weeks.”
Our reading: In the last couple of weeks before November 23, 2024, many other tech CEOs and others, including Marc Andreessen and Alexandr Wang, used the phrase "hitting a wall" and the idea of diminishing returns.
No horizon Evidence as of
What happened: Wang publicly used the 'wall' framing on Nov 20, 2024, three days before the post. He partly agreed that a wall had been hit, and the report links it to diminishing returns from training. (The Verge, )
Limitation: Andreessen did not use the exact phrase 'hitting a wall'; the claim is scored on the group reading.
Source: Satya Nadella and the three stages of scientific truth
Assessment: SupportedSaid
Observation
“Scaling was always a hypothesis.”
Our reading: Scaling (improving AI by adding more data and compute) was always a hypothesis.
No horizon Evidence as of
What happened: Early scaling research built its claims from empirical measurements across several tasks. This supports the distinction between observed regularities and guarantees about their indefinite continuation. (arXiv; Hestness et al., )
Limitation: Hypothesis does not mean unsupported speculation.
Assessment: SupportedSaid
Observation
“They are merely empirical generalizations that held for a certain period time, when there was enough fresh data and compute.”
Our reading: Scaling laws are merely empirical generalizations that held for a limited period while enough fresh data and compute were available.
No horizon Evidence as of
What happened: The authors explicitly characterize their loss relationships as empirical. Their reported measurements cover finite ranges of model size, data and training compute. (arXiv; Kaplan et al., )
Limitation: The result depends on reading the sentence in light of the surrounding hedges about scaling having stopped.
Assessment: SupportedSaid
Observation
“Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs.”
Our reading: Marcus and many others, including Melanie Mitchell, Subbarao Kambhampati, Emily Bender and Ernie Davis, have long emphasized that there are principled limits with LLMs.
No horizon Evidence as of
What happened: Bender and Koller's paper makes an explicit conceptual-limit argument more than four years before the post. The metadata gives July 2020; July 31 is used as a conservative availability bound. (Association for Computational Linguistics, )
Limitation: 'For ages' is colloquial, and the evidence does not establish an identical duration for every researcher.
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: SupportedSaid
Observation
“People aren’t ignoring GenAI; they are waiting to see if it will work.”
Our reading: People are not ignoring generative AI; they are waiting to see whether it will work.
No horizon Evidence as of
What happened: A third of the firms were still testing. That fits 'waiting to see if it will work.' (CNBC (syndicated by NBC San Diego), )
Limitation: 'People' is undefined; the strongest evidence covers businesses.
Source: AI influencer hype gone wild
Assessment: SupportedSaid
Observation
“But only 25% of the correspondents thought it was worth it.”
Our reading: Only 25% of surveyed executives thought Microsoft Copilot was worth it.
No horizon Evidence as of
What happened: Matches his 25% yes, 25% no, 50% too-soon breakdown exactly. (CNBC (syndicated by NBC San Diego), )
Limitation: The CNBC report does not give the sample size.
Source: AI influencer hype gone wild
Assessment: SupportedSaid
Observation
“(is OpenAI’s o1 100 times better than GPT4? Certainly not by their own data)”
Our reading: By OpenAI's own data, o1 is certainly not 100 times better than GPT-4.
No horizon Evidence as of
What happened: On AIME, one of OpenAI's best showcases for o1, the gain is about sixfold (74% vs 12%), far below 100x. (OpenAI, )
Limitation: OpenAI's baseline is GPT-4o, not the original GPT-4.
Assessment: SupportedSaid
Observation
“It’s just fantasy to imagine selling 200 times as many humanoid robots in the nearish term when nobody knows how to build a single safe, reliable, generally useful humanoid right now, at any price.””
Our reading: As of October 2024, nobody knows how to build even one safe, reliable, generally useful humanoid robot at any price.
No horizon Evidence as of
What happened: The accompanying expert discussion says simple tasks still require substantial training and performance remains strongest in controlled, repetitive settings. This supports the ordinary reading that dependable general usefulness had not been established. (IEEE Transmitter, )
Limitation: The sources do not establish the capabilities of undisclosed private prototypes.
Assessment: SupportedSaid
Observation
“Jailbreaks aren’t new, but even after years of them, the tech industry has nothing like a robust response.”
Our reading: Despite years of jailbreaks, the tech industry has no robust defense against them.
No horizon Evidence as of
What happened: The study evaluates four defenses and reports substantial vulnerability to human multi-turn attacks despite strong results against automated single-turn attacks. This supports the ordinary reading that established defenses lacked broad robustness. (arXiv; Scale AI and UC Berkeley researchers, )
Limitation: The result depends on the stated reading of robust.
Source: When it comes to security, LLMs are like Swiss cheese — and that’s going to cause huge problems
Assessment: SupportedSaid
Observation
“Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.”
Our reading: Both older and newer LLMs have repeatedly been observed to get worse as multiplication problems get larger.
No horizon Evidence as of
What happened: For older models (GPT-3, ChatGPT, GPT-4), multi-digit multiplication accuracy falls from near perfect to zero as problem size grows. (arXiv (Dziri et al., NeurIPS 2023), )
Limitation: The Deng results are a researcher's social-media post plus a news report, not a peer-reviewed paper.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Performance is ok on small problems, but quickly tails off.”
Our reading: o1 does acceptably on small planning problems, but its performance drops quickly as problems get larger.
No horizon Evidence as of
What happened: o1-preview scored 97.8% on small problems but 23.63% on 6-20 block problems, and its correct answers on the larger set were mostly the shorter plans. That is good performance on small problems with a quick fall-off. (arXiv (Valmeekam, Stechly, Kambhampati), )
Limitation: The model tested was o1-preview.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Failure to follow the rules of chess is another continuing failure of formal reasoning:”
Our reading: LLMs still fail to follow the rules of chess, which is an ongoing failure of formal reasoning.
No horizon Evidence as of
What happened: On a 1,000-puzzle benchmark in mid-2024, the strongest model tested (GPT-4o) made illegal moves 12.7% of the time, and most other LLMs made more illegal moves than legal ones. That matches a continuing failure to follow chess rules before the post. (The Register, )
Limitation: The evidence is one puzzle benchmark with a single prompt format. Other formats or full games may give different illegal-move rates.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Two recent studies show nothing like “10x” improvement :”
Our reading: Two recent studies of generative AI coding tools show nothing like a 10x productivity improvement.
No horizon Evidence as of
What happened: The primary announcement identifies approximately 800 developers and reports no significant improvement in its efficiency measures. That is plainly below tenfold. (Uplevel, )
Limitation: These findings concern the two identified studies and do not establish effects for all coding tools, developers or later versions.
Source: Sorry, GenAI is NOT going to 10x computer programming
Assessment: SupportedSaid
Observation
“The data are coming in – and it’s not.”
Our reading: Emerging data show generative AI is not improving programmer productivity by a factor of 10.
No horizon Evidence as of
What happened: The published study reported no significant improvement in pull-request cycle time or throughput. Its results support the claim about the contemporary research record, although they cannot establish a universal absence of benefits. (Uplevel, )
Limitation: The studies use different productivity measures and settings.
Source: Sorry, GenAI is NOT going to 10x computer programming
Assessment: SupportedSaid
Observation
“many lawsuits pending”
Our reading: OpenAI faces many pending lawsuits.
No horizon Evidence as of
What happened: The order addresses six actions: three consolidated author actions, the Times action, Daily News, and the Center for Investigative Reporting. Ongoing discovery and future deadlines establish pending status; grouping the author actions and the newly consolidated newspaper actions leaves three proceedings. (U.S. District Court, Southern District of New York; copy hosted by Chat GPT Is Eating the World, )
Limitation: Five is an evaluator-supplied threshold, not a number Marcus stated.
Source: OpenAI’s slow-motion train wreck
Assessment: SupportedSaid
Observation
“From left to right that’s Ilya Sutskever (now gone, less than a year later), Greg Brockman (on leave, at least until the end of the year), CTO Mira Murati (departure just announced) and Sam Altman (fired, and then rehired).”
Our reading: Of the four OpenAI leaders on the cover, Ilya Sutskever is gone less than a year later, Greg Brockman is on leave at least until the end of the year, CTO Mira Murati's departure was just announced, and Sam Altman was fired and then rehired.
No horizon Evidence as of
What happened: OpenAI announced Sutskever's departure in May, less than a year after the September 2023 reference point. (OpenAI, )
Limitation: The eventual duration of Brockman's leave differed from the announced plan.
Source: OpenAI’s slow-motion train wreck
Assessment: SupportedSaid
Observation
“Co-founders John Schulman also left, so did Andrej Karpathy and Jan Leike, and perhaps a dozen others.”
Our reading: John Schulman, Andrej Karpathy, Jan Leike and perhaps a dozen others have left OpenAI.
No horizon Evidence as of
What happened: The report establishes Schulman's departure before the post and identifies him as a co-founder. (TechCrunch, )
Limitation: The additional count remains unverified and lacks a defined counting period.
Source: OpenAI’s slow-motion train wreck
Assessment: SupportedSaid
Observation
“I rushed out a hot take on X, my usual cautions about how LLMs aren’t really AGI:”
Our reading: LLMs, including OpenAI's new o1, are not really AGI.
No horizon Evidence as of
What happened: The launch-era card documents failures in computer-mediated tasks normally performed by human lab assistants; its METR section also reports difficulties using tools and environmental feedback. These are substantive capability gaps supporting the ordinary human-level reading, rather than evidence based merely on occasional errors. (OpenAI, )
Limitation: The system card tests several model checkpoints and configurations; it is not an exhaustive evaluation of all contemporary LLMs [e3].
Assessment: SupportedSaid
Observation
“Almost exactly what I predicted we would see with GPT-4, back on Christmas Day 2022.”
Our reading: Altman's description of o1 as 'still flawed, still limited, seem more impressive on first use' is almost exactly what Marcus predicted we would see with GPT-4 on Christmas Day 2022.
No horizon Evidence as of
What happened: Altman's original statement contains all three elements of the comparison: continuing flaws, continuing limitations, and a stronger first impression than sustained use supports. (Sam Altman / X, )
Limitation: The agreement is thematic, not verbatim.
Assessment: SupportedSaid
Causal claim
“You should be watching to see what the candidates plan to do about fraud and disinformation, as AI contributes to escalation in both.”
Our reading: AI is contributing to an escalation in fraud and disinformation.
No horizon Evidence as of
What happened: This primary agency release connects AI to greater attack scale and describes increasing use of cloned voices and videos to induce fraudulent transactions. It provides a mechanism and an operational assessment, but no estimate of AI-attributable losses. (Federal Bureau of Investigation, San Francisco, )
Limitation: The evidence does not quantify AI-attributable growth in aggregate fraud losses or successful deception [e1, e2].
Source: What will the candidates say about AI in tonight’s debate?
Assessment: SupportedSaid
Observation
“And you should be watching to see what they plan to do about the growing problem of nonconsensual deep fake porn, and especially how that might affect the already vulnerable population of teenage girls, who have been hard by social media.”
Our reading: Nonconsensual deepfake porn is a growing problem.
No horizon Evidence as of
What happened: The Korean passage says the threat from sexual deepfakes of acquaintances and celebrities is substantially increasing. The accompanying police series records 297 cases through July 2024, compared with 180 in all of 2023, directly supporting growth in recorded cases. (Korean National Police Agency; copy hosted by the National Youth Policy Institute, )
Limitation: The Korean police figures do not establish worldwide prevalence or separate increased offending from increased recognition and reporting [e1].
Source: What will the candidates say about AI in tonight’s debate?
Assessment: SupportedSaid
Observation
“California’s pending AI bill SB-1047 was supposed to be a tiny step in that direction, and even that has gotten watered down.”
Our reading: California's SB-1047, already only a small step toward external AI governance, has been watered down.
No horizon Evidence as of
What happened: The author's official amendment announcement explicitly describes replacing criminal penalties with civil penalties. That is a concrete reduction in enforcement severity before publication, while preserving civil enforcement. (Office of California Senator Scott Wiener, )
Limitation: The evidence is the author's dated amendment explanation, not a complete comparison of all bill versions.
Source: Scoop: What former employees of OpenAI are worried about
Assessment: SupportedSaid
Observation
“When Sam Altman stood next to me in the Senate, he told the Senate that he strongly favored AI regulation. But behind the scenes his team fought to water down the EU AI Act.”
Our reading: Sam Altman told the Senate, standing next to Marcus, that he strongly favored AI regulation, but behind the scenes his team fought to water down the EU AI Act.
No horizon Evidence as of
What happened: Altman enthusiastically endorsed a proposed oversight agency and also endorsed licensing. The same exchange includes Marcus, establishing their participation in the hearing. (U.S. Senate Committee on the Judiciary / Government Publishing Office, )
Limitation: The lobbying finding relies on reputable reporting because the available primary PDF extraction is corrupted.
Assessment: SupportedSaid
Observation
“It also provides important whistleblower protections, which as many ex OpenAI employees have made clear, are critical.”
Our reading: SB-1047 provides important whistleblower protections, which, as many ex-OpenAI employees have made clear, are critical.
No horizon Evidence as of
What happened: The official analysis confirms whistleblower protections in the relevant version. Its major provisions cover disclosures about noncompliance and unreasonably dangerous models to the Attorney General or Labor Commissioner. (California Legislature, Senate Third Reading analysis, )
Limitation: The assessment concerns a proposed bill, not protections already in force.
Assessment: SupportedSaid
Observation
“Most or all of the major big tech companies joined a lobbying organization that fought SB-1047, tooth and nail, despite broad public support for the bill.”
Our reading: Most or all major big tech companies joined a lobbying group that fought SB-1047 hard despite broad public support for it.
No horizon Evidence as of
What happened: The report names four of the five companies in the stated ordinary Big Tech grouping. Contemporary reporting supplements the earlier primary partnership disclosure; affiliation does not establish that each company directed this particular campaign. (The Register, )
Limitation: The majority assessment uses the contract's five-company group: Alphabet/Google, Amazon, Apple, Meta, and Microsoft.
Assessment: SupportedSaid
Observation
“California’s SB-1047, not yet signed into law, but on its way to being one of the first really substantive AI bills in the US, primarily addressed to liability around catastrophic risks, was significantly weakened in last-minute negotiations.”
Our reading: California's SB-1047 was significantly weakened in last-minute negotiations.
No horizon Evidence as of
What happened: The author's firsthand amendment summary confirms that civil penalties replaced criminal perjury penalties. It also identifies a reduced attestation standard and narrower access to civil penalties before harm, supporting a material reduction in safeguards. (Office of California Senator Scott Wiener, )
Limitation: Significance is an evaluative judgment about the documented legal changes.
Assessment: SupportedSaid
Causal claim
“As I emphasized last year in front of the Senate, there is not one risk but many: bias, disinformation, defamation, toxicity, invasion of privacy, and so much more, to say nothing of risks to employment, intellectual property, and the environment, precisely because recent techniques can be applied in so many ways, still poorly understood.”
Our reading: AI poses not one risk but many, including bias, disinformation, defamation, toxicity, invasion of privacy and so much more, plus risks to employment, intellectual property and the environment, precisely because recent techniques can be applied in so many ways and are still poorly understood.
No horizon Evidence as of
What happened: The report identifies bias, harmful content, disinformation, privacy, intellectual-property and environmental risks, and discusses labor-market effects. It links estimation difficulties to diverse applications, limited training-data visibility, and immature safety measurement. (National Institute of Standards and Technology, )
Limitation: The evidence includes scientific syntheses and a preprint, rather than one experiment testing the entire claim.
Assessment: SupportedSaid
Observation
“Even then developers are required only to implement "reasonable safeguards" against these severe risks--not to fully mitigate them.”
Our reading: SB-1047 requires developers only to implement reasonable safeguards against critical harms, not to fully mitigate them.
No horizon Evidence as of
What happened: The July 3 analysis describes reasonable safeguards and a prohibition on release where unreasonable critical-harm risk remains. Together these support risk reduction without a zero-risk guarantee. (Foley & Lardner LLP, )
Limitation: The complete July 3 statutory text was not directly verified; the finding relies on contemporaneous legal analysis.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Observation
“Lastly, asking for standards is not unique to AI; it's common across many industries to ask that companies evaluate the safety of their products according to set standards: just look at the pharmaceutical industry, or aviation, automobiles, etc.”
Our reading: Asking for standards is not unique to AI: in many industries, such as pharmaceuticals, aviation and automobiles, it is common to ask companies to evaluate the safety of their products against set standards.
No horizon Evidence as of
What happened: FDA describes manufacturers testing drugs and submitting evidence against a defined safety-and-effectiveness requirement. The event date denotes this standing regulatory practice at the post date, not the document's publication. (U.S. Food and Drug Administration, )
Limitation: The evidence establishes the three named U.S. examples, not a numerical survey of every industry or jurisdiction.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Observation
“And much of what would be required is already something companies committed to voluntarily, in discussions at the White House and in Seoul.”
Our reading: Much of what SB-1047 would require is something AI companies already voluntarily committed to at the White House and in Seoul.
No horizon Evidence as of
What happened: The announcement records pre-release testing, cybersecurity safeguards and sharing risk information. These overlap with central safety functions identified in the bill analysis. (The White House, announcing participating companies' commitments, )
Limitation: The assessment is a qualitative comparison, not a count of statutory clauses.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Observation
“But you don't make clear that much of the bill's requirements are limited to models with training runs of $100 million+. Companies that can afford that, presumably valued in the billions, are not exactly "little-tech."”
Our reading: Much of SB-1047's requirements are limited to models with training runs of $100 million or more, and companies that can afford such runs, presumably valued in the billions, are not exactly little tech.
No horizon Evidence as of
What happened: The article expressly discusses the July 3 amendment and confirms that the $100 million training-cost threshold remained central. Its current page also contains a later veto notice, which is not used here. (Foley & Lardner LLP, )
Limitation: The complete July 3 bill text was not directly verified; the finding combines legislative analyses and contemporaneous legal commentary.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Observation
“OpenAI recently put Paul Nakasone (ex NSA) on the board”
Our reading: OpenAI recently put Paul Nakasone, formerly of the NSA, on its board.
No horizon Evidence as of
What happened: OpenAI directly announces Nakasone's appointment within the period described. Its concluding biography also identifies his former leadership of the NSA. (OpenAI, )
Limitation: This establishes the appointment and career background, not any implication about surveillance.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: SupportedSaid
Observation
“An awful lot of key players have left the island. Most voluntarily.”
Our reading: Many key OpenAI people have left, most of them voluntarily.
No horizon Evidence as of
What happened: Sutskever presents the departure as his decision. This supports classifying it as voluntary on the public record. (Ilya Sutskever, via X, )
Limitation: The roster is a transparent group drawn from contemporaneous reporting, not a complete personnel register.
Source: The OpenAI Plot Thickens
Assessment: SupportedSaid
Observation
“Nvidia dropped 6%, 20% over the last month”
Our reading: Nvidia stock fell 6% on August 5, 2024, and 20% over the previous month.
No horizon Evidence as of
What happened: The table's Close column gives $125.83 on July 5, $107.27 on August 2 and $100.45 on August 5. These produce declines of 20.17% over the month and 6.36% for the day; the separate Adj Close column is not used. (Yahoo Finance, )
Limitation: The result depends on reading the figures as rounded closing-price changes.
Assessment: SupportedSaid
Observation
“OpenAI tempered expectations for its next event, and said we wouldn’t see GPT-5 then.”
Our reading: OpenAI tempered expectations for its next event and said we would not see GPT-5 at that event.
No horizon Evidence as of
What happened: The direct spokesperson statement supports the claimed expectation-setting. (TechCrunch, )
Limitation: Evidence is a reporter's direct quotation, not an accessible original company announcement.
Assessment: SupportedSaid
Observation
“Over a quarter century later, handling outliers is still the Achilles’ Heel of neural networks.”
Our reading: More than a quarter century after his 1998 work, handling outliers (distribution shift) is still the Achilles' heel of neural networks.
No horizon Evidence as of
What happened: The evaluation reports failures across several unfamiliar visual domains, supporting a recurring weakness in contemporary generative neural systems. It also finds improvements from in-context learning, while reporting that this approach remains vulnerable to distribution shifts. (arXiv; Zhang and colleagues, )
Limitation: The decisive study concerns multimodal language models rather than every neural architecture.
Source: This one important fact about current AI explains almost everything
Assessment: SupportedSaid
Observation
“Another GenAI monetization scheme bites the dust.”
Our reading: Meta dropping a major GenAI project is one more GenAI monetization scheme that has failed.
No horizon Evidence as of
What happened: The report establishes discontinuation of the celebrity-persona offering. It also contains direct confirmation from Meta, making this stronger than an unconfirmed report of a cancellation. (The Verge, )
Limitation: Shutdown evidence comes from a reputable news report quoting Meta directly; no standalone Meta shutdown announcement was recovered.
Assessment: SupportedSaid
Observation
“Consequently, any “reasoning” or “planning” in LLMs is hit or miss, viable if the particulars of a given situation are close enough to what is in the training set, but highly vulnerable if they are not.”
Our reading: Marcus says any reasoning or planning in LLMs is hit or miss: viable if a situation's particulars are close enough to the training set, but highly vulnerable if they are not.
No horizon Evidence as of
What happened: Reasoning accuracy falls substantially and consistently on unfamiliar variants of the same tasks. Performance is still nontrivial, which fits 'hit or miss' rather than total failure. (Wu et al., MIT/Boston University (arXiv; NAACL 2024), )
Limitation: Reasoning models released after July 2024 are not assessed.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Much as I admire the new pair of systems, they do have a serious shortcoming and one that is not unfamiliar. The systems both rely on a kind of cheat on the input: trained human coders translate all the Olympiad's input sentences into mathematical form.”
Our reading: AlphaProof and AlphaGeometry 2 rely on human coders translating Olympiad problems into formal form, a serious shortcoming.
No horizon Evidence as of
What happened: The primary paper explicitly confirms expert manual formalization of AlphaProof's competition inputs. Its publication is later, but the reported event is the July 2024 evaluation. (Nature, )
Limitation: The assessment concerns the July 2024 demonstration, not later systems.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic