Supported claims by Gary Marcus, page 5 of 8
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Supported claims by Gary Marcus (731)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 5 of 8: claims 401 to 500 of 731, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Observation
“Nonetheless, GDM's new work is powerful proof of concept (pun intended!). Neural networks and symbol systems can be profitably combined, contra Hinton and LeCun.”
Our reading: Google DeepMind's work shows neural networks and symbolic systems can be profitably combined.
No horizon Evidence as of
What happened: The developer's account connects the actual IMO proofs to a transformer that proposes Lean tactics, search over proof states and automatic formal verification. It explicitly identifies the described work as ending in July 2024. (Julian Schrittwieser, AlphaProof coauthor, )
Limitation: Technical usefulness does not imply financial profitability.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Hallucinations and boneheaded errors of reasoning and what I like to call discomprehensions are, as far as I can tell, an inevitable byproduct of LLMs; certainly they have yet to disappear, despite many promises to the contrary.”
Our reading: Marcus says that, as far as he can tell, hallucinations, boneheaded reasoning errors and 'discomprehensions' are an inevitable byproduct of LLMs, and that they certainly have not yet disappeared despite many promises to the contrary.
No horizon Evidence as of
What happened: In mid-2024, LLM products whose providers claimed they 'avoid' hallucinations still hallucinated on 17% to 34% or more of test queries. (Stanford HAI / RegLab, )
Limitation: Only the 'yet to disappear' part is scored.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Long-term adoption (vs mere experimentation) has been modest, there is no significantly better tech now than there was in January; GPT-5 is still a no show.”
Our reading: GPT-5 had not been released as of July 2024.
No horizon Evidence as of
What happened: The official July release history records GPT-4o mini, not GPT-5. This supports the chronology alongside explicit reporting that GPT-5 remained forthcoming; this entry alone would not prove nonrelease. (OpenAI, )
Limitation: This establishes public release status, not the state of private development.
Assessment: SupportedSaid
Causal claim
“As Ernie Davis and I pointed out in Rebooting AI, five years ago, part of the reason we are struggling with AI in complex AI systems is that we still lack adequate techniques for engineering complex systems. That hasn’t really changed.”
Our reading: Struggles with complex AI systems stem partly from a lack of adequate techniques for engineering complex systems, which has not really changed since 2019.
No horizon Evidence as of
What happened: The researchers explain that model scale and structure make established software testing and verification methods difficult to apply. They connect that methodological limitation to the difficulty of ensuring deployment robustness. (DeepMind, )
Limitation: The causal assessment synthesizes technical explanations; it is not a controlled estimate of engineering's contribution to failures.
Source: Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call
Assessment: SupportedSaid
Forecast
“As of today, I am more confident than ever that GPT-5 won’t land this year.”
Our reading: OpenAI will not release GPT-5 in 2024.
Deadline passed Evidence as of
What happened: OpenAI introduced GPT-5 on August 7, 2025. A model first introduced in August 2025 was not released in 2024. (OpenAI, )
Limitation: This record assesses only the release timing of a model named GPT-5. It does not assess Marcus's reasons for the forecast or what GPT-5 could do.
Assessment: SupportedSaid
Forecast
“As of today, I am more confident than ever that GPT-5 won’t land this year.”
Our reading: GPT-5 will not be released in 2024.
Deadline passed Evidence as of
What happened: OpenAI's initial GPT-5 announcement is dated August 7, 2025, and describes rollout beginning that day. This places its public arrival after the forecast's 2024 deadline. (OpenAI, )
Limitation: This assesses the named public release, not private prototypes or differently named models.
Assessment: SupportedSaid
Observation
“Exhibit A: OpenAI’s CTO Mira Murati just publicly acknowledged what I long suspected: there is no mind blowing GPT-5 behind the scenes as of yet.”
Our reading: As of June 2024, OpenAI has no mind-blowing GPT-5 model waiting behind the scenes.
No horizon Evidence as of
What happened: The contemporaneous report attributes a modest internal capability lead to OpenAI's CTO. This supports the ordinary qualitative reading, although it does not independently measure private models or identify one as GPT-5. (The Decoder, )
Limitation: The decisive source is contemporaneous news reporting because the relevant primary interview transcript could not be recovered.
Assessment: SupportedSaid
Observation
“In many ways, it is much weaker than what Altman and I talked about at the Senate last year.”
Our reading: SB-1047 is in many ways much weaker than the AI regulation Marcus and Sam Altman discussed at the Senate in 2023.
No horizon Evidence as of
What happened: Marcus proposed FDA-like review before deployment. The same transcript records Altman's proposed licensing agency, license revocation, safety standards, and independent audits; it establishes the comparison rather than proving Marcus's later assessment by repetition. (U.S. Senate / Government Publishing Office, )
Limitation: The result depends on reading strength through the licensing and review mechanisms highlighted in the surrounding passage.
Assessment: SupportedSaid
Observation
“Two weeks later, it has already come out that the database that indexes the whole thing is in plain text (except on systems that are explicitly set to encrypt everything). That’s bananas. Keystone Kops level security.”
Our reading: It has come out that the database indexing Microsoft Recall's screenshots is stored in plain text, except on systems explicitly set to encrypt everything, which Marcus calls bananas and Keystone Kops level security.
No horizon Evidence as of
What happened: The preserved June documentation describes the unencrypted database and demonstrates extracting and searching its screenshot-derived text. Its June 6 example falls within the target period; June 8 is the documentation's stated FAQ update date. (Alexander Hagenah, )
Limitation: The tests concern an early preview implementation; they do not establish the protections in later Recall releases [e1, e2].
Assessment: SupportedSaid
Observation
“For one thing, the bad press about Sam Altman and OpenAI, who once seemingly could do no wrong, just keeps coming.”
Our reading: Negative press coverage of Sam Altman and OpenAI, once seen as able to do no wrong, keeps coming.
No horizon Evidence as of
What happened: This critical newsletter discusses the voice dispute, safety-team departures, and restrictive employee agreements. The event being assessed is publication of critical coverage; available_at conservatively uses the displayed modification date. (The Guardian, )
Limitation: This is a finding about the existence of repeated coverage, not its prevalence or rate of increase.
Source: AI as Reality Television
Assessment: SupportedSaid
Observation
“For all the daily claims of “exponential progress”, and for all the tens of billions that have been invested since, reliability is still a dream, and of course toxicity and hallucinations are still with us.”
Our reading: Despite tens of billions of dollars invested since 2022, deep learning systems are still unreliable and still produce toxicity and hallucinations.
No horizon Evidence as of
What happened: The May evaluation documents invented constraints and false claims of task completion, directly supporting continuing hallucinations and unreliable autonomous execution. The report also records successful shorter tasks and strong scientific-question performance, so the failures do not imply universal incompetence. (UK AI Safety Institute, )
Limitation: The reliability phrase is rhetorical and supplies no threshold.
Assessment: SupportedSaid
Observation
“The ridicule is of me; the failure is of the lords of deep learning to resolve what ails their AI.”
Our reading: The lords of deep learning have failed to resolve what ails their AI.
No horizon Evidence as of
What happened: The leading lab says its flagship model, released within the period, still hallucinates and is not fully reliable. (OpenAI, )
Limitation: 'What ails their AI' is not defined in the sentence. The grade uses the problems the post names, mainly hallucinations and reliability.
Assessment: SupportedSaid
Observation
“OpenAI called itself open, and traded on the notion of being open, but even as early as May 2016 knew that the name was misleading.”
Our reading: OpenAI traded on being 'open' while knowing since May 2016 that the name was misleading.
No horizon Evidence as of
What happened: The report describes Amodei telling Altman and Brockman that people expected OpenAI to release its source code. Altman acknowledged that this exceeded its plans and discouraged correcting the impression; this is firsthand reporting, not an internal email. (The New Yorker, )
Limitation: The decisive May conversation comes from contemporaneous journalism, not a released internal email.
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: SupportedSaid
Observation
“Altman once (in 2016) promised that outsiders would play an important role in the company’s governance; that key promise has not been kept.”
Our reading: Altman's 2016 promise that outsiders would play an important role in OpenAI governance has not been kept.
No horizon Evidence as of
What happened: The adjacent line 7a explanation assigns director elections to these members. The filing therefore describes an internally selected board, not an electorate drawn from the public. (OpenAI Inc., copy hosted by Gwern, )
Limitation: The finding concerns public electoral participation, not the absence of independent directors.
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: SupportedSaid
Observation
“OpenAI has trained on a massive amount of copyrighted material, without consent, and in many instances without compensation.”
Our reading: OpenAI has trained on a massive amount of copyrighted material without consent and often without compensation.
No horizon Evidence as of
What happened: The section documents web and book corpora, including roughly 400 billion tokens of filtered Common Crawl data. It establishes training scale, but does not itself establish the copyright or licensing status of every work. (OpenAI researchers, published on arXiv, )
Limitation: The conclusion combines company disclosures with a rights holder’s filing; the filing is not a court finding.
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: SupportedSaid
Observation
“What the LLM does is more akin to what some high school students do when they plagiarize: change a few words here or there, while still sticking close to the original.”
Our reading: What the LLM in the glue-pizza example does is more akin to what some high school students do when they plagiarize: changing a few words while still sticking close to the original.
No horizon Evidence as of
What happened: The article dates the shared screenshot to May 23 and transcribes the generated sentence. Compared with the reproduced source in e8, it retains the quantity, glue-in-sauce recommendation, and adhesive purpose with changed wording. (Hindustan Times, )
Limitation: The supported scope is the supplied example, not LLM behavior generally.
Assessment: SupportedSaid
Observation
“Many people are reaching essential the same perceptions.”
Our reading: Many people are reaching essentially the same perceptions of Sam Altman.
No horizon Evidence as of
What happened: Merchant's own commentary criticizes Altman's conduct as unusually entitled and overbearing. It is direct evidence of his perception, not independent verification of the allegations he discusses. (Blood in the Machine — Brian Merchant, )
Limitation: The undefined word 'many' makes the result depend on a qualitative reading.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Observation
“In November, the narrative was about how everybody loved Sam. Things have changed. A lot.”
Our reading: The narrative about Sam Altman, which in November was that everybody loved him, has changed a lot.
No horizon Evidence as of
What happened: WIRED reported that roughly 95% of employees had signed the letter supporting his return. This documents the prominent November narrative of overwhelming workplace support, although it does not measure public affection. (WIRED, )
Limitation: The result depends on reading 'narrative' as prominent coverage and commentary, rather than measured population sentiment.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Observation
“I shared some of my own thoughts about which Pandora’s Boxes had (LLMs) and had not (AGI) already been opened.”
Our reading: The Pandora's Box of LLMs has already been opened.
No horizon Evidence as of
What happened: Meta's release announcement offered downloadable Llama 3 models and described access through its public assistant. This establishes public availability before the post. (Meta, )
Limitation: Public availability does not establish the scale of malicious use or prove that all safeguards can be defeated.
Source: Jane Goodall, AI Skeptic
Assessment: SupportedSaid
Observation
“You might think that if OpenAI said, hey, we don’t want our software used for military purposes, Microsoft would respect that and say, ok cool, but they haven’t.”
Our reading: Microsoft has not respected OpenAI's stated wish that its software not be used for military purposes.
No horizon Evidence as of
What happened: The report places this Microsoft presentation at an October 2023 Defense Department seminar, when OpenAI's policy prohibited military use. Microsoft acknowledged the proposed application but said it had not begun implementing it. (The Intercept, )
Limitation: The result depends on reading respect as consistency with a public position, rather than contractual compliance.
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: SupportedSaid
Observation
“Playing the field with Mistral wasn’t enough; now he is building a really big model, big enough to compete directly with GPT-4, internally:”
Our reading: Beyond partnering with Mistral, Microsoft is building a large model in-house that is big enough to compete directly with GPT-4.
No horizon Evidence as of
What happened: The original article identifies MAI-1 as Microsoft's internal project and describes its intended competitive class. Its underlying source is The Information, rather than an independent technical evaluation. (The Decoder, )
Limitation: Support depends on reading compete as intended competitive positioning.
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: SupportedSaid
Observation
“You might think that after Satya offered to rescue Sam after Sam got fired, that Sam wouldn’t want to compete directly with Microsoft, but they are.”
Our reading: Sam Altman's OpenAI competes directly with Microsoft, even though Satya Nadella offered to rescue Altman after he was fired.
No horizon Evidence as of
What happened: The reported pitches present OpenAI's enterprise offering as an alternative purchase for existing Microsoft customers. That satisfies the ordinary commercial meaning of direct competition. (Reuters, republished by Yahoo Finance, )
Limitation: The customer meetings are documented by reputable reporting based on attendees, rather than public meeting records.
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: SupportedSaid
Forecast
“People who buy Altman’s narrative will continue to invest.”
Our reading: People who buy Altman's narrative will continue to invest.
Settled before its deadline Evidence as of
What happened: SoftBank explicitly endorses OpenAI's role in achieving advanced general intelligence. This establishes the belief condition under the stated public-endorsement interpretation. (SoftBank Group, )
Limitation: Public endorsement is a proxy for belief.
Source: The Sam Altman Playbook
Assessment: SupportedSaid
Observation
“At the same talk at Stanford, Altman grandoisely asserted, with no qualification, that “we’re making AGI”, without touching on the possibility that current approaches might not get us there — despite (a) the enormous known problems with reliability, reasoning, planning and hallucinations and (b) the fact that the entire field, despite being massively well-capitalized, has struggled for over year (and in fact for decades) to solve any of these problems.”
Our reading: Current AI approaches have enormous known problems with reliability, reasoning, planning and hallucinations, and the entire field, despite being massively well-capitalized, has struggled for over a year, and in fact for decades, to solve any of these problems.
No horizon Evidence as of
What happened: OpenAI explicitly acknowledged three named limitations more than a year before Marcus's post. The same section also reports reduced hallucination and improved factuality, so it supports unresolved problems rather than an absence of progress. (OpenAI, )
Limitation: The conclusion depends on reading solve as general resolution, rather than improvement or success on bounded tasks.
Source: The Sam Altman Playbook
Assessment: SupportedSaid
Causal claim
“The very process that LLMs use to generalize also creates hallucinations.”
Our reading: The same process LLMs use to generalize is what creates their hallucinations.
No horizon Evidence as of
What happened: Hallucination follows from statistical prediction itself, even with ideal data. That matches the claim that the generalizing process creates hallucinations. (arXiv / STOC 2024 (Kalai and Vempala), )
Limitation: The theory covers rare, arbitrary facts most clearly. Other kinds of hallucination may have other causes.
Assessment: SupportedSaid
Forecast
“LLMs are well on their way to becoming a commodity; price wars will be intense.”
Our reading: LLMs are becoming a commodity and price wars among providers will be intense.
Settled before its deadline Evidence as of
What happened: Reuters reports Alibaba's steep cuts, Baidu making two models free hours later, and ByteDance's earlier aggressive pricing. This directly establishes competitive price cutting rather than merely cheaper computation; reputable reporting is used for the response sequence. (Reuters, republished by Yahoo Finance, )
Limitation: Commoditization is assessed as a direction of travel, consistent with well on their way.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: SupportedSaid
Observation
“Adding visual data to LLMs hasn’t solved hallucinations.”
Our reading: Adding visual data to LLMs has not solved hallucinations.
No horizon Evidence as of
What happened: The developer of GPT-4V, the main model that added vision to an LLM, reports that it was still prone to hallucinations. (OpenAI, )
Limitation: 'Solved' is not defined. The claim is read as hallucinations remaining a material problem, not as adding vision bringing no improvement at all. The evidence packet notes that the GPT-4V system card reports fewer hallucinations during a beta, but that passage is not in the cited excerpt.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Observation
“I also urged the world to focus on global AI governance, and am pleased to see some genuine progress in that direction, especially led by the UN and the Council of Europe.”
Our reading: There has been genuine progress on global AI governance, led especially by the UN and the Council of Europe.
No horizon Evidence as of
What happened: The UN's own account records adoption without a vote of an AI resolution addressing human rights and governance frameworks. Its broad international backing establishes a concrete multilateral milestone. (United Nations in Türkiye, )
Limitation: Institutional progress is distinct from demonstrated practical effectiveness.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Observation
“Yejin Choi worried that LLMs were not reliable. They still aren’t.”
Our reading: LLMs remained unreliable as of April 2024, a year after Choi's TED talk.
No horizon Evidence as of
What happened: The March evaluation covers thirteen models from the Gemini, GPT, Claude and PaLM-2 families and documents continuing factuality limitations. It also finds better performance among larger models, so it supports continued unreliability without supporting stagnation. (arXiv; Jerry Wei and colleagues, )
Limitation: The finding concerns general factual dependability, not every task or application.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Causal claim
“The Humane AI pin/virtual assistant just doesn’t work well, and a lot of the problem is the underlying AI, which I believe is powered by OpenAI.”
Our reading: The Humane AI pin just doesn't work well, and a lot of the problem is its underlying AI, which he believes is powered by OpenAI.
No horizon Evidence as of
What happened: The launch review documents a false answer about a California food ban and a mistaken temple identification. These failures directly impair the question-answering function, although the review also identifies missing features and hardware problems. (WIRED, )
Limitation: The causal conclusion is qualitative and rests on documented product behavior, not a controlled component analysis.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Observation
“Eliezer Yudkowsky warned that AI was (inevitably!) going to kill us all. I am happy to report that so far it hasn’t.”
Our reading: So far, AI has not killed humanity as Yudkowsky warned.
No horizon Evidence as of
What happened: The governing body's report documents living humans competing in the Boston Marathon on April 15. Their existence logically establishes that AI had not killed all humanity by then. (World Athletics, )
Limitation: This assesses neither future AI risk nor Marcus’s characterization of Yudkowsky’s position.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Observation
“What it actually shows is some improvements, mostly modest, over the last few months for a bunch of different measures.”
Our reading: OpenAI's GPT-4 Turbo graph shows only mostly modest improvements over recent months.
No horizon Evidence as of
What happened: In the assistant configuration, April-minus-January changes are 4.5 DROP F1 points and 7.9, 8.9, 4.5, 1.3, and 1.6 percentage points on GPQA, MATH, MGSM, MMLU, and HumanEval. The ChatGPT configuration likewise has small changes on most measures, with larger GPQA and MATH gains and a slight HumanEval decline. (OpenAI, )
Limitation: Modest is a qualitative judgment, and relative changes can look larger.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Observation
“I think it’s fair to say that they have not. GPT-Turbo released this week still produces its share of groaners, like this interchange Phil Libin just sent:”
Our reading: Hallucinations and stupid errors in LLMs have not been solved; GPT-4 Turbo still makes them.
No horizon Evidence as of
What happened: The tests used the newly released snapshot on 133 Python exercises, allowing two attempts. Unsuccessful tasks demonstrate that coding errors persisted; this is not a measurement of factual hallucinations. (Aider, )
Limitation: The coding test measures task completion, not hallucination frequency.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Observation
“Already in recent weeks, Inflection AI largely closed shop, Stability AI is struggling, and LLM-based autonomous vehicle company called Ghost closed shop, and a software engineer on YouTube raised quite serious questions about the wildly-hyped AI coding system Devin.”
Our reading: In recent weeks Inflection AI largely closed, Stability AI is struggling, Ghost closed, and serious questions were raised about Devin.
No horizon Evidence as of
What happened: The regulator found that almost all of the team transferred, including its core expertise. This supports substantial dismantling of the original organization, under the stated interpretation. (UK Competition and Markets Authority, )
Limitation: The Inflection finding depends on the meaning of 'largely'; the company continued operating.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Observation
“Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns.”
Our reading: MMLU gains from GPT-4 to GPT-4 Turbo are small compared with earlier generational jumps, tentatively evidencing diminishing returns.
No horizon Evidence as of
What happened: Table 1 reports 32.4% for fine-tuned GPT-2 and 43.9% for GPT-3 X-Large evaluated few-shot, a reported increase of 11.5 percentage points. The different evaluation methods prevent treating this as a controlled comparison of unchanged model preparation. (Dan Hendrycks and coauthors, arXiv, )
Limitation: The records do not compare all four models under one protocol: GPT-2 was fine-tuned, GPT-3 used few-shot prompting, GPT-4 used five-shot prompting, and Turbo used zero-shot chain-of-thought prompting [e1, e2, e3].
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Forecast
“Skeptical of this, I offered Musk a $1 million bet that AGI superior to the smartest individual humans would not arrive by the end of 2025, reprinted here for those who missed it.”
Our reading: AGI superior to the smartest individual humans will not arrive by the end of 2025.
Deadline passed Evidence as of
What happened: The retrospective report says every ARC-AGI-2 task was solved by at least two independent non-expert human testers. Its analysis identifies continuing generalization limitations in 2025 AI systems. (ARC Prize Foundation, via arXiv, )
Limitation: The evidence does not inventory undisclosed systems or every system and inference configuration available by the deadline.
Source: Superhuman AGI is not nigh
Assessment: SupportedSaid
Observation
“Tesla FSD is getting better. But is getting enough better?”
Our reading: Tesla FSD is getting better, and he raises the question of whether it is getting enough better.
No horizon Evidence as of
What happened: Crowdsourced tracker data as of early April 2024 showed v12.3 with far more miles between critical disengagements than v11 or v10, a measurable improvement just before the post. (NextBigFuture, )
Limitation: The tracker figures in e1 come from crowdsourced, self-reported data. e1 notes that data for later v12.3.x releases still needed weeks to be entered.
Source: 24 hours in AI news
Assessment: SupportedSaid
Observation
“current models aren’t literally look up tables; they can generalize to some degree, but not enough.”
Our reading: Current AI models can generalize to some degree but not enough.
No horizon Evidence as of
What happened: The study evaluates 25 LLMs and four prompting techniques. It reports resilience to some numerical and wording changes alongside substantial failures under other mathematical perturbations, supporting the contextual claim of partial but unreliable generalization. (arXiv; Li and colleagues, )
Limitation: The outcome depends on interpreting 'not enough' through the surrounding discussion of reliability.
Assessment: SupportedSaid
Observation
“Neural networks (at least in the configurations that have been dominant over the last three decades) have trouble generalizing beyond the multidimensional space that surrounds their training examples.”
Our reading: Neural networks in the configurations dominant for the last three decades have trouble generalizing beyond the space surrounding their training examples.
No horizon Evidence as of
What happened: WILDS reports performance losses across ten datasets involving natural distribution shifts. This supports recurring generalization difficulty across applications, without establishing a universal inability. (arXiv; Koh and colleagues, )
Limitation: The outcome depends on reading the spatial language as unfamiliar conditions rather than a precise geometric boundary.
Assessment: SupportedSaid
Observation
“The good news is that word is finally getting out more broadly.”
Our reading: Awareness of the danger generative AI poses to the information ecosystem is finally spreading more widely.
No horizon Evidence as of
What happened: A major mainstream body said AI-driven misinformation dominated its 2024 risk outlook, three months before the post. That shows the concern had reached the mainstream. (World Economic Forum, )
Limitation: The trend ('more broadly') is inferred from high levels, not from a measured before-and-after comparison.
Assessment: SupportedSaid
Observation
“Meanwhile X itself is promoting nonsense.”
Our reading: X itself is promoting nonsense.
No horizon Evidence as of
What happened: Mashable directly observed the false story and documented its placement in Explore and the homepage sidebar. This establishes amplification by X's own features during the target period. (Mashable, )
Limitation: The assessment relies on direct news reporting with screenshots and an independent fact-check; no separately retrievable original X display was obtained.
Assessment: SupportedSaid
Causal claim
“Ironic that his own chatbot, Grok, is making the problem worse.”
Our reading: Musk's own chatbot, Grok, is making the information-pollution problem worse.
No horizon Evidence as of
What happened: Grok wrote a false headline that X promoted to users, a direct addition to information pollution. (Mashable, )
Limitation: The false Iran story began with user posts; Grok's added effect on spread is not separately measured.
Assessment: SupportedSaid
Observation
“The bad news, which outweighs the good, is that the problem is quickly getting worse.”
Our reading: Generative-AI pollution of the information ecosystem is getting worse quickly as of April 2024.
No horizon Evidence as of
What happened: A more than tenfold rise in AI content-farm sites in about seven months is fast growth in AI information pollution. (The Guardian (Nigeria), reporting NewsGuard data, )
Limitation: Tracker counts come from one commercial source and may partly reflect more searching rather than only more sites.
Assessment: SupportedSaid
Observation
“The trouble is that the bigger the model, the higher the internal costs. And current-sized models, already expensive, aren’t reliable enough for many jobs.”
Our reading: Bigger models have higher costs, and current-sized models are already expensive and not reliable enough for many jobs.
No horizon Evidence as of
What happened: The contemporaneous engineering assessment supports the general size-resource relationship and describes substantial deployment demands. It also explains that sparse architectures can weaken the relationship between total parameters and computation. (arXiv, )
Limitation: The result depends on comparing otherwise similar implementations and interpreting reliability as suitability for unverified use.
Assessment: SupportedSaid
Observation
“As noted here before, one study indicates that code quality is going down.”
Our reading: One study indicates code quality is going down amid AI-assisted coding.
No horizon Evidence as of
What happened: The study reports declining maintainability indicators in code authored through December 2023. This directly supports the limited assertion that one study indicated declining quality. (GitClear, )
Limitation: The study uses indirect quality indicators.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Observation
“On the other hand, the use cases on the negative side of the ledger are proliferating fast, literally faster than I can report them here, from fake books (undermining authors) to rings of fake websites designed to sell ads”
Our reading: Harmful uses of generative AI (fake books, ad-farm websites, election misinformation, polluted science) are proliferating rapidly.
No horizon Evidence as of
What happened: The report records 713 identified AI-generated sites, compared with fewer than 50 in May and around 600 in December 2023, and describes advertising incentives. These are discovery counts, not a controlled census of site creation. (The Guardian Nigeria, )
Limitation: Support depends on reading proliferation as broad expansion rather than proven rapid growth in every category.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Observation
“Some core problems have lingered, particularly hallucinations, instability and occasional nonsensical errors.”
Our reading: Hallucinations, instability and occasional nonsensical errors remain unsolved in GPT-4 class LLMs.
No horizon Evidence as of
What happened: The experiments demonstrate false supporting assertions that GPT-4 can recognize as wrong when asked separately. They establish hallucination and elementary-error failure modes, not a population-wide error rate. (Zhang and colleagues, arXiv, )
Limitation: The research tests selected tasks rather than representative everyday usage.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Forecast
“We can expect price wars.”
Our reading: There will be price wars among LLM providers.
Settled before its deadline Evidence as of
What happened: Reuters documents Alibaba's deep cuts followed hours later by Baidu making previously paid Ernie Speed and Ernie Lite services free. It explicitly identifies the competitive episode as extending into LLM pricing. (Reuters, republished by Yahoo Finance, )
Limitation: The decisive episode occurred in China; the claim supplied no geographic restriction.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Observation
“There has however been lots of progress both in discovering potential applications and in putting GPT-4 class models into practice.”
Our reading: Over the past year there has been substantial progress in finding applications for and deploying GPT-4 class models.
No horizon Evidence as of
What happened: The announcement identifies GPT-4-powered conversation practice and answer explanations, available immediately through Duolingo Max in selected iOS markets. This documents distinct educational applications implemented in a released product. (Duolingo, )
Limitation: The sources are reports from the deploying companies.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Observation
“Translation: Using a well-known algorithm that we didn’t invent to decide which faces of the cube to turn, we taught a robot to do motor control on a specially instrumented Rubik’s cube with bluetooth sensors inside.”
Our reading: In practice, OpenAI's 2019 claim of 'solving a Rubik's cube with one hand' meant using a well-known algorithm OpenAI didn't invent to choose which faces to turn, and teaching a robot motor control on a specially instrumented Rubik's cube with Bluetooth sensors inside.
No horizon Evidence as of
What happened: The announcement explicitly separates Kociemba's move sequence from the neural networks' learned manipulation. Its approach section states that training occurred in simulation. (OpenAI, )
Limitation: Support applies to the stated use of instrumented hardware, not exclusive reliance on it.
Source: OpenAI’s Lies and Half-Truths
Assessment: SupportedSaid
Observation
“Translation: But (revealed a few weeks later) behind the scenes we will try hard to water down your regulations.”
Our reading: While publicly asking the Senate to regulate it, OpenAI worked behind the scenes to water down regulations.
No horizon Evidence as of
What happened: The surrounding exchange supports public cooperation on regulation. Elsewhere in the same testimony, Altman advocates licensing and safety requirements. (U.S. Government Publishing Office, )
Limitation: The documented request concerned the EU AI Act; it does not establish efforts to weaken a U.S. Senate proposal [e2].
Source: OpenAI’s Lies and Half-Truths
Assessment: SupportedSaid
Observation
“GenAI is messing with science.”
Our reading: Generative AI is currently damaging science.
No horizon Evidence as of
What happened: The original paper expressly identifies Midjourney as the source of its images. It is evidence of AI provenance, not evidence that the paper's scientific assertions are reliable. (Frontiers in Cell and Developmental Biology; Xinyu Guo, Liang Dong and Dingjun Hao, )
Limitation: The conclusion concerns an identifiable publication failure, not the frequency or aggregate consequences of such failures.
Source: The Year of the Big-Balled Rat
Assessment: SupportedSaid
Observation
““Current deep-learning systems frequently succumb to stupid errors.” Still true.”
Our reading: Current deep-learning systems still often make stupid errors.
No horizon Evidence as of
What happened: Experts evaluated 26 ordinary election questions across five leading models; errors included incorrect contact details and fabricated information. GPT-4 performed substantially better, with 19% of its responses rated inaccurate, and the authors explicitly warn that this small, nonrandom sample is not representative of everyday usage. (Proof News and the Institute for Advanced Study, AI Democracy Projects, )
Limitation: The finding depends on reading frequent as recurring in ordinary-task evaluations, rather than as a population-wide frequency threshold.
Source: Two years later, deep learning is still faced with the same fundamental challenges
Assessment: SupportedSaid
Observation
““new effort by OpenAI to solve these problems wound [still] fabricat[ing] authoritative nonsense ” Still true.”
Our reading: OpenAI's new effort to solve these problems still ends up fabricating authoritative nonsense.
No horizon Evidence as of
What happened: GPT-4 confidently supplied permission and an accompanying false description of Texas polling-place rules, which the evaluators checked against the restriction on campaign apparel. The same report finds GPT-4 substantially more accurate than its peers and describes successful referrals to official polling information. (Proof News and the Institute for Advanced Study, AI Democracy Projects, )
Limitation: This example does not establish how frequently such errors occurred across OpenAI products.
Source: Two years later, deep learning is still faced with the same fundamental challenges
Assessment: SupportedSaid
Observation
“And the apparent copyright infringements that Reid Southen and I discussed in early January continue apace two months later, as I found in a quick experiment this morning:”
Our reading: The apparent copyright infringements that Marcus and Reid Southen discussed in early January 2024 were continuing apace two months later, based on a quick experiment Marcus ran on the morning of the post.
No horizon Evidence as of
What happened: The article reports its own reproduction of the described Elsa images with Copilot Designer and identifies DALL-E 3 as the underlying model. This is firsthand corroboration published on March 6; the article does not give a separate test timestamp. (The Verge, )
Limitation: The result depends on reading 'apace' as continued ready production rather than a quantitative rate claim.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Observation
“Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has trained on.”
Our reading: Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has been trained on.
No horizon Evidence as of
What happened: The same paragraph expressly withholds detailed architecture, size, hardware, compute, dataset construction, and training methods. It nevertheless identifies a Transformer-style model, next-token pretraining, public and licensed data, and reinforcement learning from human feedback. (OpenAI, )
Limitation: Official nondisclosure does not prove that outsiders knew nothing.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Observation
“They knew they were misleading people and failed to take an opportunity to correct.”
Our reading: OpenAI knew it was misleading people into believing it meant open-sourced AI and failed to take an opportunity, at a May 2016 meeting, to correct that belief.
No horizon Evidence as of
What happened: Amodei described a belief that OpenAI would release its AI source code publicly. Altman acknowledged that universal release was not the plan and then asked that the misconception not be corrected. (The New Yorker, )
Limitation: The supplied evidence consists of one reported account; no independent recording, transcript, or meeting minutes are supplied.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Observation
“On overt measures, the systems were fine. On covert measures, they were a disaster:”
Our reading: LLMs tested looked fine on overt racism measures but did very badly on covert racism measures when prompted with African American English.
No horizon Evidence as of
What happened: The abstract contrasts strongly negative dialect-based stereotypes with more positive overt stereotypes. This retrospectively corroborates the findings already reported in the March 1 preprint; the event date identifies that original reporting, not a new August experiment. (Nature, )
Limitation: The abstract does not establish that every tested model was free of overt bias.
Source: Covert racism in LLMs
Assessment: SupportedSaid
Observation
“Dropped is any mention of (a) the nonprofit (even though legally the company remains a nonprofit), (b) open source, once central both to their mission and name, and (c) the original phrasing about being "unconstrained by a need to generate financial return."”
Our reading: OpenAI's current homepage mission statement no longer mentions the nonprofit, open source, or being unconstrained by a need to generate financial return.
No horizon Evidence as of
What happened: Inspection of the complete extracted page finds none of the three elements. The quotation illustrates the page's research presentation; the absence finding rests on the full page, including its navigation and footer. (OpenAI, archived by Internet Archive, )
Limitation: The assessment covers text preserved and extracted from the archive, not unextracted video or image content.
Source: The Irony and the Agony of Elon Musk's Lawsuit against OpenAI
Assessment: SupportedSaid
Observation
“Shortly after I sent yesterday’s note on OpenAI’s latest woes, it was revealed that they are also under SEC investigation.”
Our reading: OpenAI is under SEC investigation as of late February 2024.
No horizon Evidence as of
What happened: The contemporaneous report describes an ongoing SEC investigation involving OpenAI. This supports the claim that OpenAI was under investigation at the time, while retaining Reuters' attribution to the Wall Street Journal. (Reuters, republished by Yahoo Finance, )
Limitation: No public SEC confirmation was located.
Assessment: SupportedSaid
Observation
“That’s exactly what classical AI has been trying to do for decades, with only limited success. And what systems like Gemini still struggle with.”
Our reading: Classical AI has had only limited success in turning input into checkable logical form and reasoning over it, and systems like Gemini still struggle with it.
No horizon Evidence as of
What happened: The historical synthesis describes domain restrictions and adaptation problems, including the 1984 SAVVY system. This supports limited general coverage, rather than an absence of successful applications. (arXiv; Kamath and Das, )
Limitation: Limited success is a qualitative judgment about general coverage.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Observation
“The trouble is that so far RAG has been hit and miss.”
Our reading: So far retrieval-augmented generation has produced inconsistent results.
No horizon Evidence as of
What happened: The study evaluated Bing Chat, NeevaAI, perplexity.ai, and YouChat using responses collected in February–March 2023; its methods identify their use of retrieved content. Average sentence citation support was 51.5% and citation precision was 74.5%, documenting supported outputs alongside citation failures. (Liu, Zhang and Liang; Findings of EMNLP 2023, hosted on arXiv, )
Limitation: Citation correctness is narrower than overall factual accuracy.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Forecast
“Imagining perfectly reliable, hallucination-free chatbots by this time next year is bonkers.”
Our reading: Perfectly reliable, hallucination-free chatbots will not exist by February 2025.
Deadline passed Evidence as of
What happened: The release documents continuing factual errors and evaluates several OpenAI models. Its tests exclude browsing, so they establish limitations of those model configurations rather than every retrieval-enabled chatbot. (OpenAI, )
Limitation: The outcome evidence is concentrated on OpenAI systems [e1, e2].
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Forecast
“Imagining this as a six-month project is naive.”
Our reading: Reliable, hallucination-free AI via RAG will not be achieved within six months.
Deadline passed Evidence as of
What happened: CRAG tests diverse factual questions across domains and question types. Its reported industry-system performance falls well short of broadly reliable query answering; the remaining percentage should not automatically be treated as a hallucination rate. (Yang et al., arXiv, )
Limitation: This is an assessment of demonstrated technology, not proof that no undisclosed system existed.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Forecast
“But we are kidding ourselves if it we think the solution is near at hand.”
Our reading: A solution to AI unreliability and hallucination is not near at hand.
Deadline passed Evidence as of
What happened: The BBC tested ChatGPT, Copilot, Gemini and Perplexity with access to BBC sources. Beyond the broader issue measure quoted here, it reported factual errors in 19% of answers citing BBC content, supporting continued factual unreliability. (BBC, )
Limitation: The one-year deadline is a contextual interpretation of near at hand.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Observation
“The road from research idea to reliable production remains steep.”
Our reading: Getting RAG from a research idea to reliable production remains difficult.
No horizon Evidence as of
What happened: The January report describes seven failure modes across research, education, and biomedical implementations. Its findings on testing, retrieval, answer extraction, and incomplete responses support substantial remaining production work. (arXiv; Barnett and colleagues, Deakin University, )
Limitation: The implementation report covers three cases, not a representative industry survey.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Observation
“Reality is that RAG is still in its infancy, a work in progress much as LLMs themselves are, an no more guaranteed to be correct. Like LLMs, RAG can appear spectacular at times, but there is no evidence that it is a cure-all.”
Our reading: RAG is immature, not guaranteed correct, and there is no evidence it is a cure-all for LLM problems.
No horizon Evidence as of
What happened: Three engineering case studies identify unresolved retrieval and generation failures, including failure despite the answer being supplied in context. The January report documents the then-existing reliability limitations; individual test days are not specified. (Barnett and colleagues, Deakin University; arXiv, )
Limitation: The evidence does not give 'infancy' an objective maturity threshold or show that every RAG application was immature.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Observation
“Over the last few hours, people reporting having been report a variety of problems with ChatGPT:”
Our reading: Over the last few hours before the post, people reported a variety of problems with ChatGPT.
No horizon Evidence as of
What happened: OpenAI acknowledged receiving reports during the incident window. This corroborates their existence without establishing every reported symptom. (OpenAI, )
Limitation: The evidence establishes reports, not the prevalence of malfunctions across users.
Source: ChatGPT has gone berserk
Assessment: SupportedSaid
Observation
“And one wants the system to have a nuanced understanding of what is even been asked. (That’s the clearly lacking now).”
Our reading: Current AI systems clearly lack a nuanced understanding of what is being asked.
No horizon Evidence as of
What happened: The reporter's tests produced women as nineteenth-century US senators, despite the article identifying 1922 as the first woman's Senate service. The same report found a substantially appropriate Founding Fathers response, showing inconsistent contextual handling rather than universal failure. (The Verge, )
Limitation: The result depends on reading the claim in its immediate Gemini image-generation context.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: SupportedSaid
Observation
“It’s a general issue.”
Our reading: Gemini's historically inaccurate diversity in generated images is a general issue, not limited to popes.
No horizon Evidence as of
What happened: The reporters reproduced inappropriate demographic depictions for German soldiers and nineteenth-century US senators, including female senators. These are separate historical subjects beyond popes; the report also records a more appropriate Founding Fathers result. (The Verge, )
Limitation: The tests do not measure how frequently the problem occurred.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: SupportedSaid
Observation
“You might have thought that the overactive guardrail problem would have been solved by now, what with billions more invested and ever-bigger models, not to mention the cultural backlash from the right. You’d be wrong.”
Our reading: Despite billions more in investment and bigger models, the overactive guardrail problem in AI chatbots has not been solved as of February 2024.
No horizon Evidence as of
What happened: The reporter directly tested ordinary portrait requests and observed refusals. This establishes that excessive restrictions persisted in a major assistant during the target period; a news test supplies the dated example missing from the provider's disclosure. (The Daily Dot, )
Limitation: These examples establish persistence, not the frequency of excessive refusals across assistants [e1, e2].
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: SupportedSaid
Forecast
“In short order, with Sora or some opensource counterparts that will emerge, scammers will start to make photorealistic fake videos simply to sell ads.”
Our reading: Open-source counterparts to Sora will emerge.
Settled before its deadline Evidence as of
What happened: The official release record establishes an actual video-generation release after the prediction. The later dated paper supplies fuller evidence of functionality and access. (HPC-AI Tech, )
Limitation: The conclusion depends on interpreting counterparts functionally, without an unstated requirement for Sora-level performance.
Source: Absolutely amazing ant video augurs intellectual mayhem
Assessment: SupportedSaid
Observation
“… but, um, ants have six legs.”
Our reading: A Sora-generated ant video got the ants' anatomy wrong: it did not show the six legs real ants have.
No horizon Evidence as of
What happened: An independent source says the ants in the Sora ant-tunnel video have four legs, not six. That matches Marcus's point. (HTC VIVE (VIVE POST-WAVE blog), )
Limitation: The four-leg count comes from a secondary blog post. Nobody in this review watched the video and counted the legs.
Source: Absolutely amazing ant video augurs intellectual mayhem
Assessment: SupportedSaid
Forecast
“I guarantee he won’t be the last:”
Our reading: More people besides the one already fooled will be fooled by fake Sora-generated videos.
Settled before its deadline Evidence as of
What happened: A wire service reports that Sora videos have fooled many people since September 2025. That directly matches the forecast. (The Canadian Press (via Castanet), )
Limitation: The Canadian Press makes a general statement. It does not name or count the people who were fooled.
Source: Absolutely amazing ant video augurs intellectual mayhem
Assessment: SupportedSaid
Observation
“In brief, an Air Canada chatbot confabulated (“hallucinated”) an answer – telling a customer that bereavement trips would be refunded.”
Our reading: An Air Canada chatbot hallucinated an answer telling a customer that bereavement trips would be refunded.
No horizon Evidence as of
What happened: The decision transcribes advice allowing a reduced bereavement rate through a refund application after travel, contrary to the policy it records. It finds negligent misrepresentation; the promised benefit concerned a partial fare reduction. (British Columbia Civil Resolution Tribunal, )
Limitation: Support depends on reading refunded as including a partial refund and hallucinated as a description of incorrect policy information.
Assessment: SupportedSaid
Observation
“Aside from showing that (not surprisingly) Sora is pretty clueless about chess, it’s revealing about how the mysterious Sora even works.”
Our reading: Sora is pretty clueless about chess.
No horizon Evidence as of
What happened: The report attributes the same elementary depiction errors to Rahul Agarwalla, rather than quoting Marcus. The event date refers to the original monkey demonstration, whose timestamp was checked on Altman's post; March 6 is the reporting date. (The Times of India, )
Limitation: The decisive evidence is a reported expert observation, because the original video's visual content was not available through the text reader.
Source: Sora Can’t Handle the Truth
Assessment: SupportedSaid
Observation
“Nowadays, we have “multimodal” systems, trained on both text and images, and the aforementioned “hallucinations” (perhaps better called failed approximations, as Gerben Wierda has pointed out) and other errors persist and even crop up in new ways.”
Our reading: Multimodal systems trained on text and images still hallucinate and make other errors, sometimes in new ways.
No horizon Evidence as of
What happened: The launch material documents invented entities in generated video. This supports the ordinary reading that multimodal generation retains fabrication errors and exhibits visual consistency failures. (OpenAI, )
Limitation: The demonstrations establish that errors occur, not their frequency across multimodal systems or whether multimodal training reduces them.
Source: Sora Can’t Handle the Truth
Assessment: SupportedSaid
Observation
“Turns out Sora makes stuff up, too,”
Our reading: OpenAI's Sora, like LLMs, makes things up.
No horizon Evidence as of
What happened: OpenAI identifies an unintended physical error in a launch demonstration. This supports the existence of realistic-looking errors without depending on Marcus's chessboard analysis. (OpenAI, )
Limitation: The disclosure establishes the occurrence of errors, not their frequency or how convincing viewers find the outputs.
Source: Sora Can’t Handle the Truth
Assessment: SupportedSaid
Observation
“Gary Marcus is astonished that some people still haven’t recognized that generative AI has struggled mightily when it comes to world models, high level reasoning and factuality, even as evidence for that conclusion continues to pour in, even with data of ever-greater magnitude and sophistication..”
Our reading: Generative AI has struggled mightily with world models, high-level reasoning and factuality, even with ever more data.
No horizon Evidence as of
What happened: The evaluation reports substantial failures to generate requested object relationships despite strong visual quality. This supports difficulties with spatial world consistency, without proving the absence of internal world models. (arXiv; Gokhale and colleagues, )
Limitation: The severity term is qualitative, so the outcome depends partly on the stated reading.
Source: Sora Can’t Handle the Truth
Assessment: SupportedSaid
Conditional forecast
“We will, I am certain, see more systemic glitches as more people have access.”
Our reading: As more people gain access to Sora, more systemic glitches will be found.
Settled before its deadline Evidence as of
What happened: The release expanded access beyond the February preview group, satisfying the antecedent. (OpenAI, )
Only if: as more people get access
Source: Sora’s Surreal Physics
Assessment: SupportedSaid
Observation
“Sora is not a solution to AI’s longstanding woes with space, time, and causality.”
Our reading: Sora does not solve AI's longstanding problems with space, time, and causality.
No horizon Evidence as of
What happened: The motion weakness directly establishes an unresolved physical-consistency problem. The surrounding limitations text also describes errors in spatial instructions, event trajectories, and consequences of interactions. (OpenAI, )
Limitation: The disclosure provides examples and qualitative limitations, not a representative benchmark or failure rate [e4].
Source: Sora’s Surreal Physics
Assessment: SupportedSaid
Forecast
“And god save us from the deluge of deepfakery that is to come.”
Our reading: A deluge of deepfakes is coming, enabled by systems like Sora.
Settled before its deadline Evidence as of
What happened: IWF assessed 3,443 realistic AI-generated abuse videos in 2025, versus 13 in 2024. Its accompanying observations include fabricated videos of known victims, establishing substantial harmful impersonation rather than merely benign synthetic production. (Internet Watch Foundation, )
Limitation: The result depends on interpreting an unquantified metaphor.
Source: Sora’s Surreal Physics
Assessment: SupportedSaid
Observation
“That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”
Our reading: Sora's failures of spatiotemporal continuity and object permanence are general, not one-off.
No horizon Evidence as of
What happened: OpenAI describes spontaneous appearances as a model weakness associated with a class of scenes, rather than as one isolated video defect. This directly supports recurrence without establishing how often it happens. (OpenAI, )
Limitation: The disclosure gives no representative sample or numerical prevalence estimate; it does not show that most or all videos fail [e2].
Source: Sora’s Surreal Physics
Assessment: SupportedSaid
Observation
“But the truth is it doesn’t work that well yet.”
Our reading: Programming in English with LLMs does not yet work very well.
No horizon Evidence as of
What happened: On CodeContests validation problems, direct prompting solved 19% with five attempts; a structured testing and revision process reached 44%. This supports substantial reliability limitations while also documenting useful capability and improvement. (arXiv; Tal Ridnik, Dedy Kredo and Itamar Friedman, )
Limitation: The competitive-programming evaluation does not establish a representative success rate for everyday programming or direct control through English prompts [e1].
Source: The hottest new programming language is English! Or maybe not.
Assessment: SupportedSaid
Observation
“OpenAI has never disclosed those numbers for GPT-4; I can scarcely imagine what they will be for GPT-5.”
Our reading: OpenAI has never disclosed the water consumption figures for GPT-4.
No horizon Evidence as of
What happened: The assessment includes OpenAI's GPT-4. The following sentence identifies broader environmental impact among the universally undisclosed indicators, and the preceding discussion explicitly includes water use. (Stanford CRFM and collaborating researchers, via arXiv, )
Limitation: A negative disclosure claim cannot exclude an obscure or subsequently removed publication with absolute certainty.
Source: Seven reasons why the world should say No to Sam Altman
Assessment: SupportedSaid
Observation
“And, let’s be honest folks, the “alignment problem” isn’t remotely solved.”
Our reading: The AI alignment problem is not remotely solved.
No horizon Evidence as of
What happened: The experiments show a substantial robustness gap: standard safety training did not reliably remove deliberately implanted deceptive behavior. This supports the ordinary reading that general alignment remained unresolved, without proving that deployed models spontaneously developed such behavior. (arXiv; Evan Hubinger and coauthors, )
Limitation: The decisive study was a preprint using deliberately constructed backdoors.
Source: Seven reasons why the world should say No to Sam Altman
Assessment: SupportedSaid
Causal claim
“In a system that can write emails, make appointments, sign checks, etc, the consequences of unreliability, discomprehension, and hallucination all escalate.”
Our reading: When LLMs can take actions like writing emails or signing checks, the consequences of their unreliability and hallucinations escalate.
No horizon Evidence as of
What happened: The researchers reproduced destructive execution in a real sandbox, rather than merely generating harmful text. The paper traces agent failures involving misinterpretation and unsafe assumptions into consequential tool actions; the exact experiment date is not reported. (arXiv, )
Limitation: The evidence supports increased potential consequences, not inevitable harm from every action.
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: SupportedSaid
Forecast
“This list is probably not exhaustive and will continue to grow.”
Our reading: The list of OpenAI/LLM security issues will continue to grow.
Settled before its deadline Evidence as of
What happened: Tenable publicly disclosed a ChatGPT Search attack that bypassed isolation to inject and extract memories. The disclosure adds a distinct product-specific issue to the earlier list; the event dated here is public disclosure, following private reporting in December 2024. (Tenable, )
Limitation: This establishes additional vulnerability disclosure, not a rate of future incidents.
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: SupportedSaid
Observation
“More evidence against his claims just keeps piling up.”
Our reading: Evidence against Hinton's claims that GenAI understands and doesn't store things keeps accumulating.
No horizon Evidence as of
What happened: The public study documents sensitivity to meaning-preserving formatting changes. This supplies evidence of fragile generalization, although it does not establish an absence of understanding. (arXiv; Sclar and colleagues, )
Limitation: The result depends on reading the statement as an evidentiary observation rather than a conclusive theory of understanding.
Source: Further Trouble in Hinton City
Assessment: SupportedSaid
Observation
“Reliability remain a problem.”
Our reading: Reliability remains a problem for large language models.
No horizon Evidence as of
What happened: Table 3 reports GPT-4 prompt-level accuracy of 76.89% under strict checking and 79.30% under loose checking. The November tests therefore show frequent instruction failures, while also showing that most tested prompts succeeded. (arXiv; Jeffrey Zhou and colleagues, )
Limitation: Benchmark performance varies by task and scoring method.
Assessment: SupportedSaid
Observation
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading: Scaling has not been enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
No horizon Evidence as of
What happened: OpenAI describes GPT-4 as a milestone in scaling deep learning while acknowledging factual and reasoning errors. The same disclosure reports reduced hallucinations and improved factuality, showing progress without complete resolution. (OpenAI, )
Limitation: The result depends on treating understanding as observable interpretation and generalization.
Assessment: SupportedSaid
Observation
“Compositionality remains a problem. So does negation.”
Our reading: Compositionality and negation remain problems for large language models.
No horizon Evidence as of
What happened: The researchers tested multiplication, logic puzzles and dynamic programming. GPT-4's multiplication accuracy declined sharply with problem complexity; step-by-step prompting improved simpler cases without eliminating the harder failures. (Allen Institute for Artificial Intelligence, )
Limitation: Results depend on task, model version and prompt.
Assessment: SupportedSaid
Observation
“It’s notorious for making illegal moves in chess.”
Our reading: ChatGPT is notorious for making illegal moves in chess.
No horizon Evidence as of
What happened: The researchers tested gpt-3.5-turbo-0301 across 3,200 games. Their procedure terminated games after ten consecutive illegal attempts, and their results document persistent rule-following failures. (arXiv; Mu-Tien Kuo, Chih-Chung Hsueh and Richard Tzong-Han Tsai, )
Limitation: Notoriety is qualitative.
Assessment: SupportedSaid
Observation
“But the variety of the examples along with the technical literature on memorization and data leakage in large language models make clear that this kind of reconstruction certainly can happen with some regularity.”
Our reading: Reconstruction of training text by large language models can happen with some regularity.
No horizon Evidence as of
What happened: Thousands of distinct extracted examples establish recurrence under a targeted attack. Because ChatGPT's training corpus was unavailable, the researchers checked matches against a large auxiliary corpus; this is less direct than verification against a disclosed training set. (arXiv; Nasr and colleagues, )
Limitation: Marcus supplies no numerical meaning for 'some regularity'.
Assessment: SupportedSaid
Observation
“LLMs don’t store things in simple databases, but when presented with enough data, their regeneration of distributed bits has demonstrably duplicated lengthy training materials multiple times, a phenomenon that is called memorization in the field.”
Our reading: Given enough data, LLMs have demonstrably duplicated lengthy training materials multiple times (memorization).
No horizon Evidence as of
What happened: The GPT-2 study documents an entire code file and complete license texts among its extended extractions. These are multiple substantial reproductions, well beyond ordinary short-phrase overlap. (Carlini et al., arXiv, )
Limitation: Extraction methods can involve deliberate prompting and specialized decoding.
Assessment: SupportedSaid
Observation
“Neural networks do effectively memorize some of what they are trained on, and indeed there is a whole subfield now about the topic.”
Our reading: Neural networks effectively memorize some of what they are trained on, and there is now a whole subfield devoted to this topic.
No horizon Evidence as of
What happened: The paper reports substantial extraction from neural language models and directly checks known training corpora for open models. Its methods and review of preceding studies also support the existence of an established research line. (Nasr and colleagues, arXiv, )
Limitation: For closed models, the researchers used an auxiliary corpus because the complete training data were unavailable.
Assessment: SupportedSaid
Observation
“An entire cast of deepfaked people include a deep faked CFO scammed a company out $25 million.”
Our reading: Deepfaked participants, including a deepfaked CFO, on a video call scammed a company out of $25 million.
No horizon Evidence as of
What happened: CNN reports the transfers and their US-dollar equivalent, alongside police descriptions of the fabricated CFO and colleagues. It also reports Arup's confirmation that it was the victim; the amount supports the approximate reading. (CNN, )
Limitation: Support depends on reading $25 million as approximate, not exact.
Source: Deepfaked Shit is Getting Real
Assessment: SupportedSaid
Observation
“Even scarier is that the employee who was tricked was suspicious, not naive, and still fell for it:”
Our reading: The deepfake scam worked even on an employee who was suspicious rather than naive.
No horizon Evidence as of
What happened: CNN attributes initial suspicion to police superintendent Baron Chan. The following passages explain that the employee suspected phishing, set aside those doubts after seeing apparent colleagues, and agreed to transfer the money. (CNN, )
Limitation: The account of suspicion comes through police reporting, rather than a direct employee interview.
Source: Deepfaked Shit is Getting Real
Assessment: SupportedSaid
Observation
“I am here to tell you that OpenAI’s research, analyzed carefully, shows no such thing, and that there is plenty of reason to be concerned.”
Our reading: OpenAI's bioweapons research, analyzed carefully, does not show that GPT-4 poses at most a slight risk of helping create biological threats, and there is plenty of reason to be concerned.
No horizon Evidence as of
What happened: Limited sensitivity prevents treating nonsignificance as a tight risk bound. The study also distinguishes information access from physical implementation and leaves the translation from scores to risk unsettled. (OpenAI, )
Limitation: OpenAI itself acknowledged uncertainty; Marcus's criticism should not imply that the study concealed all these limitations.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Observation
“For one thing, in the expert population, the effect is in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation, and release).”
Our reading: In OpenAI's threshold analysis, among experts the effect was in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation and release).
No horizon Evidence as of
What happened: Joining the complete response file to e1 confirms higher expert threshold-success rates with model access on every task. The date identifies the published dataset being assessed, not an inferred experiment date. (OpenAI, )
Limitation: These observed threshold proportions alone do not establish statistical significance or real-world biological threat capability.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons