What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
We recorded 23 claims from this post. 4 were checked against evidence. 19 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (23)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“I won’t say that this can’t happen. But it is deeply speculative.”
What we tested: Marcus will not rule out the upside scenario in which Altman, OpenAI and their partners achieve AGI that conquers sickness and climate change and Altman uses his cut to subsidize everyone else, but he calls that scenario deeply speculative.
Assessment: Can’t be checkedSaid
“First, in The Information earlier this week was a hint that OpenAI might not be knocking it out of the park economically.”
What we tested: There are signs OpenAI may not be doing very well economically.
Assessment: UnresolvedSaid
“Chatbots are struggling to make the money people envision, so he is doubling down to (a) agents, and (b) a multitrillion-dollar chips-and-energy infrastructure play.”
What we tested: Chatbots are struggling to make the money people envision, so Altman is doubling down on agents and on a multitrillion-dollar chips-and-energy infrastructure play.
Assessment: SupportedSaid
“This list is probably not exhaustive and will continue to grow.”
What we tested: The list of OpenAI/LLM security issues will continue to grow.
What happened: Tenable publicly disclosed a ChatGPT Search attack that bypassed isolation to inject and extract memories. The disclosure adds a distinct product-specific issue to the earlier list; the event dated here is public disclosure, following private reporting in December 2024. (Tenable, )
Assessment: Can’t be checkedSaid
“Rogue employees could take over your devices, and bad actors will have tremendous incentive to hack OpenAI’s security.”
What we tested: If OpenAI's software can take over user devices, rogue employees could take over devices and attackers will have strong incentive to hack OpenAI.
Assessment: UnresolvedSaid
“They are already a target for attack, and as long as they are successful they will continue to be; if they have direct access to user devices, the risks of them getting hacked will increase, and the spoils of war for the attackers will increase, since it enables essentially infinite mischief (e.g., draining bank balances, stealing credit card numbers, placing fraudulent transactions, extortion based on private photos, holding devices for ransom, and so on).”
What we tested: If OpenAI gains direct access to user devices, the risk of it being hacked and the rewards to attackers will increase.
Assessment: SupportedSaid
“In a system that can write emails, make appointments, sign checks, etc, the consequences of unreliability, discomprehension, and hallucination all escalate.”
What we tested: When LLMs can take actions like writing emails or signing checks, the consequences of their unreliability and hallucinations escalate.
What happened: The researchers reproduced destructive execution in a real sandbox, rather than merely generating harmful text. The paper traces agent failures involving misinterpretation and unsafe assumptions into consequential tool actions; the exact experiment date is not reported. (arXiv, )
Assessment: Can’t be checkedSaid
“Giving the keys to the castle to an LLM, at our current level of technological readiness, seems to me to be batshit insane.”
What we tested: At current technological readiness, giving LLMs full control of users' systems is reckless and should not be done.
Assessment: Can’t be checkedSaid
“Combine that with worries about security, and it’s all a colossal accident waiting to happen.”
What we tested: Marcus calls device-controlling LLM agents, given their privacy problems combined with worries about security, 'a colossal accident waiting to happen'.
Assessment: Can’t be checkedSaid
“One screwup (and we all know LLMs are prone to screwups) and a LOT of people could die.”
What we tested: If LLM agents ran on Defense Department computers, a single LLM error could kill many people.
Assessment: Can’t be checkedSaid
“Without massive technical advances, chains of LLMs are unlikely to give us that.”
What we tested: Without massive technical advances, chains of LLMs are unlikely to provide a stable and secure basis for agents.
Assessment: Can’t be checkedSaid
“But they need to be built on a stable and secure basis.”
What we tested: AI agents need to be built on a stable and secure foundation.
Assessment: Can’t be checkedSaid
“In the very best case, LLM-based agents will change the world for the better. But I don’t see that happening, certainly not this decade.”
What we tested: Marcus does not see LLM-based agents changing the world for the better, which he calls the very best case, and certainly not this decade.
Assessment: Can’t be checkedSaid
“The best realistic case is that LLM-based agents never see widespread release. If we are lucky, their likely problems with reliability and all the associated risks will cause them to die on the vine.”
What we tested: The best realistic outcome is that LLM-based agents never see widespread release, dying because of their likely reliability problems.
Assessment: Can’t be checkedSaid
“In short, I am putting LLM-based agents distributed to millions of customers at the top of my list of most dangerous ideas of 2024.”
What we tested: Marcus puts LLM-based agents distributed to millions of customers at the top of his own list of the most dangerous ideas of 2024.
Assessment: Can’t be checkedSaid
“I doubt we will have the maturity we need in the next several years.”
What we tested: AI agent technology is unlikely to reach the needed maturity for reliable use in the next several years.
Assessment: Can’t be checkedSaid
“If they throw LLM-based agents over the fence, we are all in grave danger.”
What we tested: If OpenAI releases LLM-based agents carelessly, everyone will be in grave danger.
Assessment: Can’t be checkedSaid
“My guess is that Altman never gets the money, even in ten yearly $700B tranches.”
What we tested: Altman will probably never raise the $7 trillion, even in ten yearly $700B tranches.
Assessment: Can’t be checkedSaid
“To begin with, it’s just not clear that anywhere near that much capital is available.”
What we tested: It is not clear that anywhere near $7 trillion of capital is available for Altman's project.
Assessment: Can’t be checkedSaid
“Whatever number the investors might run, I am not remotely convinced that we are particularly close to AGI.”
What we tested: Whatever numbers investors might run, Marcus is not remotely convinced that we are particularly close to AGI.
Assessment: Can’t be checkedSaid
“Nor am I confident that Altmann or anyone else has any idea in the near term about how to make AGI safe.”
What we tested: Marcus is not confident that Altman or anyone else has any idea, in the near term, of how to make AGI safe.
Assessment: Can’t be checkedSaid
“Nor am I convinced that the proceeds (if they did succeed) would be redistributed remotely equitably.”
What we tested: Marcus is not convinced that, if Altman's $7 trillion project succeeded, its proceeds would be redistributed remotely equitably.
Assessment: Can’t be checkedSaid
“Even if the $7 trillion bet succeeded (on its own terms), it might tear apart society through radically increased unemployment and social inequality or redrawn geopolitics that are hard even to fathom.”
What we tested: Even if the $7 trillion bet succeeded, it might tear apart society through unemployment, inequality or upheaval in geopolitics.
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.