Gary Marcus Marcus on AI Published 2024-01-27

OpenAI’s Got 9.9 Problems, and Twitch Ain’t One

We recorded 25 claims from this post. 5 were checked against evidence. 20 can’t be checked. 5 more are held. Read the post at its source.

Claims from this post (25)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

Assessment: Can’t be checkedSaid 2024-01-27
“Regardless of how the internal investigation comes out, 2024 is a looking a lot more challenging, both for Altman and for the company.”
What we tested: 2024 looks likely to be a much more challenging year for OpenAI and Sam Altman.
Assessment: Can’t be checkedSaid 2024-01-27
“It would not be surprising if OpenAI spends a huge part of its energy this year defending lawsuits.”
What we tested: OpenAI may spend a huge part of its energy in 2024 defending lawsuits.
Assessment: Can’t be checkedSaid 2024-01-27
“First, lawsuits against the company are likely to come fast and furious.”
What we tested: OpenAI will likely face a rapid flurry of lawsuits in 2024.
Assessment: Can’t be checkedSaid 2024-01-27
“The request is likely in violation both of common law and of international conventions around intellectual property such as the Berne convention, and publishers, writers, and artists have every right to fight it tooth and nail.”
What we tested: OpenAI's request for a copyright exemption is likely in violation of common law and international IP conventions like Berne.
Assessment: Can’t be checkedSaid 2024-01-27
“Second, battles over copyright materials may wind up cutting massively into profits, and may be unavoidable.”
What we tested: Copyright battles may cut massively into OpenAI's profits and may be unavoidable.
Assessment: Can’t be checkedSaid 2024-01-27
“Given how many different kinds of copyrighted works they are drawing on, and from how many vendors (many media outlets, many book publishers, many film studios, etc), and given how commercial their usage is of those sources, the licensing costs may quickly mount.”
What we tested: OpenAI's licensing costs for copyrighted material may quickly mount.
Assessment: UnresolvedSaid 2024-01-27
“Third, at least so far OpenAI lacks both profits and a moat.”
What we tested: As of January 2024, OpenAI has neither profits nor a competitive moat.
Assessment: Can’t be checkedSaid 2024-01-27
“Large language models such as ChatGPT may quickly become commodities; we can expect price wars, and profits may continue to be elusive, or modest at best.”
What we tested: Large language models such as ChatGPT may quickly become commodities, price wars can be expected, and profits may continue to be elusive or modest at best.
Assessment: Can’t be checkedSaid 2024-01-27
“Fifth, although the underlying technology initially improved rapidly, leading to a lot of excitement, it may soon, perhaps this year or next, reach a plateau.”
What we tested: LLM technology may reach a plateau, perhaps in 2024 or 2025.
Assessment: Can’t be checkedSaid 2024-01-27
“Progress depends heavily on having vast amounts of data, and it is likely that most of the high quality data sources have already been tapped.”
What we tested: Most high-quality data sources for LLM training have likely already been tapped.
Assessment: Can’t be checkedSaid 2024-01-27
“If a plateau is reached in the near-term, a lot of the enthusiasm may dissipate, and investors may start to temper their investments.”
What we tested: If LLMs plateau in the near term, enthusiasm may dissipate and investors may temper their investments.
Assessment: Can’t be checkedSaid 2024-01-27
“Moreover more data tends to go with bigger models that more expensive to operate; the increases in cost may not be accompanied by big enough gains in performance, at some point leading the big companies to step off the gas.”
What we tested: The rising costs of bigger models trained on more data may not be accompanied by big enough gains in performance, at some point leading the big companies to step off the gas.
Assessment: Can’t be checkedSaid 2024-01-27
“Sixth, although ChatGPT has already widely been adopted by computer programmers and perhaps others, and been explored by a great many Chief Information Officers and the like, it is still not clear that there will be enough adoption to justify the company’s stratospheric valuations.”
What we tested: It is unclear that ChatGPT adoption will be enough to justify OpenAI's very high valuation.
Assessment: UnresolvedSaid 2024-01-27
“Many companies are still in wait and see mode, trying the software but not putting into full-time production.”
What we tested: Many companies are trying ChatGPT without putting it into full-time production.
Assessment: SupportedSaid 2024-01-27
“It has become apparent that large language models, OpenAI’s core technology, sometimes leak private data, and that this problem has not been resolved; hallucinations too remain problematic.”
What we tested: LLMs sometimes leak private data, an unresolved problem, and hallucinations remain problematic.
What happened: This is the researchers’ dated assessment of the continuing vulnerability, not a claim that their experiments occurred on publication day. They explain why a word-repetition filter would address one exploit without eliminating memorization and divergence. (Authors of Scalable Extraction of Training Data from (Production) Language Models, 2023-11-28)
Assessment: SupportedSaid 2024-01-27
“OpenAI’s unusual relationship with Microsoft is sure to receive considerable scrutiny.”
What we tested: OpenAI's unusual relationship with Microsoft is sure to receive considerable scrutiny.
What happened: The decision describes examination of internal documents, information-request responses and the partnership’s commercial operation. Its analysis covers governance, compute supply and intellectual-property rights, including developments after the forecast. (UK Competition and Markets Authority, 2025-03-05)
Assessment: Can’t be checkedSaid 2024-01-27
“(Regulation in the EU and elsewhere may force OpenAI to confront problems with their software around unreliability, privacy, its off-label use to generate misinformation, and in the service of cybercrime, and so on).”
What we tested: EU and other regulation may force OpenAI to confront unreliability, privacy, misinformation, and cybercrime problems.
Assessment: SupportedSaid 2024-01-27
“And candor of the company as a whole will remain an issue.”
What we tested: OpenAI's candor as a company will remain an issue.
What happened: Vox compared leadership claims of ignorance about equity-related exit provisions with signed company documents. This establishes a substantive corporate candor dispute, without conclusively proving what each executive knew. (Vox, 2024-05-22)
Assessment: Can’t be checkedSaid 2024-01-27
“Ninth, internal tensions are likely to remain.”
What we tested: Internal tensions at OpenAI are likely to persist in 2024.
Assessment: Can’t be checkedSaid 2024-01-27
“This call could easily put a further strain on the company, and perhaps lead to other uncomfortable revelations if California chooses to investigate.”
What we tested: Public Citizen's call could strain OpenAI and, if California investigates, lead to uncomfortable revelations.
Assessment: Can’t be checkedSaid 2024-01-27
“It’s also worth noting that OpenAI is not alone is all of this; many other generative AI companies are likely to encounter intense legal and public pressure this year, around potential cases of copyright infringement.”
What we tested: Many generative AI companies are likely to face intense legal and public pressure over copyright in 2024.
Assessment: Can’t be checkedSaid 2024-01-27
“Valuations for OpenAI and some of its peers may drop, sharply.”
What we tested: Valuations of OpenAI and some peers may drop sharply.
Assessment: Can’t be checkedSaid 2024-01-27
“External investors pouring money into the company (most recently at a valuation rumored to be $86 billion dollars) may find themselves disappointed, unclear on how profits to justify those valuations could ever be made.”
What we tested: External investors putting money into OpenAI, most recently at a rumored $86 billion valuation, may find themselves disappointed and unclear on how profits to justify those valuations could ever be made.
Assessment: Can’t be checkedSaid 2024-01-27
“Here’s my own guess: OpenAI, backed by Microsoft’s trillions, and now deeply tied to Microsoft, will survive, but the bloom will fade from the rose.”
What we tested: Marcus guesses that OpenAI, backed by and deeply tied to Microsoft, will survive, but that the bloom will fade from the rose.
Assessment: Can’t be checkedSaid 2024-01-27
“A lot of what has driven excitement about OpenAI has been a sense of unlimited potential. That could soon change.”
What we tested: The perception of OpenAI's unlimited potential could soon change.

Held from this post (5)

These records did not pass every check, so nothing about them is published. Held is not a finding.

The record does not meet the current rules. Our reading, not assessed: There is no immediate fix for LLM hallucinations, and one may not arrive soon.
The two reviews reached different assessments. Our reading, not assessed: Many of the major companies that looked into ChatGPT in 2023 have been held back to a greater or lesser degree by its hallucination errors.
The two reviews read the claim differently
The record does not meet the current rules. Our reading, not assessed: ChatGPT and related systems have a truthiness problem: some of what they say is true and some is not, and it is very difficult for the end user to anticipate which will be true.
The criteria are ambiguous. Our reading, not assessed: Lawsuit and licensing payouts will cut heavily into OpenAI's potential future profits.