What exactly are the economics of AI?
We recorded 12 claims from this post. 3 were checked against evidence. 9 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (12)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“As I argued yesterday, the whole thing could turn out to be a fad, but it’s still way too early too tell.”
What we tested: Generative AI could turn out to be a fad, though it is too early to tell.
Assessment: UnresolvedSaid
“Driverless cars are powered by AI, but so far the money that has gone into them, on order of $100 billion dollars, vastly exceeds the actual revenue that has come out of them.”
What we tested: Investment in driverless cars, around $100 billion, vastly exceeds the revenue they have generated so far.
Assessment: Can’t be checkedSaid
“That said, I don’t think OpenAI is actually in any kind of imminent danger, but nor do I think they are out of the woods.”
What we tested: Marcus does not think OpenAI is in any kind of imminent danger, but he also does not think it is out of the woods.
Assessment: UnresolvedSaid
“Coding is a solid use case that will continue to keep generative AI in the limelight for a long time. Programmers will never go back.”
What we tested: Coding will remain a solid use case keeping generative AI prominent for a long time, and programmers will not abandon it.
Assessment: Can’t be checkedSaid
“I am doubtful that Microsoft would let OpenAI go under anytime soon, given the optics, and impact on their own stock price.”
What we tested: Microsoft is unlikely to let OpenAI go under anytime soon.
Assessment: Can’t be checkedSaid
“LLMs will probably get cheaper to operate over time, both as the hardware required inevitably gets cheaper, and as people figure out how to make them more efficient.”
What we tested: LLMs will probably become cheaper to operate over time as hardware cheapens and efficiency improves.
Assessment: SupportedSaid
“The underlying technology—often referred to as Foundation Models, because systems are fine-tuned on top of large pretrained (foundation) model—is deeply unstable and thus difficult to engineer into reliable products.”
What we tested: Foundation models are deeply unstable, which makes them difficult to engineer into reliable products.
What happened: Comparisons of March and June model versions document formatting failures that prevented code execution and caused a LangChain parser to fail, directly connecting model behavior to integration difficulty. The results also limit the claim: GPT-4's agent performance improved, and removing non-code wrappers increased its code-test success from 52% to 70%, so behavioral change did not imply declining underlying capability. (arXiv; Lingjiao Chen, Matei Zaharia, and James Zou, )
Assessment: Can’t be checkedSaid
“We now know that answers can even change from one month to the next—which makes third-party engineering of large language models into complex systems a huge uphill struggle. In some mission-critical cases that engineering challenge may be insuperable.”
What we tested: Because LLM answers can even change from one month to the next, third-party engineering of large language models into complex systems is a huge uphill struggle, and in some mission-critical cases that challenge may be insuperable.
Assessment: Can’t be checkedSaid
“A lot of talent and investors might move on to a new set of shiny things, fleeing generative AI as quickly as some of the same people not long ago fled crypto.”
What we tested: A lot of talent and investors might move on to a new set of shiny things, fleeing generative AI as quickly as some of the same people recently fled crypto.
Assessment: Can’t be checkedSaid
“If some people get off the bus, and valuations begin to drop a little, that could launch a potent negative feedback spiral, sparking a sudden deceleration in values that heretofore had been rapidly accelerating.”
What we tested: If some people get off the bus and generative AI valuations begin to drop a little, that could launch a potent negative feedback spiral, sparking a sudden deceleration in values that had been rapidly accelerating.
Assessment: Can’t be checkedSaid
“The astronomical valuations for Generative AI companies might be justified, but might well not be. Thus far, the valuations seem to be predicated on hopes and dreams, without really factoring in the serious engineering risks.”
What we tested: The astronomical valuations of generative AI companies might be justified but might well not be, and so far they seem to be predicated on hopes and dreams without really factoring in the serious engineering risks.
Assessment: Can’t be checkedSaid
“It’s not a house of cards, but the financial foundation of so-called Foundation Models is hardly as robust as it seems.”
What we tested: The financial foundation of so-called foundation models is not a house of cards, but it is hardly as robust as it seems.
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The criteria are ambiguous. Our reading, not assessed: Most of the basic generative AI technology is well understood and to a large degree readily copyable, and it is not clear that any commercial company, such as OpenAI, has a durable moat to keep competitors out.
- The record does not meet the current rules. Our reading, not assessed: AI works well for some uses like ads and coding but is not reliable enough for many others such as autonomous driving, medical diagnosis, and chat-style search.
- The criteria are ambiguous. Our reading, not assessed: Current revenue for OpenAI and generative AI appears modest, perhaps roughly in the ballpark of operating costs but not far beyond them; apart from chat-style search, which doesn't (yet?) work well, no killer app has yet emerged; and investors will not be infinitely patient.
- Attribution is uncertain
- The criteria are ambiguous. Our reading, not assessed: As of August 2023, generative AI has made perhaps hundreds of millions of dollars, probably not billions, and certainly not hundreds of billions.