Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
We recorded 20 claims from this post. 8 were checked against evidence. 12 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (20)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: MixedSaid
Observation
“The study Mollick linked to doesn’t actually show what he claims. If you read it carefully, it says literally nothing about capabilities improving.”
Our reading: The study Mollick cited does not show that LLM capabilities are improving; it only addresses compute efficiency.
No horizon Evidence as of
Limitation: Language-model performance is narrower than general capability.
Assessment: Can’t be checkedSaid
Observation
“I could be persuaded that on some measures there was a doubling of capabilities for some set of months in 2020-2023, but I don’t see that case at all for the last 13 months.”
Our reading: Marcus could be persuaded that LLM capabilities doubled on some measures for some months in 2020-2023, but he does not see that case at all for the 13 months since GPT-4's release.
No horizon Evidence as of
Limitation: The packet contains no evidence items establishing a defined capability comparison.
Assessment: Can’t be checkedSaid
Observation
“What really got me to thinking about all this was a graph from OpenAI a couple days ago, touting their latest model, GPT 4 Turbo, which I have always suspected was a failed attempt at GPT-5.”
Our reading: Marcus suspects GPT-4 Turbo was a failed attempt at building GPT-5.
No horizon
Assessment: SupportedSaid
Observation
“What it actually shows is some improvements, mostly modest, over the last few months for a bunch of different measures.”
Our reading: OpenAI's GPT-4 Turbo graph shows only mostly modest improvements over recent months.
No horizon Evidence as of
Limitation: Modest is a qualitative judgment, and relative changes can look larger.
Assessment: Can’t be checkedSaid
Observation
“Whatever doubling there might have been, has perhaps come to an end.”
Our reading: Any doubling of LLM capabilities has perhaps come to an end.
No horizon
Assessment: SupportedSaid
Observation
“Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns.”
Our reading: MMLU gains from GPT-4 to GPT-4 Turbo are small compared with earlier generational jumps, tentatively evidencing diminishing returns.
No horizon Evidence as of
Limitation: The records do not compare all four models under one protocol: GPT-2 was fine-tuned, GPT-3 used few-shot prompting, GPT-4 used five-shot prompting, and Turbo used zero-shot chain-of-thought prompting [e1, e2, e3].
Assessment: Can’t be checkedSaid
Observation
“My gut sense is that we haven’t reached the real ceiling on MMLU yet, and that this is a genuine sign of diminishing returns.”
Our reading: The MMLU plateau is likely a genuine sign of diminishing returns rather than a ceiling effect.
No horizon
Qualifier: MMLU real ceiling not yet reached
Assessment: Can’t be checkedSaid
Observation
“If two graphs I plotted are remotely correct, Mollick’s claim that “the rate of improvement in Large Language models show capabilities doubling every 5 to 14 months” is no longer holding true.”
Our reading: If his graphs are roughly correct, LLM capabilities are no longer doubling every 5 to 14 months.
No horizon Evidence as of
Limitation: This does not endorse Mollick's wording.
Assessment: Can’t be checkedSaid
Forecast
“The wall that I once warned about, in 2022, may finally be approaching.”
Our reading: LLM progress may be approaching the wall Marcus warned about in 2022.
No horizon
Assessment: UnresolvedSaid
Observation
“One more way to look at this, is this graph I just saw: enormous convergence on GPT-4 level performance, in multiple models released since, yet nothing decisively ahead.”
Our reading: Multiple models released since GPT-4 have converged on GPT-4-level performance with none decisively ahead.
No horizon Evidence as of
Limitation: A coding benchmark cannot establish general capability parity.
Assessment: SupportedSaid
Observation
“I think it’s fair to say that they have not. GPT-Turbo released this week still produces its share of groaners, like this interchange Phil Libin just sent:”
Our reading: Hallucinations and stupid errors in LLMs have not been solved; GPT-4 Turbo still makes them.
No horizon Evidence as of
Limitation: The coding test measures task completion, not hallucination frequency.
Assessment: UnresolvedSaid
Observation
“Another way to think about this (see third graph above) is that there about 5 - 7 recent models are on a par with GPT-4, but none is clearly and decisively ahead.”
Our reading: About 5 to 7 recent models are on par with GPT-4, but none is clearly ahead.
No horizon Evidence as of
Limitation: The earlier four-model list does not contradict a later approximate five-to-seven-model count.
Assessment: Can’t be checkedSaid
Observation
“And of course advancing on benchmarks isn’t itself enough anyway; few benchmarks capture the complexity of the real world. Even if LLMs could max out on all existing benchmark, we might still have a long way to go.”
Our reading: Advancing on benchmarks is not enough by itself because few benchmarks capture the complexity of the real world, and even if LLMs could max out all existing benchmarks, we might still have a long way to go.
No horizon
Qualifier: even if LLMs could max out on all existing benchmarks
Assessment: Can’t be checkedSaid
Conditional forecast
“If we really have changed regimes, from rapid progress to diminishing returns, and hallucinations and stupid errors do linger, LLMs may never be ready for prime time.”
Our reading: If we really have changed regimes from rapid progress to diminishing returns, and hallucinations and stupid errors do linger, LLMs may never be ready for prime time.
No horizon
Only if: if we really have changed regimes from rapid progress to diminishing returns
Assessment: Can’t be checkedSaid
Forecast
“In the most extreme case, OpenAI’s $86 billion valuation could look in hindsight like a WeWork moment for AI.”
Our reading: In the most extreme case, OpenAI's $86 billion valuation could in hindsight look like a WeWork moment for AI.
No horizon
Qualifier: in the most extreme case
Assessment: Can’t be checkedSaid
Forecast
“Instead, as I warned in August, we may well be in for a correction.”
Our reading: The AI market may well be in for a correction.
No horizon
Assessment: SupportedSaid
Observation
“Already in recent weeks, Inflection AI largely closed shop, Stability AI is struggling, and LLM-based autonomous vehicle company called Ghost closed shop, and a software engineer on YouTube raised quite serious questions about the wildly-hyped AI coding system Devin.”
Our reading: In recent weeks Inflection AI largely closed, Stability AI is struggling, Ghost closed, and serious questions were raised about Devin.
No horizon Evidence as of
Limitation: The Inflection finding depends on the meaning of 'largely'; the company continued operating.
Assessment: UnresolvedSaid
Conditional forecast
“If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation.”
Our reading: If enthusiasm for GenAI dwindles and market valuations plummet, AI and LLMs will not disappear, and LLMs will still have their place as tools for statistical approximation.
No deadline stated Evidence as of
Only if: if enthusiasm for GenAI dwindles and market valuations plummet
Assessment: Can’t be checkedSaid
Forecast
“But that place may be smaller; it is entirely possible that LLMs on their own will never live up to last year’s wild expectations.”
Our reading: LLMs' role may be smaller than hoped, and LLMs alone may never live up to 2023's wild expectations.
No horizon
Qualifier: LLMs on their own
Assessment: Can’t be checkedSaid
Forecast
“Reliable, trustworthy AI is surely achievable, but we may need to go back to the drawing board to get there.”
Our reading: Reliable, trustworthy AI is surely achievable, but we may need to go back to the drawing board to get there.
No horizon
Qualifier: achievability stated as 'surely'
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: There are numerous signs that LLM development has reached a period of diminishing returns.
- The criteria are ambiguous. Our reading, not assessed: Bright humans can probably score something like 90%+ on the NYT Connections game, but current LLMs are not close to that level.