Happy 1st Birthday, GPT-4
We recorded 7 claims from this post. 4 were checked against evidence. 3 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (7)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“We might be reaching a plateau in terms of sheer capability.”
What we tested: LLM capability may be reaching a plateau around GPT-4 level.
Assessment: SupportedSaid
“There has however been lots of progress both in discovering potential applications and in putting GPT-4 class models into practice.”
What we tested: Over the past year there has been substantial progress in finding applications for and deploying GPT-4 class models.
What happened: The announcement identifies GPT-4-powered conversation practice and answer explanations, available immediately through Duolingo Max in selected iOS markets. This documents distinct educational applications implemented in a released product. (Duolingo, )
Assessment: UnresolvedSaid
“But putting LLMs into production is hard; most work to date has been preliminary.”
What we tested: Putting LLMs into production is hard and most deployment work to date has been preliminary.
Assessment: SupportedSaid
“Some core problems have lingered, particularly hallucinations, instability and occasional nonsensical errors.”
What we tested: Hallucinations, instability and occasional nonsensical errors remain unsolved in GPT-4 class LLMs.
What happened: The experiments demonstrate false supporting assertions that GPT-4 can recognize as wrong when asked separately. They establish hallucination and elementary-error failure modes, not a population-wide error rate. (Zhang and colleagues, arXiv, )
Assessment: SupportedSaid
“We can expect price wars.”
What we tested: There will be price wars among LLM providers.
What happened: Reuters documents Alibaba's deep cuts followed hours later by Baidu making previously paid Ernie Speed and Ernie Lite services free. It explicitly identifies the competitive episode as extending into LLM pricing. (Reuters, republished by Yahoo Finance, )
Assessment: Can’t be checkedSaid
“Climate impact needs to be considered, especially as models continue to grow.”
What we tested: The climate impact of AI models should be considered, especially as they grow.
Assessment: Can’t be checkedSaid
“Profitability remains in question.”
What we tested: The profitability of LLM businesses remains uncertain.
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The criteria are ambiguous. Our reading, not assessed: Companies are starting to temper their LLM expectations, many of which were unrealistic.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: Bad actors appear to be using LLMs for cybercrime and disinformation.
- The two reviews read the claim differently
- The two reviews read the claim differently