ChatGPT, at Age Two
We recorded 6 claims from this post. 5 were checked against evidence. 1 can’t be checked. 10 more are held. Read the post at its source.
Claims from this post (6)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Observation
“But a lot of of theories about how ChatGPT would be used in practice have fizzled out.”
Our reading: Many theories about how ChatGPT would be used in practice have fizzled out.
No horizon Evidence as of
Limitation: The population of predictions is undefined.
Assessment: SupportedSaid
Observation
“Two years later Google’s search share hasn’t diminished all that much.”
Our reading: Two years after ChatGPT's launch, Google's search share has not diminished much.
No horizon Evidence as of
Limitation: The figures come from secondary reporting of Statcounter because historical primary data were not retrievable.
Assessment: UnresolvedSaid
Observation
“There has been a ton of experimentation, but relatively few well-documented success stories, and some fairly negative stories like a recent one at Business Insider on GPT-powered CoPilot,”
Our reading: There has been much experimentation with ChatGPT-style AI but relatively few well-documented success stories.
No horizon Evidence as of
Limitation: The survey measures deployment and measurement practices, not the number of public success stories.
Assessment: UnresolvedSaid
Observation
“Two years on, on the most important question of all – factuality and reliability — we are still pretty much where we were when ChatGPT first came out: wishing and hoping.”
Our reading: Two years after ChatGPT came out, on factuality and reliability, which he calls the most important question, we are still pretty much where we were at its release: wishing and hoping.
No horizon Evidence as of
Limitation: Neither study establishes aggregate progress from November 2022 to November 2024 across representative factuality and reliability tasks.
Assessment: UnresolvedSaid
Observation
“Yet hundreds of billions more have been invested on further speculation that scaling would somehow magically cure problems that actually appear to be inherent with the technology.”
Our reading: Hundreds of billions more have been invested on further speculation that scaling would somehow magically cure problems that appear to be inherent to the technology.
No horizon Evidence as of
Limitation: The spending categories cannot be added without checking overlap and accounting treatment.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“At the end of the Ezra Klein interview, I called for a massive investment in new approaches to AI. I still think that is the way to go.”
Our reading: He still thinks a massive investment in new approaches to AI, as he called for at the end of the Ezra Klein interview, is the way to go.
No horizon
Held from this post (10)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: ChatGPT's tendency to generate fabricated content is as strong as ever in 2024.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: None of the industry's efforts (scaling, RAG, prompting) has eliminated hallucinations.
- The record does not meet the current rules. Our reading, not assessed: LLM hallucinations may have declined but have not vanished.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: Hallucinations are still a regular occurrence in LLMs.
- The criteria are ambiguous. Our reading, not assessed: Many Fortune 500 companies are somewhat underwhelmed by the real-world results they get from ChatGPT.
- The record does not meet the current rules. Our reading, not assessed: Despite projections by investors and tech companies, hallucinations and errors of misunderstanding in ChatGPT did not soon recede.
- The record does not meet the current rules. Our reading, not assessed: Hallucinations in LLMs still have not gone away despite scaling and RAG.
- The record does not meet the current rules. Our reading, not assessed: RAG, scaling and system prompts have not eradicated LLMs' inherent tendency to hallucinate.