Is ChatGPT Really a “Code Red” for Google Search?
We recorded 13 claims from this post. 6 were checked against evidence. 7 can’t be checked. 6 more are held. Read the post at its source.
Claims from this post (13)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“The trouble is, those guardrails don’t come for free; the system is still shallow; ChatGPT doesn’t really know what it is it is guarding against; it mostly seems instead to be just looking for keywords, which is why it is prone to generating nonsense like this:”
What we tested: ChatGPT's guardrails don't really understand what they guard against and appear to rely mostly on keywords, which makes ChatGPT prone to producing nonsense.
Assessment: Can’t be checkedSaid
“I like to call this sort of nonsense discomprehension: mindless answers that show that the systems has no clue what it is that is talking about.”
What we tested: ChatGPT's mindless answers ('discomprehension') show it has no clue what it is talking about.
Assessment: Can’t be checkedSaid
“There’s another problem, too, arguably even bigger concern in the context of search engines: hallucination.”
What we tested: For search engines, hallucination is a further problem for LLM-based systems and arguably an even bigger concern than the shallowness, or discomprehension, he described just before.
Assessment: UnresolvedSaid
“Once again, the result was fluent prose, and once again, hallucination ruled:”
What we tested: YouChat, though tailored for search, produced fluent but hallucinated answers to his friend's queries.
Assessment: Can’t be checkedSaid
“It is insidious the way that truth and falsity are so thoroughly and authoritatively mixed together. I for one am not ready for our post-truth information overlords.”
What we tested: The way truth and falsity are thoroughly and authoritatively mixed together in LLM-driven search answers like the ones he examined (such as the Neeva bio) is insidious, and he personally is not ready for 'post-truth information overlords'.
Assessment: SupportedSaid
“Large language models are not databases. They are glommers-together-of-bits-that don’t always belong together.”
What we tested: LLMs are not databases; they combine bits of text that don't always belong together.
What happened: The technical description identifies generation as successive probabilistic predictions. This supports the distinction from retrieving explicit records; the database comparison is the evaluator's inference from the described mechanism. (BigScience collaboration / arXiv, )
Assessment: UnresolvedSaid
“But as they do so, as they glom stuff together, something often gets lost in translation: which bits of text do and do not truly belong together.”
What we tested: When LLMs paraphrase and combine text, they often lose track of which pieces truly belong together, producing errors.
Assessment: UnresolvedSaid
“As it happens, large language models are also wicked hard to update, typically requiring full retraining, sometimes taking weeks or months.”
What we tested: LLMs are very hard to update, usually needing full retraining that can take weeks or months.
Assessment: UnresolvedSaid
“Which means for example that ChatGPT, released in November, is so old it doesn’t know who owns Twitter:”
What we tested: ChatGPT's knowledge is out of date; it doesn't know who owns Twitter.
Assessment: Can’t be checkedSaid
“But until those bits all fit together in a way we can trust, I prefer to borrow a memorable phrase from Ariana Grande: Thank U, Next.”
What we tested: Until the pieces of LLM-based search fit together in a way that can be trusted, he personally prefers to pass on it ('Thank U, Next').
Assessment: Can’t be checkedSaid
“In the quest for a newfangled search I suspect we are once again in for a rough ride.”
What we tested: Developing LLM-based search will probably be a difficult, drawn-out process, as with driverless cars.
Assessment: SupportedSaid
“Driverless cars have taken many years longer than we were originally told; outliers (AKA edge cases) have, at least so far, prevented them from making the transition from demo to widely-available reality.”
What we tested: Driverless cars have taken years longer than promised, because edge cases have so far kept them from becoming widely available.
What happened: The report documents missed rollout expectations, including Tesla's 2020 robotaxi promise, and explains how unexpected situations create stops requiring human help. This supports a material operational barrier to expansion, though it does not isolate edge cases from other causes. (Reuters, republished by Business Recorder, )
Assessment: Can’t be checkedSaid
“When it comes to LLM-driven search, caveat emptor!”
What we tested: Users should be cautious when relying on LLM-driven search.
Held from this post (6)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- Attribution is uncertain
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: LLMs are text predictors, essentially turbocharged autocomplete, not databases.
- The record does not meet the current rules. Our reading, not assessed: Hybrid search/LLM systems are only at proof-of-concept stage and are nothing like systems that can be relied upon.
- The record does not meet the current rules. Our reading, not assessed: Perplexity.ai and you.com are exploring an interesting idea in hybrids of classical search and LLMs, which may allow swifter updates.