What to Expect When You’re Expecting … GPT-4
We recorded 27 claims from this post. 24 were checked against evidence. 3 can’t be checked. 14 more are held. Read the post at its source.
Claims from this post (27)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Observation
“As I told NYT columnist Farhad Manjoo, ChatGPT, like earlier, related systems is “still not reliable, still doesn’t understand the physical world, still doesn’t understand the psychological world and still hallucinates.””
Our reading: ChatGPT is not reliable, does not understand the physical or psychological world, and hallucinates.
No horizon Evidence as of
Limitation: The result depends on treating understanding as practical reasoning competence.
Assessment: SupportedSaid
Observation
“Although ChatGPT can write about anything, it is also easily confused.”
Our reading: ChatGPT is easily confused.
No horizon Evidence as of
Limitation: The arithmetic demonstrations were selected to probe weaknesses.
Assessment: UnresolvedSaid
Observation
“Which is to say it makes stuff up, regularly; a fair bit of what it is says simply isn’t true.”
Our reading: ChatGPT regularly makes things up and a fair amount of its output is false.
No horizon Evidence as of
Limitation: This does not mean fabrication was absent or rare.
Assessment: SupportedSaid
Observation
“In short, ChatGPT might sound like the Star Trek computer, but for now, you still can’t trust it.”
Our reading: ChatGPT cannot currently be trusted.
No horizon Evidence as of
Limitation: The conclusion concerns unverified factual reliance, not every possible use.
Assessment: SupportedSaid
Forecast
“I guarantee that minds will be blown.”
Our reading: GPT-4 will impress people enormously when released.
Settled before its deadline Evidence as of
Limitation: The accounts are illustrative, not a representative opinion survey.
Assessment: UnresolvedSaid
Forecast
“GPT-4 will be trained on considerably more, a significant fraction of the internet as a whole.”
Our reading: GPT-4 will be trained on considerably more data than GPT-3, a significant fraction of the internet.
No deadline stated Evidence as of
Limitation: This is an evidence gap, not evidence that GPT-4 used little data.
Assessment: SupportedSaid
Forecast
“As OpenAI has learned, bigger in many ways means better, with outputs more and more humanlike with each iteration. GPT-4 is going to be a monster.”
Our reading: Because scale improves output in many ways, GPT-4 will be very powerful.
Settled before its deadline Evidence as of
Limitation: The decisive evaluation was conducted by the model developer.
Assessment: UnresolvedSaid
Forecast
“Although GPT-4 will definitely seem smarter than its predecessors, its internal architecture remains problematic.”
Our reading: GPT-4 will seem smarter than predecessors but its architecture remains problematic.
No deadline stated Evidence as of
Limitation: The comparative evaluation was conducted by the model's developer.
Assessment: Can’t be checkedSaid
Forecast
“I suspect that what we will see is a familiar pattern: immense initial buzz, followed by a more careful scientific inspection, followed by a recognition that many problems remain.”
Our reading: GPT-4 will likely generate immense buzz followed by scientific scrutiny and recognition that many problems remain.
No horizon
Assessment: UnresolvedSaid
Conditional forecast
“GPT-4 may be better at faking term papers, but if it follows the same playbook as it predecessors, it still won’t really understand the world, and seams will eventually show.”
Our reading: If GPT-4 follows its predecessors' approach, it will not truly understand the world and its flaws will eventually show.
No deadline stated Evidence as of
Only if: if it follows the same playbook as its predecessors
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If all seven predictions prove correct, I hope that the field will finally realize that it is time to move on.”
Our reading: Marcus hopes that, if all seven of his predictions prove correct, the field will finally realize that it is time to move on.
No horizon
Only if: if all seven predictions prove correct
Assessment: SupportedSaid
Forecast
“Guardrails (a la ChatGPT) may be in place, but the guardrails will teeter between being too weak (beaten by “jailbreaks”) and too strong (rejecting some perfectly reasonable requests).”
Our reading: Guardrails like ChatGPT's may be in place for GPT-4, but they will teeter between being too weak (beaten by jailbreaks) and too strong (rejecting some perfectly reasonable requests).
Settled before its deadline Evidence as of
Limitation: The evidence establishes both failure modes, not repeated oscillation between safety settings over time [e1, e2, e3].
Assessment: UnresolvedSaid
Forecast
“Without external aids it won’t be able beat Meta’s Cicero in Diplomacy; it won’t be able to drive a car reliably; it won’t be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.”
Our reading: Without external aids, GPT-4 won't be able to beat Meta's Cicero in Diplomacy, won't be able to drive a car reliably, and won't be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.
No deadline stated Evidence as of
Limitation: The assistance boundary materially affects which demonstrations qualify.
Assessment: SupportedSaid
Forecast
“It will often do what you want, sometimes not—and it will remain difficult to anticipate which in advance..”
Our reading: It will remain difficult to anticipate in advance when GPT-4 will or won't do what users want.
Settled before its deadline Evidence as of
Limitation: The finding is strongest for task correctness and settings where mistakes carry high costs.
Assessment: UnresolvedSaid
Forecast
“Its natural language output still won’t be something that one can reliably hook up to downstream programs; it won’t be something, for example, that you can simply and directly hook up to a database or virtual assistant, with predictable results.”
Our reading: GPT-4's natural language output still won't be something one can reliably hook up to downstream programs, for example simply and directly to a database or virtual assistant with predictable results.
No deadline stated Evidence as of
Limitation: The numerical evaluation is provider-reported and concerns complex schemas.
Assessment: UnresolvedSaid
Forecast
“GPT-4 will not have reliable models of the things that it talks about that are accessible to external programmers in a way that reliably feeds downstream processes.”
Our reading: GPT-4 will not have reliable models of the things it talks about that are accessible to external programmers in a way that reliably feeds downstream processes.
No deadline stated Evidence as of
Limitation: The meaning of accessible models is ambiguous.
Assessment: SupportedSaid
Forecast
“The system will still not be able to restrict its output to reliably following a shared set of human values around helpfulness, harmlessness, and truthfulness.”
Our reading: GPT-4 will not reliably follow human values of helpfulness, harmlessness and truthfulness.
Settled before its deadline Evidence as of
Limitation: The judgment depends on an ordinary, nonnumerical reading of reliability.
Assessment: UnresolvedSaid
Forecast
“In retrospectives written in 2043, intellectual historians will conclude that there was an initial overemphasis on large language models, and a gradual but critical shift of the pendulum back to more structured systems with deeper comprehension.”
Our reading: In retrospectives written in 2043, intellectual historians will conclude that there was an initial overemphasis on large language models and a gradual but critical shift of the pendulum back to more structured systems with deeper comprehension.
Not yet due Evidence as of
Limitation: The forecast leaves the author population and degree of agreement unspecified.
Assessment: SupportedSaid
Forecast
“GPT-4 will solve many of the individual specific items used in prior benchmarks, but still get tripped up, particularly in longer and more complex scenarios.”
Our reading: GPT-4 will solve many prior benchmark items but still fail, especially on longer, complex scenarios.
Settled before its deadline Evidence as of
Limitation: Benchmark success does not establish whether individual items were learned or memorized.
Assessment: SupportedSaid
Forecast
“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”
Our reading: GPT-4 will make a significant number of hard-to-predict stupid errors.
Settled before its deadline Evidence as of
Limitation: The judgment depends on reading hard to predict as unexpectedly uneven performance, rather than requiring a direct prediction experiment.
Assessment: SupportedSaid
Forecast
““Alignment” between what humans want and what machines do will continue to be a critical, unsolved problem.”
Our reading: Alignment will remain a critical unsolved problem with GPT-4.
Settled before its deadline Evidence as of
Limitation: This assesses behavioral alignment in GPT-4, not every meaning of alignment.
Assessment: UnresolvedSaid
Forecast
““Scaling” alone—building bigger and models until they absorb the entire internet — will prove useful, but only to a point.”
Our reading: Scaling alone will prove useful only up to a point.
No deadline stated Evidence as of
Limitation: The claim specifies no deadline, capability threshold, or operational test of trustworthy general intelligence.
Assessment: SupportedSaid
Forecast
“Some of its advice will be head-scratchingly bad.”
Our reading: Some of GPT-4's advice will be very bad.
Settled before its deadline Evidence as of
Limitation: The study concerns a particular infectious-disease setting and does not measure all GPT-4 advice.
Assessment: Can’t be checkedSaid
Conditional forecast
“When AGI (artificial intelligence) comes, large language models like GPT-4 may be seen in hindsight as part of the eventual solution, but only as part of the solution.”
Our reading: When AGI comes, large language models like GPT-4 may be seen in hindsight as part of the eventual solution, but only as part of it.
No horizon
Only if: when AGI comes
Assessment: SupportedSaid
Forecast
“When queried on medicine it will either resist answering (if there are aggressive guardrails) or occasionally spout plausible-sounding but dangerous nonsense.”
Our reading: On medical queries GPT-4 will either refuse (with aggressive guardrails) or occasionally produce plausible but dangerous nonsense.
Settled before its deadline Evidence as of
Limitation: The decisive evaluation concerns bloodstream-infection management and the tested GPT-4 version.
Assessment: SupportedSaid
Forecast
“It will remain turbocharged pastiche generator, and a fine tool for brainstorming, and for first drafts, but not trustworthy general intelligence.”
Our reading: GPT-4 will be a pastiche generator useful for brainstorming and drafts but not trustworthy general intelligence.
Settled before its deadline Evidence as of
Limitation: The experiment evaluates people working with GPT-4 on consulting tasks, not dependable performance across all general tasks.
Assessment: SupportedSaid
Forecast
“GPT-4 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
Our reading: GPT-4 will be reckless and hard to control like its predecessors.
Settled before its deadline Evidence as of
Limitation: Adversarial test results do not establish an ordinary-use failure rate.
Held from this post (14)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: GPT-4 by itself will not be an AGI capable of arbitrary tasks.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: GPT-4's reasoning about physical, psychological and mathematical worlds will remain unreliable.
- The criteria are ambiguous. Our reading, not assessed: Bad actors will eventually replicate much of GPT-4 without guardrails and use knock-offs to create any narratives they wish.
- The record does not meet the current rules. Our reading, not assessed: GPT-4 will still commonly and easily hallucinate, escalating the risk of LLM-generated misinformation.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: ChatGPT has guardrails in place now, but sometimes those guardrails are poor.
- The record does not meet the current rules. Our reading, not assessed: GPT-4 will not be trustworthy and complete enough to give reliable medical advice, despite having absorbed a large fraction of the Internet.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: Examples of concealed bias in GPT-4 will be discovered within days or months of release.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews read the claim differently