Gaslighting and reality in AI
We recorded 5 claims from this post. 4 were checked against evidence. 1 can’t be checked. 1 more is held. Read the post at its source.
Claims from this post (5)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
“I said the problem wasn’t math per se, but that large language models didn’t represent robust world models of how events fold over time; I stand by this 100% even today.”
What we tested: Large language models do not represent robust world models of how events unfold over time, and this was true from 2019 through early 2023.
Assessment: UnresolvedSaid
“And I still attribute those problems to a lack of world models that represent the ways in which things unfold over time. I still see this as the central flaw in large language models.”
What we tested: The dopey outputs of GPT-style LLMs are caused by a lack of world models of how things unfold over time, which is the central flaw of LLMs.
Assessment: UnresolvedSaid
“But more than three years on, it’s still—even after a new update a few days ago focused around math—having lot of problems with easy word problems like this:”
What we tested: Even after its January 30, 2023 math-focused update, ChatGPT still has many problems with easy word problems.
Assessment: SupportedSaid
“Its output is more fluent, and lots of individual mistakes have been addressed.”
What we tested: Since 2019, GPT's output has become more fluent and many individual mistakes have been fixed.
What happened: Under matched prompting, evaluators distinguished GPT-2 from human text more accurately than GPT-3 from human text across stories, news, and recipes. This supports improved naturalness, although humanlikeness is an indirect fluency measure. (arXiv; Elizabeth Clark and colleagues, )
Assessment: Can’t be checkedSaid
“My goalposts haven’t shifted; I still want A(G)I that knows what the f___ it’s talking about, and GPT still doesn’t.”
What we tested: GPT still does not understand what it is talking about.
Held from this post (1)
These records did not pass every check, so nothing about them is published. Held is not a finding.