ChatGPT in Shambles
We recorded 6 claims from this post. 4 were checked against evidence. 2 can’t be checked. 3 more are held. Read the post at its source.
Claims from this post (6)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Observation
“failed, multiple times, to properly count to 50”
Our reading: In Marcus's test, ChatGPT failed several times to count to 50 correctly.
No horizon Evidence as of
Limitation: The shared transcript linked in Marcus's post was not found, so the rows in his tables could not be counted.
Assessment: UnresolvedSaid
Observation
“failed, multiple times, to include a full list of all US states”
Our reading: In Marcus's test, the free version of ChatGPT failed several times to produce a complete list of all 50 US states.
No horizon Evidence as of
Limitation: The shared transcript linked in Marcus's post was not found, so none of his tables could be checked against the list of 50 states.
Assessment: UnresolvedSaid
Observation
“reported that the letter h could be a vowel, at least when it appeared in the word Northwest”
Our reading: In Marcus's test, ChatGPT reported that the letter h could be a vowel, at least in the word Northwest.
No horizon Evidence as of
Limitation: The shared transcript and the screenshots of the Northwest Territories answer were not found.
Assessment: UnresolvedSaid
Observation
“issued numerous corrections that were wrong, never acknowledging uncertainty until after its errors were called out.”
Our reading: In Marcus's test, ChatGPT issued many wrong corrections and acknowledged no uncertainty until its errors were pointed out.
No horizon Evidence as of
Limitation: No substantive conversation messages were available as admissible outcome evidence.
Assessment: Can’t be checkedSaid
Observation
“(In fairness, ChatGPT doesn’t really lie; it just spews text that often bears little resemblance to reality, but you get my drift).”
Our reading: ChatGPT does not truly lie, but it often produces text that bears little resemblance to reality.
No horizon
Assessment: Can’t be checkedSaid
Observation
“As against all the constant claims of exponential progress that I see practically every day, ChatGPT still seems likes pretty much the same mix of brilliance and stupidity that I wrote about, more than two years ago:”
Our reading: Despite constant claims of exponential progress, ChatGPT still seems like pretty much the same mix of brilliance and stupidity that Marcus wrote about more than two years earlier.
No horizon
Qualifier: still seems like
Held from this post (3)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The criteria are ambiguous. Our reading, not assessed: In Marcus's test, ChatGPT could not count the vowels in the names of Canadian provinces correctly.
- The two reviews read the claim differently
- Attribution is uncertain