“It could sometimes produce outputs that were correct and acceptable in these regards but not reliably.”
Our reading, the sentence that was checked:
ChatGPT could sometimes produce outputs that were correct and acceptable in the regards listed (counting, arithmetic, event order, physical reasoning, human psychology, fabrication, bias), but not reliably.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Large Language Models like ChatGPT say The Darnedest Things, paragraph 3.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-large-language-models-like-chatgpt-17d031-a1
We recorded 10 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-01-09. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- in these regards (the failure types in figures 1-7) (scope, stated by the speaker)
- sometimes, but not reliably (qualifier, stated by the speaker)
From the speaker
- could sometimes produce outputs that were correct and acceptable
- in these regards
- but not reliably
Our interpretation
- Supported if release-era ChatGPT gave correct or acceptable answers on some tries and wrong or unacceptable ones on others, for the same kinds of prompts.
- The period is 2022-11-30 to 2023-01-09.
Counts as supported if: Evidence that release-era ChatGPT's outputs varied, being correct on some tries or phrasings and wrong on others, in reasoning, factual accuracy or bias.
Counts as contradicted if: Evidence that ChatGPT was consistently correct in these areas, or consistently wrong with no correct outputs.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
OpenAI's launch post says ChatGPT was sensitive to rephrasing and retries: it could fail one phrasing of a question and answer a slight rephrase correctly. On bias, Piantadosi told The Daily Beast in December 2022 that the harmful outputs did not appear 'every time exactly the same', while plain requests got refusals. Bang et al. found early ChatGPT right on about 63% of reasoning items, which fits correct sometimes but not reliably. Together these support the claim that outputs were sometimes correct and acceptable but inconsistent.
Limitations
- The evidence covers the seven failure types unevenly.
- The benchmark's test dates are not stated, so it is used as context only.
Evidence (3 items)
Supports the claim: OpenAI,
ChatGPT is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times. For example, given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.
Why it matters here: OpenAI confirms that at launch the same question could get correct or failed answers depending on phrasing and retries.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: The Daily Beast (via Yahoo News),
When I asked for things in nonstandard ways—for example, as a table or as a program—ChatGPT was happy to write horrible stuff. It doesn't do it every time exactly the same, but it's clear there is a ton of bad content inside of these systems.
Why it matters here: Shows that on bias, outputs varied: sometimes acceptable refusals, sometimes harmful content.
- Where
- Paragraph 18
- Publisher
- The Daily Beast (via Yahoo News)
- Kind of source
- News report
- Describes events of
- Public from
Supports the claim: arXiv (Bang et al., AACL 2023),
ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner.
Why it matters here: Early ChatGPT was right on about two thirds of reasoning items: correct sometimes, but not reliably. The test dates are not stated, so this is context only.
- Where
- Abstract
- Publisher
- arXiv (Bang et al., AACL 2023)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Large Language Models like ChatGPT say The Darnedest Things
It could sometimes produce outputs that were correct and acceptable in these regards but not reliably.
- Where
- paragraph 3
- Published
- 2023-01-09
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-large-language-models-like-chatgpt-17d031-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-large-language-models-like-chatgpt-17d031-a1.