“In many cases, a naive user might take the results, invariably presented as fact, as correct, when they are not.”
Our reading, the sentence that was checked:
GPT-4 with plug-ins invariably presents its results as fact, and in many cases its results are incorrect even though a naive user might take them as correct.
Assessment: Mixed Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Getting GPT to work with external tools is harder than you think, paragraph 17.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-getting-gpt-to-work-with-external-0e36c9-a1
We recorded 16 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- in many cases (scope, stated by the speaker)
- a naive user might take them as correct (qualifier, stated by the speaker)
From the speaker
- In many cases
- a naive user might take the results
- invariably presented as fact
- as correct, when they are not
Our interpretation
- The checkable components are recurring incorrect declarative answers and the assertion that results are invariably presented as factual.
- Might describes a possibility, not an observed rate of users being misled.
- Invariably is read as an absolute. Explicit uncertainty about a substantive result is a counterexample; numerical rounding alone is not.
- The target is the linked June–August 2023 experiments described in the post.
Counts as supported if: The records contain recurring incorrect answers presented confidently, and no substantive qualifications of results within the assessed collection.
Counts as contradicted if: Explicit qualifications contradict invariably; an absence of recurring confidently wrong answers would contradict the separate frequency component.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Mixed
Part of the claim holds and part does not, and the record says which part is which.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The records support recurring confidently wrong answers. They also contain explicit warnings that numerical estimates could differ from reality, so invariably is too strong literally. The possibility of misleading a naive reader is plausible, but actual user acceptance was not measured.
Limitations
- The mixed result depends on reading invariably literally.
- The records come from the underlying study, not an independent replication.
- The motivated-problem date identifies the completed summer testing period; individual response dates are not supplied.
Evidence (2 items)
Supports the claim: New York University; Ernest Davis and Scott Aaronson,
The correct answer is H.
Why it matters here: The answer key for problem 3 is F, while the output confidently selects H. Problems 1 and 4–7 also contain incorrect declarative answers, supporting the recurring confident-error component.
- Publisher
- New York University; Ernest Davis and Scott Aaronson
- Kind of source
- Independent measurement
- Describes events of
- Public from
Goes against the claim: New York University; Scott Aaronson and Ernest Davis,
these are estimates
Why it matters here: The life-in-the-universe calculation explicitly warns about uncertainty in its estimates and repeats that qualification after reporting percentages. This contradicts the absolute assertion that results invariably appear as unqualified facts.
- Publisher
- New York University; Scott Aaronson and Ernest Davis
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Getting GPT to work with external tools is harder than you think
In many cases, a naive user might take the results, invariably presented as fact, as correct, when they are not.
- Where
- paragraph 17
- Published
- 2023-10-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-getting-gpt-to-work-with-external-0e36c9-a1: Current. Outcome: Mixed. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-getting-gpt-to-work-with-external-0e36c9-a1.