“Just like we can’t guarantee that they won’t make stuff up, we don’t know how to guarantee that they won’t write flawed code.”
Our reading, the sentence that was checked:
We cannot guarantee that AI systems will not fabricate or write flawed code.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What was 60 Minutes thinking, in that interview with Geoff Hinton?, paragraph 62.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-was-60-minutes-thinking-in-that-41beff-a1
We recorded 20 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-10. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- we can’t guarantee that they won’t make stuff up
- we don’t know how to guarantee that they won’t write flawed code
Our interpretation
- The subject is contemporary general-purpose language models producing open-ended factual answers and code.
- Guarantee means general assurance against these errors, not a high benchmark score or successful checking of an individual output.
- This is a statement about the technology available in 2023, not a claim that such guarantees are impossible forever.
- For this ongoing technical state, use the 2023 reporting window through publication; there is no standard quarterly reporting cycle.
Counts as supported if: Contemporary developer disclosures acknowledge factual fabrication, and primary tests demonstrate flawed generated code despite ordinary validation, with no established general guarantee found.
Counts as contradicted if: A method available by the post provides a generally applicable guarantee that these systems' open-ended factual answers and generated code are free of the stated errors.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporary developer disclosures acknowledged factual fabrication, and independent code evaluations demonstrated flawed outputs that passed existing tests. These records support the absence of general assurances for the open-ended chatbot generation discussed by Marcus. They do not establish that every output is unreliable or that individual programs cannot be checked.
Limitations
- This is an empirical assessment of contemporary general-purpose systems, not a mathematical impossibility claim.
- Guarantees for restricted tasks would need to be assessed under their own specifications and assumptions.
Evidence (2 items)
Supports the claim: OpenAI, via arXiv,
makes reasoning errors
Why it matters here: The developer explicitly acknowledges factual fabrication in its contemporary model. The surrounding discussion recommends review and grounding appropriate to the application rather than claiming assured factual correctness.
- Publisher
- OpenAI, via arXiv
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: arXiv; Jiawei Liu and colleagues,
logically flawed solution
Why it matters here: The paper demonstrates a ChatGPT-generated sorting error missed by the original benchmark tests and exposed by additional inputs. This shows why successful test results did not guarantee correct generated code.
- Publisher
- arXiv; Jiawei Liu and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What was 60 Minutes thinking, in that interview with Geoff Hinton?
Just like we can’t guarantee that they won’t make stuff up, we don’t know how to guarantee that they won’t write flawed code.
- Where
- paragraph 62
- Published
- 2023-10-10
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-was-60-minutes-thinking-in-that-41beff-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-was-60-minutes-thinking-in-that-41beff-a1.