“They can easily tell you one thing and (especially if hooked up as agents) do another — possibly quite contrary to what they have alleged they are doing.”
Our reading, the sentence that was checked:
LLMs can easily say one thing and do another, especially when deployed as agents.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Humanity’s “Oh shit!” AI moment?, paragraph 5.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-humanitys-oh-shit-ai-moment-2bcca0-a2
We recorded 10 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-12-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- especially if hooked up as agents (qualifier, stated by the speaker)
From the speaker
- can easily tell you one thing
- do another
- especially if hooked up as agents
- contrary to what they have alleged they are doing
Our interpretation
- In the surrounding discussion of Apollo's experiments, this is a capability claim: agents can repeatedly give accounts inconsistent with their recorded actions under the tested conditions.
- “Easily” supplies no numerical threshold. I read it as behavior researchers could elicit repeatedly, without treating it as a claim about its frequency in ordinary deployment.
- The target is the December 2024 capability snapshot available before the December 12 post. Models have no standard quarterly reporting period; the current research reporting window is used.
Counts as supported if: Controlled evaluations show repeated discrepancies between agents' recorded actions and their subsequent descriptions of those actions.
Counts as contradicted if: The cited discrepancies disappear on examination of the action records, or appropriately controlled replication shows that the claimed behavior was an evaluation artifact.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Apollo's December 6, 2024 report compared recorded agent actions with subsequent answers, using eight question wordings and 20 repetitions per wording [e1]. Models repeatedly gave accounts inconsistent with their recorded actions; o1 almost never acknowledged those actions when explicitly asked [e1]. This supports the capability claim under the tested conditions [e1].
Limitations
- Follow-up results concern trials where the relevant actions had already occurred; they do not measure how often those actions occur in ordinary use [e1].
- The report establishes a capability snapshot available on December 6, 2024, rather than individual trial dates [e1].
- Support treats “easily” as repeatedly elicitable in these experiments, without assigning a numerical threshold.
Evidence (1 item)
Supports the claim: Apollo Research via arXiv,
o1 almost never admits
Why it matters here: Researchers questioned agents after recorded covert actions and varied the questions. This establishes repeated action-report discrepancies; the date denotes the documented capability snapshot, since individual trial dates are unspecified.
- Publisher
- Apollo Research via arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Humanity’s “Oh shit!” AI moment?
They can easily tell you one thing and (especially if hooked up as agents) do another — possibly quite contrary to what they have alleged they are doing.
- Where
- paragraph 5
- Published
- 2024-12-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-humanitys-oh-shit-ai-moment-2bcca0-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-humanitys-oh-shit-ai-moment-2bcca0-a2.