“The unrelenting hallucinations problem frequently rears its ugly head:”
Our reading, the sentence that was checked:
Hallucinations frequently occur in AI agents.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AI Agents have, so far, mostly been a dud, paragraph 24.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-ai-agents-have-so-far-mostly-been-38b9ed-a2
We recorded 23 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-08-03. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- The unrelenting hallucinations problem
- frequently rears its ugly head
- (in AI agents, per context)
Our interpretation
- He names no period. I read it as describing AI agents in 2025 up to the post date (2025-08-03).
- Plain reading: in 2025, fabricated information (made-up facts, people, accounts, or system states) was a recurring, well-documented failure of deployed or tested AI agents, not a rare one-off.
Counts as supported if: Several documented 2025 agent deployments or tests show agents making up facts, people, accounts, or results.
Counts as contradicted if: 2025 agent evaluations and deployment reports show fabrication to be rare or negligible.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Marcus said hallucinations frequently show up in AI agents. Anthropic's report on its shop-running agent (spring 2025) documents several fabrications. The agent invented a person named Sarah, a Venmo account, and a meeting with security. In July 2025, Ars Technica reported two separate coding agents, Google's Gemini CLI and Replit, that confabulated operations that never happened. Replit's agent also produced fake data and false test results. These are repeated, documented cases across several vendors in the months before the post, which fits the plain reading of 'frequently'.
Limitations
- No standard rate measure; 'frequently' is judged from repeated documented cases.
- Incident reports may overrepresent failures.
Evidence (2 items)
Supports the claim: Anthropic,
On the afternoon of March 31st, Claudius hallucinated a conversation about restocking plans with someone named Sarah at Andon Labs—despite there being no such person.
Why it matters here: Anthropic's own report on its shop-running agent describes it inventing a person and a conversation. The same report lists a hallucinated Venmo account and a hallucinated meeting with security.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: Ars Technica,
The core issue appears to be what researchers call “confabulation” or “hallucination”—when AI models generate plausible-sounding but false information. In these cases, both models confabulated successful operations and built subsequent actions on those false premises.
Why it matters here: Two separate coding agents from different companies (Google Gemini CLI and Replit) hallucinated in July 2025. The article also says Replit created fake data and false test results.
- Where
- paragraph 5
- Publisher
- Ars Technica
- Kind of source
- News report
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AI Agents have, so far, mostly been a dud
The unrelenting hallucinations problem frequently rears its ugly head:
- Where
- paragraph 24
- Published
- 2025-08-03
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-ai-agents-have-so-far-mostly-been-38b9ed-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-ai-agents-have-so-far-mostly-been-38b9ed-a2.