“And those demos won’t work all that reliably.”
Our reading, the sentence that was checked:
The AI agent demos of 2025 will not work very reliably.
Assessment: Supported Deadline passed on 2026-10-29 Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AI Agents: Hype versus Reality, redux, paragraph 6.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-ai-agents-hype-versus-reality-redux-0d2294-a3
We recorded 7 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-01-15. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Forecast
- Deadline, as the speaker wrote it
- “this year”
- Deadline state when assessed
- Deadline passed
- Deadline state on 2026-10-29
- Deadline passed (the release’s reference date)
From the speaker
- All that we will have this year are demos
- those demos won't work all that reliably
- this year (2025)
Our interpretation
- Plain reading: AI agents released or shown in 2025 would fail a large share of realistic multi-step tasks, falling well short of the near-perfect reliability needed to trust them without supervision.
- Independent or academic benchmarks of realistic agent tasks published in 2025 are the right test. Top-agent success rates well below 90% on such tasks count as 'not all that reliable'.
- This assessment does not test whether 'all we will have are demos'. It tests only the reliability part.
Counts as supported if: Evaluations published in 2025 show leading agents failing a substantial share (for example, a third or more) of realistic multi-step tasks, or documented live deployments show frequent errors.
Counts as contradicted if: Evaluations published in 2025 show leading agents completing realistic, varied multi-step tasks with high reliability (for example, above about 90%) across the board.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Did the forecast come true?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- Deadline passed
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
In January 2025, Marcus forecast that the year's AI agent demos would not work very reliably. The 2025 horizon has passed. Under the contract, the claim is supported if 2025 evaluations show leading agents failing a substantial share of realistic multi-step tasks. A third or more is the contract's example. In the September 2025 revision of TheAgentCompany, the most competitive agent completed 30% of workplace tasks autonomously (e1). On CRMArena-Pro, published in May 2025, leading agents reached about 58% single-turn success and about 35% multi-turn success on business tasks (e2). On 29 September 2025, Anthropic reported that Claude Sonnet 4.5 led OSWorld, a computer-task benchmark, at 61.4% (e3). In the same release, Anthropic reported 77.2% on SWE-bench Verified, a coding benchmark, averaged over 10 trials (e4). Each of the first three results implies failure on more than a third of tasks, which meets the support condition. None of the four results reaches the contract's contradiction bar of about 90%.
Limitations
- 'Not all that reliably' has no numeric threshold. The outcome depends on the contract's example that failing a third or more of tasks counts as not reliable.
- The evidence is mostly benchmark scores. It does not give failure rates for agents in real deployments.
- Results vary by domain. The 77.2% coding score (e4) is well above the other results.
- The OSWorld and SWE-bench figures (e3, e4) were reported by Anthropic, not by an independent party.
- This assessment does not test the claim that 2025 would bring only demos.
- OpenAI's Operator evaluation page could not be read (HTTP 403).
Evidence (4 items)
Supports the claim: arXiv (Carnegie Mellon University et al.),
find that the most competitive agent can complete 30% of tasks autonomously.
Why it matters here: In the 2025 revision, the best agent finished only 30% of realistic workplace tasks on its own. That is far from reliable.
- Publisher
- arXiv (Carnegie Mellon University et al.)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv (Salesforce AI Research),
leading LLM agents achieve only around 58% single-turn success on CRMArena-Pro, with performance dropping significantly to approximately 35% in multi-turn settings.
Why it matters here: Leading agents failed about 42% of single-turn and about 65% of multi-turn business tasks in mid-2025.
- Where
- Abstract
- Publisher
- arXiv (Salesforce AI Research)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Anthropic,
On OSWorld, a benchmark that tests AI models on real-world computer tasks, Sonnet 4.5 now leads at 61.4%. Just four months ago, Sonnet 4 held the lead at 42.2%.
Why it matters here: Even the leading computer-use agent of late 2025 failed about 39% of real-world computer tasks. Progress was fast, but reliability was still limited.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: Anthropic,
We report 77.2%, which was averaged over 10 trials, no test-time compute, and 200K thinking budget on the full 500-problem SWE-bench Verified dataset.
Why it matters here: Coding agents reached about 77% on a well-defined coding benchmark. This is the strongest 2025 counterpoint, though they still failed nearly a quarter of tasks.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AI Agents: Hype versus Reality, redux
And those demos won’t work all that reliably.
- Where
- paragraph 6
- Published
- 2025-01-15
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-ai-agents-hype-versus-reality-redux-0d2294-a3: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-ai-agents-hype-versus-reality-redux-0d2294-a3.
HeldBy this site
The final-text check failed; overclaims: This is the strongest 2025 counterpoint, though they still failed nearly a quarter of tasks..
About assessment marcus-ai-agents-hype-versus-reality-redux-0d2294-a2, which was not published.