AI Agents: Hype versus Reality, redux
We recorded 6 claims from this post. 5 were checked against evidence. 1 can’t be checked. 7 more are held. Read the post at its source.
Claims from this post (6)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Forecast
“I do genuinely think we will all have our own AI agents, and companies will have armies of them.”
Our reading: In the long term, everyone will have their own AI agents and companies will have armies of them.
No deadline stated Evidence as of
Limitation: The packet has no population-level data on personal AI agent use.
Assessment: UnresolvedSaid
Forecast
“And they will be worth trillions, since eventually (no time soon) they will do a huge fraction of all human knowledge work, and maybe physical labor too.”
Our reading: Eventually, though not soon, AI agents will do a huge fraction of human knowledge work, perhaps physical labor too, and be worth trillions.
No deadline stated Evidence as of
Limitation: The only evidence is one benchmark from late 2025. It measures what agents can do on a set of projects, not what share of real knowledge work they actually do.
Assessment: ContradictedSaid
Forecast
“All that we will have this year are demos.”
Our reading: In 2025, AI agents will exist only as demos.
Deadline passed Evidence as of
Limitation: The result depends on reading 'demos' as covering all AI agent products, not only general-purpose ones.
Assessment: Can’t be checkedSaid
Forecast
“But not this year (or next, or the one after that, and probably not this decade, except in narrow use cases).”
Our reading: AI agents probably will not do a huge fraction of human knowledge work this decade, except in narrow use cases.
No horizon
Qualifier: except in narrow use cases
Assessment: SupportedSaid
Forecast
“The thing about scheduling is that it needs to work reliably. And it won’t.”
Our reading: OpenAI's ChatGPT task-scheduling agent will not work reliably.
Settled before its deadline Evidence as of
Limitation: The evidence is anecdotal. There is no measured failure rate.
Assessment: SupportedSaid
Observation
“That is in part because you can’t build reliable agents unless you can solve the hallucination problem, and there is still no principled trustworthy answer to that problem.”
Our reading: As of January 2025 there is no principled, trustworthy solution to the hallucination problem.
No horizon Evidence as of
Limitation: 'Principled trustworthy answer' has no stated threshold. The assessment uses the contract's reading: a method that reliably prevents hallucinations in general use.
Held from this post (7)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: Reliable AI agents cannot be built unless the hallucination problem is solved.
- The criteria are ambiguous. Our reading, not assessed: If what you want is agents with common sense, sound reasoning, tight grounding in reality, reliable instruction-following and enough theory of mind to infer intentions, generative AI alone is not the ticket, because those are exactly the sticking points that have stymied it over and over again.
- The two reviews reached different assessments. Our reading, not assessed: Throughout 2025 there will be many more reports of AI agents being brittle and failing.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: As with generative AI, with AI agents some prompts will work some of the time and some of the time they will give strange results, and it will be difficult to know in advance what to expect.