“Every time I think about the numbers and lack of RoI for most pilot studies and the reliability problems I think so.”
Our reading, the sentence that was checked:
Most AI pilot studies show no return on investment, and AI has ongoing reliability problems.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Could generative AI turn out to be the tech industry’s Vietnam? And could public backlash lead AI to a better place?, paragraph 13.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-could-generative-ai-could-turn-out-e82bc9-2-a2
We recorded 7 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-05-20. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- most pilot studies (qualifier, stated by the speaker)
From the speaker
- lack of RoI for most pilot studies
- the reliability problems
Our interpretation
- In context, AI means enterprise generative AI. Pilot studies means trial deployments, rather than published academic studies.
- Most means more than half of pilots. Lack of ROI means pilots have not demonstrated positive financial returns after their costs; it does not mean they can never pay off.
- Reliability problems means recurring factual errors or failures to follow supplied evidence in practical tasks. The claim does not require all systems to fail or performance to be deteriorating.
- The statement describes an ongoing state without specifying a reporting period. I use January–March 2026, the latest completed calendar quarter before the post, as the observation period.
- The two assertions are assessed together. Support for reliability alone cannot settle the majority-of-pilots assertion.
- This is a claim about the prevalence of pilot outcomes, not a named company's financial accounts; metric_scope is null.
Counts as supported if: Representative evidence shows that more than half of enterprise generative-AI pilots had not demonstrated positive financial ROI during the target period, and contemporaneous evaluations document recurring reliability failures.
Counts as contradicted if: Comparable representative evidence shows that at least half of pilots had demonstrated positive financial ROI, or sufficiently broad evaluations establish that the asserted recurring reliability problems no longer persisted. Opposing findings on the two components would warrant mixed.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The March 25, 2026 preprint supports continuing reliability problems through its evaluation of hallucination and factual grounding. [e4] The assertion that most enterprise generative-AI pilots lacked demonstrated positive financial ROI in January–March 2026 remains unsettled by the available studies. [e1, e2, e3] NANDA reports limited measurable financial impact in its January–June 2025 research, outside the target period. [e1] KPMG's survey ending March 17, 2026 classified 8% of organizations as having established ROI; it did not count pilot outcomes. [e2] ETR's January 21–28, 2026 survey found half of respondents said AI value exceeded cost, but this also does not establish the share of pilots with positive ROI. [e3] These different periods and units leave the majority-of-pilots assertion unresolved despite support for the reliability component. [e1, e2, e3, e4]
Limitations
- No cited study provides representative pilot-level counts with comparable financial costs and returns for January–March 2026. [e1, e2, e3]
- NANDA's absence of measurable P&L impact does not establish measured nonpositive ROI. [e1]
- The reliability evidence is a preprint covering selected models and tasks; its submission date does not establish when the experiments occurred. [e4]
Evidence (4 items)
Supports the claim: arXiv; Zhuo Li and coauthors,
Hallucination remains
Why it matters here: The paper documents continuing hallucination and factual-grounding problems while evaluating a method that reduces them. It supports the reliability component as reported at submission, without establishing any relationship to enterprise pilot ROI.
- Publisher
- arXiv; Zhuo Li and coauthors
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Project NANDA; copy hosted by MLQ,
no measurable P&L
Why it matters here: The preliminary report supports Marcus's concern in its January–June 2025 sample, but falls outside the target period. Its interview-based estimates, variable success definitions, and P&L-impact measure do not establish a representative 2026 pilot ROI rate.
- Publisher
- Project NANDA; copy hosted by MLQ
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: KPMG International,
enterprise value.
Why it matters here: The survey ended March 17 after collecting responses from 2,110 business leaders. Its 8% figure classifies organizations by AI maturity and established enterprise outcomes; it does not count profitable and unprofitable pilots.
- Publisher
- KPMG International
- Kind of source
- Independent measurement
- Describes events of
- Public from
Context only: ETR Research,
AI value outweighs cost
Why it matters here: ETR surveyed 600 technology leaders January 21–28 and found substantial reported value relative to cost. This challenges a sweeping inference of absent business returns, but respondent-level results cannot contradict a claim about the majority of pilots.
- Publisher
- ETR Research
- Kind of source
- Independent measurement
- Describes events of
- Public from
What would settle it
A representative study of enterprise generative-AI pilots observed in January–March 2026, reporting the pilot denominator, sampling method, costs, realized financial benefits, and unmeasured outcomes separately, would settle whether more than half lacked demonstrated positive financial ROI.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Could generative AI turn out to be the tech industry’s Vietnam? And could public backlash lead AI to a better place?
Every time I think about the numbers and lack of RoI for most pilot studies and the reliability problems I think so.
- Where
- paragraph 13
- Published
- 2026-05-20
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-could-generative-ai-could-turn-out-e82bc9-2-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-could-generative-ai-could-turn-out-e82bc9-2-a2.