The core problems of reliability and factuality in LLMs have not been solved.

Was it true for the period described?

Outcome: Supported

The dated evidence matches what the claim committed to, under the criteria frozen before grading.

Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.

Evidence as of

2026-10-28

Deadline state when assessed

No horizon

Assessment recorded

2026-10-29

Policy version

2026-10-28.7

Rationale

The support condition asks for tests from about March to June 2026 that show frontier models still giving confident wrong answers at substantial rates. On 2026-04-23, Artificial Analysis, an independent benchmark operator, reported hallucination rates on its AA-Omniscience knowledge benchmark (e1). This rate measures how often a model gives a wrong answer instead of declining when it does not answer correctly. It was 86% for GPT-5.5 (xhigh), 50% for Gemini 3.1 Pro Preview and 36% for Claude Opus 4.7 (max) (e1). The same source said GPT-5.5's gain over GPT-5.4 came mostly from knowledge, with a modest improvement in hallucination (e1). A secondary summary published 2026-04-29 reports that OpenAI's system card found GPT-5.5's claims 23% more likely to be correct than GPT-5.4's, with factual errors 3% less often (e2). Those gains are relative to the previous model. They do not report near-zero error rates. The support condition is met. The contradiction condition, near-zero error rates on standard factuality benchmarks, is not met by any item in the packet.

Limitations

• AA-Omniscience uses hard knowledge questions and counts only non-correct answers. It does not show error rates on everyday tasks.
• 'Reliability' is broader than factuality. This assessment uses hallucination and factual-error measures only.
• The system-card figures come from a secondary summary, not the system card itself.
• There is no agreed threshold for when hallucination counts as solved.

Evidence (2 items)

• Supports the claim: Artificial Analysis, 2026-04-23

  However, it has a hallucination rate of 86% - vs Opus 4.7 (max) at 36%, and Gemini 3.1 Pro Preview at 50%. This makes it more likely to answer a question when it does not 'know' the answer.

  Why it matters here: An independent test from April 2026 shows the three leading frontier models still giving confident wrong answers instead of abstaining, at rates of 36% to 86% of non-correct responses. That is not a solved problem.

  Where

  Key takeaways, bullet 'Top AA-Omniscience accuracy, but trailing the frontier on hallucination'

  Publisher

  Artificial Analysis

  Kind of source

  Benchmark result

  Describes events of

  2026-04-23

  Public from

  2026-04-23

• Context only: AI2Work, 2026-04-23

  OpenAI's own system card reports GPT-5.5's individual claims are 23% more likely to be factually correct than GPT-5.4's, and responses contain factual errors 3% less often.

  Why it matters here: Shows real but incremental factuality gains reported by OpenAI. The gains are measured against a prior model and do not claim factual errors were eliminated.

  Where

  section 'The Benchmark Disconnect'

  Publisher

  AI2Work

  Kind of source

  Secondary summary

  Describes events of

  2026-04-23

  Public from

  2026-04-29