“No LLM is fully coherent in it’s “beliefs”, because by nature they parrot a variety conflicting things from their training, which is itself inconsistent, rather than reasoning their way a coherent view the way a rational person might.”
Our reading, the sentence that was checked:
No LLM has fully coherent beliefs, because LLMs repeat conflicting material from inconsistent training data instead of reasoning their way to a coherent view.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, 1984, but with LLM’s, paragraph 8.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-1984-but-with-llms-8e842e-a3
We recorded 4 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-21. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- No LLM is fully coherent in its 'beliefs'
- because by nature they parrot a variety of conflicting things from their training
- training ... is itself inconsistent
- rather than reasoning their way to a coherent view
Our interpretation
- The claim has two parts. First, every LLM shows inconsistent stated views across related prompts. Second, the cause is that LLMs repeat conflicting training data instead of reasoning.
- Both parts must be supported. The period is the state of LLMs up to the post date.
Counts as supported if: Consistency studies show that major LLMs give inconsistent opinions across paraphrased or related prompts, and controlled studies trace that inconsistency to conflicting training data rather than to other causes.
Counts as contradicted if: Some LLM is shown to be fully consistent, or studies show the inconsistency comes mainly from something other than conflicting training data (for example prompt format or post-training).
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The claim has two parts, and the contract requires support for both. The first part is that no LLM is fully coherent. An ACL 2024 paper (first posted February 2024) found that models gave different answers depending on how they were forced to answer. It also found that their answers lacked paraphrase robustness (e1). That fits the claim for the models it tested. It does not cover every LLM. The second part is that the cause is repeating conflicting training data instead of reasoning. No study in the evidence tests this cause. Rozado's February 2024 preprint gives preliminary evidence that political preferences may be embedded mostly after pretraining, during fine-tuning or reinforcement learning (e2). The author calls this not conclusive. It is about where political preferences come from, not about coherence. Neither the support condition nor the contradiction condition is met, so the outcome is unresolved.
Limitations
- 'Parrot' versus 'reason' is partly a theoretical framing that studies do not measure directly.
- The consistency evidence comes from one study. It covers only the models that study tested, and the abstract does not name them.
- No evidence covers every LLM, so the absolute 'no LLM' cannot be checked.
Evidence (2 items)
Context only: arXiv (ACL 2024),
We show that models give substantively different answers when not forced; that answers change depending on how models are forced; and that answers lack paraphrase robustness.
Why it matters here: Supports the incoherence part for the models tested. It does not test whether conflicting training data is the cause.
- Where
- Abstract
- Publisher
- arXiv (ACL 2024)
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: arXiv,
Though not conclusive, our results provide preliminary evidence for the intriguing hypothesis that the embedding of political preferences into LLMs might be happening mostly post-pretraining.
Why it matters here: Suggests that stated views are shaped largely by fine-tuning, not only by pretraining data. That complicates the causal story, but it is preliminary and not about coherence directly.
- Where
- Abstract
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Controlled studies that train models on consistent versus conflicting data and then measure how consistent their stated opinions are, plus broad consistency tests of the major models available up to mid-2025.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, 1984, but with LLM’s
No LLM is fully coherent in it’s “beliefs”, because by nature they parrot a variety conflicting things from their training, which is itself inconsistent, rather than reasoning their way a coherent view the way a rational person might.
- Where
- paragraph 8
- Published
- 2025-06-21
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-1984-but-with-llms-8e842e-a3: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.