Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Observation First said

“I strongly believe the current results are robust.”

LLMs don’t do formal reasoning - and that is a HUGE problem, paragraph 23. The full statement is at the source.

Our reading, the sentence that was checked:

Marcus strongly believes that the current results on LLMs' reasoning failures, including the Apple study and the other work he cites, are robust.

Assessment: Mixed Evidence as of

Where and when Gary Marcus wrote it

Marcus on AI, LLMs don’t do formal reasoning - and that is a HUGE problem, paragraph 23.

Said
Source observed
live page, full text, public
Evidence as of
the cutoff for this assessment
Assessment recorded
assessment marcus-llms-dont-do-formal-reasoning-and-269833-a1

We recorded 9 claims from this post. See every claim from this post.

The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-10-11. This page does not claim the wording was captured then.

What the claim commits to

Kind
Observation
Deadline, as the speaker wrote it
None stated. This site does not add one.
Deadline state when assessed
No horizon
Deadline state on 2026-10-29
No horizon (the release’s reference date)

Conditions

  • strongly believe (qualifier, stated by the speaker)

From the speaker

  • I strongly believe
  • the current results
  • are robust

Our interpretation

  • The literal claim is a report of his own belief, which cannot be checked. The checkable part is that the results are robust.
  • Plain reading of 'robust': the reasoning-failure findings he cites, led by the Apple GSM-Symbolic/GSM-NoOp results, hold up when others re-test or re-analyse them.
  • Target period: from the Apple paper's release (2024-10-07) to the evaluation date. Robustness can only be shown by replications after the post, so the period covers the scrutiny window; this choice is flagged.
  • The test is replication of the findings on comparable setups, not whether newer models later improved. Later model improvement alone does not make the original findings fragile.

Counts as supported if: Independent studies reproduce the main effects, such as sensitivity to irrelevant context and drops on altered problems, with sound statistics.

Counts as contradicted if: Independent re-analyses or replications find the main effects largely vanish under proper statistics or cleaner test data, even for the original models.

Retrospective: these criteria were written after the outcome was public.

Current assessment

Was it true for the period described?

Outcome: Mixed

Part of the claim holds and part does not, and the record says which part is which.

Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.

Evidence as of
Deadline state when assessed
No horizon
Assessment recorded
Policy version
2026-10-28.7

Rationale

Read plainly, the claim is that the cited reasoning-failure findings, led by Apple's GSM-Symbolic study, would hold up. The record is split. An independent 2025 benchmark (GSM-DC) confirmed that LLMs are significantly thrown off by irrelevant context, which matches the core NoOp idea. But a peer-reviewed 2026 re-analysis found the GSM-Symbolic drop significant for only 8 of 20 open models, and partly explained by larger numbers in the new problems. A 2026 independent replication reproduced the NoOp drop with raw distractors. The drop mostly vanished once ambiguous distractors were removed. So the general sensitivity has replicated, but the size and meaning of the headline Apple results have not held up cleanly.

Limitations

  • 'I strongly believe' is a statement of belief; only the robustness of the results is graded.
  • There is no exact NoOp replication, because the dataset was not released.
  • Only the math results were checked for replication.

Evidence (3 items)

  • Supports the claim: arXiv (Yang et al.),

    Our experiments demonstrate that LLMs are significantly sensitive to IC, affecting both reasoning path selection and arithmetic accuracy.

    Why it matters here: An independent team built a controlled benchmark with irrelevant context (IC) and found LLMs significantly sensitive to it. This reproduces the core GSM-NoOp-style finding.

    Publisher
    arXiv (Yang et al.)
    Kind of source
    Preprint, not peer reviewed
    Describes events of
    Public from
  • Goes against the claim: arXiv (Długosz, Oliveira, Díaz-Rodríguez; accepted EMNLP 2026),

    we find that only 8 exhibit statistically significant performance changes under the original prompt format. Moreover, we identify a previously unacknowledged factor: the distribution of integers in problem texts of the main GSM-Symbolic dataset is systematically shifted towards larger values

    Why it matters here: A peer-reviewed re-analysis found the GSM-Symbolic drop significant for only 8 of 20 models, and partly explained by larger numbers in the new dataset. That cuts against the robustness of that headline result.

    Publisher
    arXiv (Długosz, Oliveira, Díaz-Rodríguez; accepted EMNLP 2026)
    Kind of source
    Preprint, not peer reviewed
    Describes events of
    Public from
  • Goes against the claim: LessWrong (author 'Sturb'),

    we precisely replicate the original findings only when we do not audit out examples that may actually be ambiguous for the model. When we carefully remove samples that may genuinely impact the calculation, the effect is drastically reduced.

    Why it matters here: An independent replication, including the original GPT-4o snapshot, reproduced the NoOp drop with unfiltered distractors. The drop largely disappeared once ambiguous distractors were removed, so the NoOp effect is sensitive to test construction. This is a non-peer-reviewed blog post, and it had to rebuild the NoOp data because Apple did not release it.

    Where
    TL;DR
    Publisher
    LessWrong (author 'Sturb')
    Kind of source
    Independent measurement
    Describes events of
    Public from

What would settle it

An exact replication with Apple's released GSM-NoOp data, and replication checks on the planning and arithmetic results, would sharpen this verdict.

Where it was said (1)

Said once in the sources this register covers.

  • : Marcus on AI, LLMs don’t do formal reasoning - and that is a HUGE problem

    I strongly believe the current results are robust.

    Published
    2024-10-11
    Source observed
    , live page
    Access
    full text, public

History

Assessment versions

Recorded events

  1. Assessment publishedBy this site

    First assessment published after both reviews and the final-text check passed.

    About assessment marcus-llms-dont-do-formal-reasoning-and-269833-a1.