LLMs are not very good at following explicit verbal instructions.

Was it true for the period described?

Outcome: Supported

The dated evidence matches what the claim committed to, under the criteria frozen before grading.

Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.

Evidence as of

2026-10-28

Deadline state when assessed

No horizon

Assessment recorded

2026-10-29

Policy version

2026-10-28.7

Rationale

The evidence supports the ordinary reliability reading. August PaLM 2 S results show frequent failures on explicit, automatically checkable constraints. Tests of June GPT-4 also found severe failures on particular instruction types. These were systematic tests, not isolated anecdotes. The qualification matters: GPT-4 followed some simple instructions well, and CELLO found substantial differences between models and tasks. The evidence supports unreliable compliance in 2023, not an inability to follow instructions generally. Calling that 'not very good' remains an interpretive judgment.

Limitations

• The conclusion depends on reading the claim as a reliability criticism.
• PaLM 2 S is a smaller model and cannot represent all frontier systems.
• Some GPT-4 failures were capitalization errors; their practical importance depends on the application.
• IFEval's November GPT-4 results are outside the target period and were not used.
• These studies do not establish a human-employee comparison or assess spoken-instruction comprehension.

Evidence (3 items)

• Supports the claim: arXiv; Jeffrey Zhou and colleagues, 2023-08

  PaLM 2 S 43.07 55.76 46.95 59.11

  Why it matters here: PaLM 2 S satisfied every checked instruction on 43.07% of prompts under strict scoring and 46.95% under relaxed scoring; individual-instruction accuracy was 55.76% and 59.11%, respectively. These August results show frequent explicit-constraint failures, but this smaller model alone cannot establish performance across all leading LLMs.

  Where

  Table 3; section 3 dates PaLM 2 S testing to August 2023

  Publisher

  arXiv; Jeffrey Zhou and colleagues

  Kind of source

  Preprint, not peer reviewed

  Describes events of

  2023-08

  Public from

  2023-11-14

• Supports the claim: arXiv; Lingjiao Chen, Matei Zaharia and James Zou, 2023-06

  Overall, we observe that GPT-4 followed less user instructions over time.

  Why it matters here: The June GPT-4 version complied with only 0.5% of the study's answer-extraction requests, with capitalization errors accounting for failures despite otherwise correct bracket placement. However, the same study found good performance on three individual text-formatting instructions in both March and June, so the evidence establishes uneven reliability rather than general inability.

  Where

  Section 4, instruction-following experiments and Figures 13–14

  Publisher

  arXiv; Lingjiao Chen, Matei Zaharia and James Zou

  Kind of source

  Preprint, not peer reviewed

  Describes events of

  2023-06

  Public from

  2023-10-31

• Context only: arXiv; Qianyu He and colleagues, 2023-09

  Different models also excel in specific tasks.

  Why it matters here: CELLO identifies task-specific strengths and a substantial advantage for GPT-4 and GPT-3.5 over smaller open models. This limits a sweeping interpretation of Marcus's statement; the benchmark uses Chinese complex instructions and partial-credit measures, not a representative everyday failure rate.

  Where

  Experiment, Fine-grained Evaluation; General Comparisons

  Publisher

  arXiv; Qianyu He and colleagues

  Kind of source

  Preprint, not peer reviewed

  Describes events of

  2023-09

  Public from

  2023-09-17