“And leaving more and more code writing to generative AI, which grasps syntax but not meaning, as a new benchmark from Stephen Wolfram shows, is not the answer, either.”
Our reading, the sentence that was checked:
Generative AI grasps syntax but not meaning, as shown by a new Stephen Wolfram benchmark.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call, paragraph 5.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-dont-look-up-the-massive-microsoftcrowdstrike-d05737-2-a1
We recorded 5 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-07-19. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- generative AI
- grasps syntax but not meaning
- as a new benchmark from Stephen Wolfram shows
Our interpretation
- In this coding context, the plain reading is that contemporary generative models could produce valid program syntax while failing to reliably implement the requested behavior.
- Functional correctness is an observable indicator of handling intended meaning, but errors alone do not establish a complete absence of semantic understanding.
- The assessment concerns the empirical capability claim and its attribution to Wolfram's benchmark, rather than the recommendation that delegating more coding 'is not the answer'.
- Because the post describes a new benchmark and an ongoing capability, the target is the latest reportable month up to publication: July 1–19, 2024.
Counts as supported if: Dated results from the referenced benchmark show broadly strong syntactic validity alongside substantial failure to implement intended behavior, with methods sufficient to establish that distinction for the tested models.
Counts as contradicted if: The referenced results do not show the claimed syntax-versus-functionality distinction, or reproducible tests within the relevant scope establish dependable handling of intended behavior contrary to the operational reading.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The search located Wolfram's project, its live results and a later archived table, but did not recover a verified result set for the period before Marcus's post. That prevents a firm historical assessment of the benchmark attribution. There is also an interpretive limit: producing valid syntax while failing functional tests can establish unreliable coding performance, but does not alone establish that a model lacks all grasp of meaning. The related study in e1 does not settle either issue.
Limitations
- This is an evidence gap, not a finding that Marcus misreported the benchmark.
- Later benchmark rows were not treated as July measurements merely because they name older models.
- The operational coding interpretation is narrower than a general claim about understanding.
Evidence (1 item)
Context only: Gong and colleagues; arXiv,
pretraining methods and data quality have more impact than model size.
Why it matters here: This evaluation identifies training-related explanations for coding performance differences. It concerns a different benchmark and an earlier period, so it cannot decide either the July Wolfram results or the assertion about meaning.
- Publisher
- Gong and colleagues; arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A dated original July result set with model versions and reproducible syntax and functional tests would settle the operational benchmark claim. The broader statement about meaning would additionally require a defined, validated test of semantic competence.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call
And leaving more and more code writing to generative AI, which grasps syntax but not meaning, as a new benchmark from Stephen Wolfram shows, is not the answer, either.
- Where
- paragraph 5
- Published
- 2024-07-19
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-dont-look-up-the-massive-microsoftcrowdstrike-d05737-2-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-dont-look-up-the-massive-microsoftcrowdstrike-d05737-2-a1.