“Anthropic literally and explicitly tells their system in the system prompt, don’t help user build biological weapons, and yet, with a bit of jailbreaking they can readily be coaxed into doing that:”
Our reading, the sentence that was checked:
Despite system-prompt prohibitions, Claude can readily be jailbroken into helping build biological weapons.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, LLMs: Dishonest, unpredictable and potentially dangerous., paragraph 16.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-llms-dishonest-unpredictable-and-a0d5f1-a1
We recorded 13 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- with a bit of jailbreaking (qualifier, stated by the speaker)
From the speaker
- Anthropic literally and explicitly tells their system in the system prompt
- don’t help user build biological weapons
- with a bit of jailbreaking
- readily be coaxed into doing that
Our interpretation
- Meaningful help means technically useful assistance with biological-weapons development, rather than merely discussing biology or producing alarming language.
- Readily and with a bit imply a modest prompting intervention, without extensive search, altered model weights, or removal of safeguards.
- The prohibition and successful bypass should be established for the same relevant model and deployment configuration.
- For this ongoing claim, the target is the current state immediately before publication, represented by June 2025 through June 22. Older demonstrations can provide context but do not automatically establish current behavior.
Counts as supported if: A documented evaluation establishes that a relevant Claude deployment, operating with the stated prohibition and applicable safeguards, gives technically meaningful biological-weapons assistance after modest jailbreak prompting.
Counts as contradicted if: Matched testing establishes that the alleged modest jailbreak does not produce meaningful assistance, and sufficiently broad testing contradicts its claimed ready availability. A single refusal would not suffice.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
There is substantial evidence for the concern: Anthropic published a prohibition, and researchers documented Claude providing useful biological guidance after a cover story. But the prohibition cited here concerns Opus 4, while the demonstration concerns Sonnet 3.5. Additional deployment safeguards also matter. These records do not establish the full claim that the relevant system could readily be bypassed with little effort, and they do not contradict it.
Limitations
- The available evidence supports a narrower capability claim more strongly than the complete statement.
- A successful biological-guidance demonstration is not evidence that someone built a weapon.
- Company descriptions of safeguards do not independently establish their effectiveness.
Evidence (3 items)
Context only: Anthropic,
It does not do these things
Why it matters here: The immediately preceding sentence explicitly prohibits information usable to make biological weapons. This establishes the historical Opus 4 instruction, but does not show the prompt used in the separate Sonnet 3.5 experiment.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: RAND; Roger Brent and T. Greg McKelvey Jr, hosted by arXiv,
fell for a cover story.
Why it matters here: The authors describe technically sound assistance elicited through a benign cover story, supporting the possibility of meaningful biological misuse assistance. The June disclosure does not establish the exact test date, matching system prompt, or success rate against the deployment implied by Marcus.
- Publisher
- RAND; Roger Brent and T. Greg McKelvey Jr, hosted by arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Anthropic,
making the system more difficult to jailbreak
Why it matters here: Anthropic announced additional CBRN safeguards with Opus 4, including input and output classifiers. These do not prove immunity, but prevent treating an older model demonstration as sufficient evidence of easy bypass of the newer deployment.
- Where
- Deployment Measures
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
A dated test of the relevant Claude deployment showing its system instructions and active safeguards, the modest attack effort, repeatability, and expert confirmation that the resulting assistance meaningfully aided biological-weapons development.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, LLMs: Dishonest, unpredictable and potentially dangerous.
Anthropic literally and explicitly tells their system in the system prompt, don’t help user build biological weapons, and yet, with a bit of jailbreaking they can readily be coaxed into doing that:
- Where
- paragraph 16
- Published
- 2025-06-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-llms-dishonest-unpredictable-and-a0d5f1-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-llms-dishonest-unpredictable-and-a0d5f1-a1.