“First, there’s no way to anticipate how many tokens a prompt will actually burn, which makes any kind of budgeting a non-starter.”
Our reading, the sentence that was checked:
There is no way to anticipate how many tokens a prompt will actually burn, which makes any kind of budgeting for metered usage a non-starter.
Assessment: Contradicted Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, Why Are We Still Doing This?, paragraph 41.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-why-are-we-still-doing-this-d9599d-a1
We recorded 33 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-03-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- no way to anticipate
- how many tokens a prompt will actually burn
- any kind of budgeting
- a non-starter
Our interpretation
- The plain reading is that uncertainty about total token consumption prevents practical budgeting for metered LLM use.
- Total consumption includes generated reasoning and repeated calls, rather than only the initial prompt's input tokens.
- Practical budgeting can use estimates, spending limits, and a chosen trade-off between cost and completed work. It does not require knowing every request's exact final token count.
- The relevant capabilities must have existed by the March 17, 2026 post. Later improvements cannot retrospectively contradict this claim.
- This is assessed as a causal assertion, so no reporting-period target is imposed.
Counts as supported if: Evidence shows that token uncertainty prevents workable advance estimates or spending controls for useful metered LLM activity, even with available budgeting methods.
Counts as contradicted if: Pre-post documentation and measured results demonstrate that users can estimate or constrain metered consumption sufficiently to carry out useful work within planned budgets, despite imperfect per-request prediction.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Contradicted
The dated evidence goes against what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The categorical conclusion that token uncertainty makes any budgeting impossible is contradicted. Before the post, Anthropic documented monthly spending limits and controllable thinking budgets. BAMAS also tested advance cost estimates and reported useful work within budgets, including no overruns on its GSM8K tasks. The narrower concern remains valid: exact task costs can be uncertain, and the same paper reports overruns on other datasets. Budgeting is possible without guaranteeing either an exact token count or successful completion at a fixed cost.
Limitations
- The result depends on reading budgeting in its ordinary practical sense, rather than as an exact price guarantee.
- Spending limits can stop work before completion.
- The research covers selected benchmark tasks and reports some budget overruns.
- Provider documentation establishes available controls, not their usability for every consumer.
Evidence (3 items)
Goes against the claim: Anthropic,
Implement monthly spend limits
Why it matters here: Anthropic documented a way to constrain monthly API expenditure before the post. This contradicts the claim that every form of budgeting is unavailable, but does not guarantee that a task finishes before its budget is exhausted.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Goes against the claim: Anthropic,
“thinking budget”
Why it matters here: A pre-post model exposed a developer-controlled reasoning budget. This provides control over one source of variable consumption, rather than exact prediction of an entire multi-call task.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Goes against the claim: arXiv; Liming Yang and coauthors,
BAMAS incurs no OOB tasks.
Why it matters here: The study uses advance token-cost estimates and reports useful task performance with no budget overruns on this dataset. It also reports overruns on MBPP and MATH, supporting the narrower concern about uncertainty while contradicting the impossibility of practical budgeting.
- Publisher
- arXiv; Liming Yang and coauthors
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, Why Are We Still Doing This?
First, there’s no way to anticipate how many tokens a prompt will actually burn, which makes any kind of budgeting a non-starter.
- Where
- paragraph 41
- Published
- 2026-03-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-why-are-we-still-doing-this-d9599d-a1: Current. Outcome: Contradicted. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment wyea-why-are-we-still-doing-this-d9599d-a1.