Misplaced panic over AI progress
We recorded 14 claims from this post. 4 were checked against evidence. 10 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (14)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“Claude Code is a real advance; Mythos probably builds on some of what is learned there.”
What we tested: Claude Code is a real advance, and Mythos probably builds on some of what was learned from it.
Assessment: SupportedSaid
“There is plenty of headroom left on the current METR set of tasks if you simply demand 80% success, even more headroom if you demand 95% success.”
What we tested: There is plenty of headroom left on the current METR task set if you demand 80% success, and even more headroom if you demand 95% success.
What happened: The linked official data give an 80% horizon of about 3.10 hours, well below the page's 16-hour upper reliable measurement range. Applying the documented fitted curve implies a still shorter 95% horizon; that implication is an evaluator calculation, not a published validated 95% measurement. (METR, )
Assessment: Can’t be checkedSaid
“For exactly that reason, Wildeford’s concern about Mythos “breaking the graph” is a red herring.”
What we tested: Wildeford's concern that Mythos has 'broken' the METR time-horizon graph is a red herring, because the graph only measures 50% success.
Assessment: Can’t be checkedSaid
“More broadly, the key problem with GenAI has been reliability; a graph that demands only 50% success does not address reliable performance. At all.”
What we tested: Reliability has been the key problem with GenAI, and the METR graph at 50% success does not address reliable performance at all.
Assessment: MixedSaid
“The graph pertains only to software-development tasks. Not general intelligence.”
What we tested: The METR time-horizon graph pertains only to software-development tasks, not general intelligence.
What held: METR explains that its task distribution omits important features of ordinary work. Together with the documented domain restrictions, this supports the narrower point that the graph is not a direct demonstration of general intelligence. (METR, )
What did not: METR describes the TH1.1 graph's distribution as spanning three technical domains. That contradicts the exclusive wording under the ordinary, narrower meaning of software development. (METR, )
Assessment: Can’t be checkedSaid
“(Incidentally, this is yet another vindication of neurosymbolic AI – not proof that LLMs themselves can be perpetually scaled. Nor a proof that another trillion dollars will indefinitely continue the trends shown in the graph.)”
What we tested: Recent progress vindicates neurosymbolic AI and does not prove LLMs can be perpetually scaled or that another trillion dollars will continue the trend indefinitely.
Assessment: Can’t be checkedSaid
“As noted in my newsletter, a lot of the advance in recent months is likely from the incorporation of symbolic tools (like code interpreters, verification, and harnesses) rather than from model scaling per se.”
What we tested: Much of the recent-months AI advance is likely due to symbolic tools like code interpreters, verification and harnesses rather than model scaling itself.
Assessment: Can’t be checkedSaid
“Instead, its techniques likely work best with things like coding and math, where formal verification (good old symbolic AI for the win!) can straightforwardly apply.”
What we tested: Mythos's techniques likely work best with things like coding and math, where formal verification can straightforwardly apply.
Assessment: Can’t be checkedSaid
“To anticipate that Anthropic will have $2T revenue in 2030 is a perfect example of what I have often called the trillion pound baby fallacy”
What we tested: Anticipating that Anthropic will have $2T revenue in 2030 is a perfect example of the trillion pound baby fallacy.
Assessment: UnresolvedSaid
“Babies don’t keep doubling forever, and nor will AI progress.”
What we tested: AI progress will not keep doubling indefinitely.
Assessment: Can’t be checkedSaid
“some types of challanges (e.g., reasoning accurately with respect to world models, reducing hallucinations etc) may simply not be amenable to current approaches”
What we tested: AI progress may hit limits from resource constraints, benchmark-focused tooling, formal verification limits in less formal problems, and challenges like world-model reasoning and hallucinations not amenable to current approaches.
Assessment: UnresolvedSaid
“We can be absolutely sure that the task length “time horizon” for AI is not going to keep doubling until “time horizons will be 580 times the age of the universe” as Lisan al-Ghaib joked.”
What we tested: We can be absolutely sure that the AI task-length 'time horizon' will not keep doubling until time horizons reach 580 times the age of the universe.
Assessment: Can’t be checkedSaid
“And most importantly, solving (some aspects of) software design is not open-ended intelligence. AI is definitely getting better at some things, but there is no reason to think that it is close to fully general yet.”
What we tested: AI is improving at some things but there is no reason to think it is close to fully general intelligence; software design progress is not open-ended intelligence.
Assessment: Can’t be checkedSaid
“In short, there is no need (yet?) to panic.”
What we tested: In short, there is no need (yet?) to panic over the AI progress shown by the METR graph.
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: Mythos will likely score under 20% (perhaps under 10%) on the Remote Labor Index, with no meaningful improvement on physical jobs.
- The two reviews read the claim differently