What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
We recorded 10 claims from this post. 4 were checked against evidence. 6 can’t be checked. 6 more are held. Read the post at its source.
Claims from this post (10)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“The results themselves are, without question, genuinely impressive.”
What we tested: Cicero's Diplomacy results are, without question, genuinely impressive.
Assessment: UnresolvedSaid
“As it turns out, the architecture of Cicero differs profoundly from most of what’s been talked about in recent years in AI.”
What we tested: Cicero's architecture differs profoundly from most AI approaches discussed in recent years.
Assessment: Can’t be checkedSaid
“(The contest was a “blitz” version of the game, limited to five minutes a move, which favors computers, since the blitz version is presumably less dialogue- and negotiation-intensive than the full version of the game; in a longer version of the game, humans may still dominate.)”
What we tested: The five-minute-a-move blitz version of Diplomacy favors computers, presumably because it is less dialogue- and negotiation-intensive than the full game, and in a longer version humans may still dominate.
Assessment: Can’t be checkedSaid
“Cicero is in many ways a marvel; it has achieved by far the deepest and most extensive integration of language and action in a dynamic world of any AI system built to date.”
What we tested: Cicero is in many ways a marvel and has achieved by far the deepest and most extensive integration of language and action in a dynamic world of any AI system built to date.
Assessment: SupportedSaid
“Strikingly, and in opposition to much of the Zeitgeist, Cicero relies quite heavily on hand-crafting, both in the data sets, and in the architecture; in this sense it is in many ways more reminiscent of classical “Good Old Fashioned AI” than deep learning systems that tend to be less structured, and less customized to particular problems.”
What we tested: Against much of the Zeitgeist, Cicero relies quite heavily on hand-crafting in both its data sets and its architecture, and in this sense it is in many ways more reminiscent of classical Good Old Fashioned AI than of deep learning systems, which tend to be less structured and less customized to particular problems.
What happened: The supplement specifies rules for corrupting game entities, targeted examples for counting and negation errors, and a separately configured classifier ensemble. Together with its explicit filtering and strategic-planning procedures, these document substantial task-specific engineering in both data and architecture. (Science / AAAS; supplementary PDF hosted by Gwern, )
Assessment: SupportedSaid
“It could not be immediately applied to the challenge of, say, customer service or guiding the actions of a domestic robot, or well, almost anything else, really.”
What we tested: Cicero cannot be immediately applied to tasks such as customer service or domestic robotics, or almost anything besides blitz Diplomacy.
What happened: Meta explicitly limits the released agent's capabilities to Diplomacy while describing other applications as prospects for its technology. This supports Marcus's distinction between immediate deployment and later adaptation. (Meta, )
Assessment: Can’t be checkedSaid
“Our best guess is that if you played Diplomacy with alternative rules, the system would like to have retrained almost from scratch,”
What we tested: If Diplomacy were played with alternative rules, Cicero would probably need to be retrained almost from scratch.
Assessment: Can’t be checkedSaid
“as things stand now, the prospects for generalization seem to us to be somewhat limited; the style of work might well be useful in other problems, but it may be that not much of the specifics of the architecture would survive if the system were applied to other problems”
What we tested: Saying he cannot predict this with any great confidence, Marcus judges that, as things stand now, Cicero's prospects for generalization seem somewhat limited: its style of work might well be useful in other problems, but it may be that not much of its specific architecture would survive if the system were applied to other problems.
Assessment: SupportedSaid
“Cicero makes extensive use of machine learning, but is hardly a poster child for simply making ever bigger models (so-called “scaling maximalism”), nor for the currently popular view of “end-to-end” machine learning of in which some single general learning algorithm applies across the board, with little internal structure and zero innate knowledge.”
What we tested: Cicero makes extensive use of machine learning but is hardly a poster child for scaling maximalism or for the popular end-to-end view of machine learning with little internal structure and zero innate knowledge.
What happened: The announcement describes a pretrained and fine-tuned language model combined with a specialized planning engine and message filters. This supports the factual claim that the system's design goes beyond enlarging a single general model. (Meta AI, )
Assessment: Can’t be checkedSaid
“Cicero may change that calculus.”
What we tested: Cicero may change the field's view of machine learning as a universal solvent.
Held from this post (6)
These records did not pass every check, so nothing about them is published. Held is not a finding.