Doug Lenat, 1950-2023
We recorded 7 claims from this post. 1 was checked against evidence. 6 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (7)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
“My own view is that Cyc has been neither a success nor a failure, but somewhere in between: I see it as a ground-breaking, clarion experiment that never fully gelled.”
What we tested: In Marcus's own view, Cyc has been neither a success nor a failure but somewhere in between: a ground-breaking, clarion experiment that never fully gelled.
Assessment: Can’t be checkedSaid
“No, Cyc didn’t set the world on fire, but yes, it will seem more and more important in hindsight, as we eventually make real progress towards artificial general intelligence.”
What we tested: Cyc will seem more and more important in hindsight as real progress toward artificial general intelligence is eventually made.
Assessment: Can’t be checkedSaid
“But every single one of them should know something about Cyc.”
What we tested: Every young AI researcher should know something about Cyc.
Assessment: Can’t be checkedSaid
“Not because Cyc will get used out of the box, as some sort of drop-in replacement for Large Language Models, but because what Lenat tried to do - to get machines to represent and reason about common sense — still must be done.”
What we tested: Getting machines to represent and reason about common sense still has to be done, though Cyc itself will not serve as a drop-in replacement for LLMs.
Assessment: Can’t be checkedSaid
“He didn’t fully succeed – we will need a different path – but he picked the critical mountain that we still must cross.”
What we tested: Capturing common sense is the critical problem AI still has to solve, but it will take a different path from Cyc's.
Assessment: SupportedSaid
“And even if this specific example is patched up, there will inevitably be others of a similar flavor.”
What we tested: Even if this specific LLM error is patched, other errors of a similar flavor will inevitably appear.
What happened: Google identifies AI Overviews as powered by a customized language model and acknowledges advice about putting glue on pizza, alongside mishandling a question about eating rocks. These are subsequent commonsense failures matching the surrounding description of plausible answers containing nonsense. (Google, )
Assessment: Can’t be checkedSaid
“The biggest problem with Cyc from an academic perspective is that it’s proprietary.”
What we tested: From an academic perspective, Cyc's biggest problem is that it is proprietary.
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: From LLMs we sometimes see an illusion of common sense, depending on the vagaries of the training set and the precise wording of a question, but something is obviously still missing.
- The record does not meet the current rules. Our reading, not assessed: Large language models repeatedly struggle with common sense.
- The record does not meet the current rules. Our reading, not assessed: Depending on the wording, any given LLM might or might not answer questions like the example correctly on any given day, and LLMs tend to be accurate with some wordings and inaccurate with others.
- The two reviews read the claim differently