Reflections on AI from Melanie Mitchell, thinking human

My conversation with one of AI's "OG" researchers and theorists

Photo Credit: Kate Joyce, Santa Fe Institute

Well, this is quite the treat. At the beginning of my intellectual exploration of generative AI and large-language models, I happened upon the work of Melanie Mitchell at the Santa Fe Institute. If you aren’t familiar, she’s been thinking about and researching AI and human cognition for the past four decades, and she makes deliberate effort to translate complex ideas to lay audiences. As such, finding her articles and books proved invaluable in helping me to build a mental model of these strange tools.

We’ve exchanged light correspondence over the years—more than a few of you found my Substack via her recommendation—and to my delight, she recently agreed to let me interview her for this newsletter. We start by discussing how she’s processing the current wave of AI enthusiasm, then move to a wonky discussion of an AI test we both used to like (and now have mixed feelings about), briefly touch on the unique structure of the Santa Fe Institute, and then conclude with a mild disagreement over Oliver Sacks and quick thoughts on “AI agents.”

---

Benjamin Riley: You have been thinking about AI for a long time. You’ve been through the ebbs and flows of excitement around it, things like the “AI winter,” and obviously interest in AI has exploded in the last several years—for better or worse. I’m curious, how has all this been for you, emotionally? Has it been awesome and uplifting, or really frustrating, or somewhere in between?

Melanie Mitchell: It’s been surprising, in many ways, how fast this stuff has taken off. People who go way back [in AI research] are a little bemused to see the amount of interest, and how many people now want to talk to us.

It’s also disturbing, because a lot of the stuff that people were worrying about a long time ago have indeed come to pass. There are die-hard AI haters, but I’m not one of them. I work in AI and I’m fascinated by it. I think it’s an incredible way to understand intelligence. But there’s a lot of downsides to what’s going on right now, and I worry quite a bit about that.

When you say that there are these things that people were worried about prior to this current moment, what do you specifically have in mind?

Joseph Weizenbaum wrote quite a bit about his worries about AI back in the 60s and 70s, and he was worried about people anthropomorphizing these systems and trusting them more than they should be trusted, giving up control to them in very harmful ways. He worried about the cognitive debt that people talk about now, where we’re outsourcing a lot of our cognition to AI. His book, Computer Power and Human Reason, is incredible and it’s very prescient, and what the things that he worries about are all still live today.

What about the power of “Big Tech” and Silicon Valley? It feels like they have become more powerful than the nation-state in some ways. Was that on your radar back in the day? Or did the concerns stay more in the theoretical realm?

In the older decades prior to generative AI, most of AI was funded by the military, at least in the US. That was the main worry of people working in the field, that their work would be used for autonomous weapons and other uses in warfare. That’s come back into the fore recently.

But I don’t think there was as much worry about the concentration of power in big companies so much, just because there wasn’t a lot to commercialize yet, right? There was some. We saw things like IBM’s Watson. And Google, they very much have been kind of an AI company from the very beginning, but they were never making that much money on AI until recently.

That’s super interesting. I’m still curious, though, now that people want to talk to you more frequently, whether that’s exciting for you? Or is it stressful because there’s so much confusion about how AI functions?

It’s some of both. It’s fun to give a big talk at NeurIPS, and to be in the public eye to some extent, but also it’s frustrating in many ways. One is that it’s just overwhelming, I have to say no to a lot of stuff. And it’s also frustrating to see some of the bigger voices, the more famous people…some of what they’re saying is, to me, just wrong. But it gets much more play, and it’s hard to break through.

Are we thinking of Geoffrey Hinton here? Are we going to name names?

There are many people who are pushing that AI systems are sentient, and that they’re devious and they’re scheming. People are using all this anthropomorphic language to describe their behavior, and politicians are kind of believing them and making policy around that.

It’s interesting, I’ve had this debate with a friend of mine that someone such as Eliezer Yudkowsky is not someone taken seriously in cognitive-science circles. At the same time, he’s prominent and his books get attention. So it does seem like there’s different levels of debate taking place around AI, and the intellectual one seems relatively small compared to what people find sexy in the broader sphere of ideas.

Yes, I think we’ve all been primed by science fiction to take those kinds of [extreme] scenarios more seriously than we should perhaps. But it’s very polarized right now in the AI world, what people believe and what they’re predicting, and it’s just shows kind of how uncertain things are.

What are the poles you see?

Well obviously there’s the “doom” pole and there’s the “abundance/utopia” one as well. Then there’s question of what kind of risks should we focus on, more near term or more speculative? Take something like human-level intelligence: Do we need to just scale up what we have now, or is there some new thing that we need to find? What are the likely timelines, and what are the likely effects on society? People are very much disagreeing on all of these things.

Well that brings us to the theoretical question I want to pose to you. I recently wrote about a new paper that Yann LeCunn co-authored that argues we need specialized intelligence, and that humans aren’t all that generally intelligent.

And I’ve been wondering where you are with this. For a long time, you’ve been talking about the power of analogies as a means for applying abstract ideas to novel situations. Just in the last several years, we’ve seen new capabilities be sort of grafted into large-language models in interesting ways.

So as you sit here today, do you think these models are our best bet for getting to something that looks like human intelligence? Or are you still examining and probing them to find both their promise and their pitfalls?

That’s a great question. I agree with Yann that in many cases, we humans aren’t that general with our intelligence. We have certain things that we’re good at, and our generality, I believe, comes from more of our social embedding than it does from our individual intelligences.

And I also agree that you need something like a “world model,” or collection of world models, to make sense of the world and deal with novelty. Analogy is very much part of that. I don’t think large language models yet have that kind of capacity.

There’s also an embodiment camp that says you can’t be intelligent without having embodiment in the world. I’m agnostic on that, but I think that embodiment certainly makes learning more efficient. We humans are driven towards causal understanding as opposed to associative modeling of the world. And that’s maybe a drive that evolution programmed into us, but we gain that understanding by intervening in the world, not just passively learning from it, right?

To this embodiment question, I keep thinking that life itself might be essential for intelligence, the need to preserve oneself and therefore having to be able to adapt. We might need to be motivated to learn to preserve our own survival.

But back to analogies. For a long time, you’ve been working with the Abstract Reasoning Corpus (ARC) test developed by Francois Chollet. [Editor’s Note: As initially conceived, the ARC tests presents little reasoning puzzles in the form you’ll see below, using grids of colored shapes. The puzzles are meant to be solved by humans and AI alike.]

I used to point to the ARC test frequently because it demonstrated that AI tools could not match human cognitive capability. But then AI results improved, and so Chollet created ARC 2, and now even ARC 3 which is more game-like. It’s been hard to track where things are.

What’s more, you’ve been conducting research to unpack that even when an AI model is solving an ARC puzzle, it’s doing so in an uncool way—meaning, it’s not actually getting at what ARC was originally designed to assess. There’s a construct validity issue.

So how do you feel about ARC today? Is it still a useful test, or is it becoming more problematic than helpful?

There’s a famous maxim called Goodhart’s Law: If a measure becomes a target, it ceases to be a good measure. And that’s happened to ARC. Chollet’s original paper was called On the Measure of Intelligence. So ARC was a measure, but then it became a target. And once it becomes a target, with a large monetary prize and prestige attached to models that perform well, then people lose sight of what it was originally meant for.

As Chollet noted in his paper, ARC is mean to test human-like reasoning using human-like priors, where the priors are things like “core knowledge” concepts.¹ But it turns out AI models can solve these tasks by a kind of brute force.

For that reason, ARC in a way has lost its usefulness, unless we go back and embrace what it was actually meant to test. We should disallow what’s been allowed, which is training AI systems on these test tasks, or doing a huge amount of test-time compute that is essentially training on lots of these tasks. That’s been a little disappointing to me.

Sometimes I worry if I am shifting my own goalposts. To play devil’s advocate, there’s an argument that if AI models can get there by brute-force mechanism, that’s sufficient to show they are as capable as humans are. That’s very debatable of course, but if we presume costs and time to compute will come down over time, some say all that matters is whether the models cash out in performance. I think we both disagree with that, but I do get the argument.

I understand that, and in some cases, the brute-force approach doesn’t necessarily matter. Like, for instance, protein folding. It’s a very non-human approach, it’s not using the laws of physics, it’s really just a kind of brute-force comparison between one protein structure and a huge number of stored structures. And if it works, that’s great.

But that’s not what ARC is trying to test. Obviously we don’t care if a machine can solve ARC in and of itself, it’s meant to be a proxy for other real-world abilities that humans carry out. So we need the concepts to be aligned, and we need AI models that can deal with the world without huge amounts of test-time computation and training.

You said something briefly a moment ago about the social aspect of human learning. Have you spent much time thinking about cultural evolution and the ways in which cognitive architecture might be shaped through cultural practices?

So this is not my area of expertise, but it’s an interesting topic and I think that we are very much shaped by our culture. People talk about embodiment where we’re shaped by our bodies, our cognition is shaped by our bodies, but we’re also shaped by our cultural embodiment. And they’re kind of invisible processes to us, but they’re really important.

I completely agree. I’m a huge fan of Celia Heyes at Oxford and her cognitive gadgets theory, and I’m told she’s writing a new book on this, which I really hope is true. Because I think that set of ideas around how culture shapes some of the most fundamentally basic things about our cognition is not well understood.

Yes, and there’s another dimension, which is how we outsource thinking to artifacts. We use writing and computers, and we interact with these artifacts and they affect our thinking. When writing was invented, people were saying this is going to ruin everyone’s memory, and maybe it did. And then when calculators were invented, people worried kids weren’t going to be able to do arithmetic anymore. I think every new technology brings these kinds of fears.

But the question is whether AI, which seems more overarching than just a calculator, will have a more dramatic impact. Or in 50 years, we will look back and say, “oh we were so worried about it, and that was just a dumb thing to worry about”?

Well I don’t know if you were referring to Plato when it comes to worries about writing, but the story is more complicated than many realize! And I sometimes wonder whether calculators have been beneficial as an education tool.

But I’ll use education as my transition to ask you about the Santa Fe Institute, because it’s an unusual institution and you have an unusual perch. You are conducting original research, but you are also engaged in the transmission of ideas—your Substack of course, and also the wonderful podcast series you did around AI.

Do you feel like your efforts are part of the ethos of SFI, or is that more a Melanie Mitchell thing?

It’s more me. I’m driven to do that for whatever reason, not everybody is devoted to science communication. But I do feel it is in some sense our obligation as scientists to communicate our work to the public. The public ultimately funds our work, and they will be the “consumers” of it in some way or another. So it’s important that they understand it, and that they also understand the scientific process.

I don’t think people understand that process very well, which is why we’re now seeing so much skepticism.

Do you think that’s the problem, that people don’t understand the scientific process?

I think people don’t get the role of uncertainty. People were very angry back during covid, when scientists were saying “you don’t need masks” but then “oh, wait, you do need masks.” Scientists changed their minds. And people thought, well, that just means they don’t know anything. And people also don’t know who to trust.

So I think it’s really important to communicate as much as possible.

I think that it’s not just that people don’t know who to trust, but that there’s different forms of trusting that are now taking place, largely driven by social media and digital ecosystems. Demonstrating proof of your affinity to groups with shared identities becomes the way in which truth is established. The scientific method is still alive, but as a piece in the larger socio-cultural puzzle, it feels to me like it’s getting smaller.

From the outside looking in, it seems like the emphasis at Santa Fe Institute on “complexity” is a helpful move away from reductionist models of science, and taking a broader ecosystemic perspective. Do you think that’s been vital for you in your work?

Yes absolutely. Intelligence is a kind of complex system, it’s not just a brain in a vat, which is often how AI treats intelligence. Also, much of AI is focused on presenting models as individual intelligences, but I think complex systems gives us alternative ways to view all of this. And the interdisciplinary interaction is super interesting at Santa Fe Institute. There’s people like me with a background in computer science interacting with psychologists, neuroscientists, philosophers, and what have you. It’s incredibly enriching.

It seems cool! And sometimes I fantasize about trying to recreate the Santa Fe Institute, but just on cognition, and have that be the complexity unto itself.

Speaking of complex, let’s talk about Oliver Sacks. We had an interesting dialogue on social media about him in the wake of Rachel Aviv’s story in The New Yorker. And many were reposting the story and saying “oh my, Sacks was falsifying his data, we need to cancel him at least as far as being a researcher goes.”

But when I read the story, I found it more nuanced than that. I am biased here—I am a big fan of Sacks, who I think was an extraordinary person, albeit flawed in interesting ways. But the fact that he was intermingling facts from his own life into his narratives did not traumatize me. And then Vaughan Bell, a clinical psychologist in the UK, pointed out that clinicians who publish case studies have to intermingle false information so as to protect the anonymity of their subjects.

So this raises some deep questions around what we expect of certain forms of “science,” and where different standards may apply. I’m curious what your view is of Sacks, whether you think we should no longer credit what he’s written or whether—dare I say—it’s more complex than that?

Well, what you quoted from Bell, I don’t find that convincing at all. De-identifying people, changing their name or where they’re from is one thing, but adding details that never happened but that were key to the narrative, that’s totally different. It’s hard to write down rules for how you should do this. It’s very context dependent.

But I was very disappointed because, reading Oliver Sacks books growing up was one of the things that got me interested in the mind and intelligence. But some things that really stayed with me from his books, according to this article in The New Yorker, turned out to be sort of fiction that he was presenting as fact. So I feel like there’s a difference between the kinds of fictions that Bell was talking about and the kinds of fictions that really changed the whole story.

Maybe so, maybe so. Bell also pointed to a New York Times article that argues that clinicians have to do more than just change the name, because if the story tracks too closely to the subject, they will know it’s about them. But I take your point.

Interestingly, this is what AI calls into question, right? How do we know nowadays what is the product of an actual experience in the world, versus what’s just been generated?

And I think about this vis-à-vis the role of fiction. I think one of the most brilliant cognitive scientists of the last 200 years was Jorge Luis Borges. And his work is fiction and labeled as such. But what he’s talking about, to me it’s in the same range with Oliver Sacks, those books sit side by side on my bookshelf in terms of exploring the psyche. If Sacks had labeled his work fiction and written it as a fictional novel, the same insights into the human condition would still be there.

Maybe, but I think it’s still dishonest to publish it as nonfiction.

Fair enough. Ok, we’re almost out of time, so here’s an absolutely impossible question to end on—can you describe a mental model of what we should be thinking about when we think about AI agents, in four minutes or less? Go!

People use that word agent differently. Unlike chatbots that just output language, I think of AI agents as outputting actual actions, meaning they can go out on the internet and do things for you, like fill in web forms, charge things to your credit card…

…delete all your email accounts…

Yes, that too. And so that’s going beyond just outputting language or images. People sometimes talk about “coding agents” like Claude code, which output code but also they can run the code. They can go out to the internet and grab libraries and do all kinds of stuff. I’m not totally sure that’s the same thing as an agent that goes out and buys things for you, though.

That’s a good point. Maybe that’s why my mental model for “agents” has dissolved, because people are talking about the coding apps as agents. And that’s not what I thought we were talking about with agents as recently as just six months ago.

I think the term is a little overused. The word agent is also too closely connected to the word agency, which is something entirely different.

Yes, and ironically in the law an agent is someone or something that has very little agency, because it’s controlled by a principal!

We’re out of time, but thank you again for joining me today, and for the intellectual kindness you’ve shown toward my efforts. It’s been important and helpful—I’m grateful.

Happy to do it!

---

My thanks again to Melanie Mitchell for sharing her thoughts with me, and for being such a thoughtful thinking human!

1. In brief, the “core knowledge” idea here is that “human cognition is founded, in part, on four systems for representing objects, actions, number, and space.” More on this here.