In the span of a single 24-hour period, the International Space Station will orbit our planet 16 times. Imagine being one of six astronauts aboard—or rather, imagine being all six, whirling around and around and around with both time and space distorted in ways all but unimaginable to our lived experiences on Earth.

This is the premise of Samantha Harvey’s Orbital. It is a remarkable book. There are no chapters per se; time is divided by orbits. It’s not the sort of novel that can be “spoiled” really, but if you don’t want to know anything more about what unfolds, stop here.

How are we writing the future of humanity?

This question pops up in orbit 10, via an email that’s been sent to the American astronaut Shaun from a magazine seeking his comment on a separate lunar mission that’s just underway. Shaun thinks he’s the wrong person to ask, partially because life in space mostly involves the immediate present; when oxygen is precious, even breathing requires deliberation. He thinks too that to become an astronaut requires more brazen selfishness than self reflection, “the propulsion of a self determination that burns everything in its path.”

America will turn 250 this week, at least by one measure. Hard to feel celebratory. A country presently consumed by its own brazen selfishness, burning everything in its path.

How are we writing the future of humanity?

Unsure of how to respond to the question, Shaun shares it with one of his fellow space travelers, the Italian astronaut Pietro.

The future of humanity?, Pietro repeats back.

Yep, how are we writing it?

With the gilded pens of billionaires, I guess.

How are we writing the future of humanity?

Someone told me once that Jeff Bezos wrote about going to space when he was in high school. I don’t know if it’s true. I do know that, when William Shatner went into space at age 90 on one of Bezos’s private space flights, he (Shatner) wrote poignantly about the profound grief arising from the experience:

Here is a video of Shatner trying to explain some of these feelings to Bezos shortly after they’d returned to Earth. Bezos nods absently before interrupting to ask his wife to bring them a bottle of champagne.

How are we writing the future of humanity?

Nowadays, I’m unable to be away from my phone for sustained durations of time, even when reading. This bothers me of course. But ironically, because each orbit in Orbital is only a few pages, it’s well suited for mental fidgeting; in between orbits I’ve been hopping over to BlueSky to get my algorithmic fixes.

This morning, between orbit 9 and 10, I read this story by James Ball, a journalist in England, about the sad last gasps of Zuckerberg’s dying Metaverse, a different form of empty space. We can all delight in its death, I hope. Virtual reality is one future that various techno-enthusiasts have tried to write for 70 years, maybe longer, only to fail again and again and again. An unwanted space. Gilded pens the billionaires may have, but their ideas still suck. Ball observes:

How are we writing the future of humanity?

There is no time, Lou Reed tells us:

This is no time for celebration/This is no time for saluting flags/This is no time for inner searchings/The future is at hand

One of my dad’s favorite musicians, and my mother too. The secular prophet of the Riley family.

How are we writing the future of humanity?

When my father died, I found among his things an early version of an Oculus Rift. I was surprised, as he’d never mentioned the Metaverse to me; I wonder if he was just a little bit embarrassed by his interest. But of course it makes sense. He loved new technologies and the novels of William Gibson (I did too, once).

Among other things, in the early 1980s Gibson coined the phrase cyberspace to describe the intermediated digital world accessible we’d jack into using decks. Interesting etymology, cyberspace. The melding of cybernetics, the purported science of feedback systems, with the abstraction of space, something that is everywhere and nowhere. Feedback into nowhere.

The future is already here, it’s just not evenly distributed—you may have heard that from Gibson too.

Which already-present future awaits us, I wonder. Is it this one? This one? This one?

How are we writing the future of humanity?

“AI is just a tool,” something I’ve said and written more than once. I’ve done so deliberately, in the hopes of demystifying AI, making it intelligible, calming people down, perhaps neutralizing to some minor degree both the grandiose abundant promises of the techno-salvationists and the ludicrous death-cult fantasies of the so-called AI doomers. Humans are toolmakers, and we made AI. It’s just a tool.

L.M. Sarcasas warns me, be careful with that metaphor, or more pointedly, just stop using it. He reminds us that to call AI just a tool implies a neutrality that does not exist, as if it’s simply a matter of choosing between useful applications of AI versus those that are harmful. This obscures the ways in which AI permeates our environment and shapes our perceptions, as surely as language does. “I am astounded by how blithely some insist that it is all as simple as learning to use AI well, as if we had not just undergone a nearly 20-year, society-wide experiment showing that a so-called ‘tool,’ say a smartphone or a social media platform, will (mal)form even the most vigilant and virtuous user into its own image and shape. This is the blindness at the heart of modern technological hubris.”

I know this, and at times I’ve remembered to say it aloud. Ursula Franklin observed that technology is a mindset, a system, an ideology. Audrey Watters reminds us of this every week. AI may be a tool, but it’s not just a tool, and that distinction is vital. I vow to remember this.

How are we writing the future of humanity?

A quiet power that emerges early in Orbital: to see our planet not as an intermediate holdover space for our temporary existence, but rather the culmination of all that has preceded us, and that which protects us—in this sense, Heaven.

Shatner again: “I turned back toward the light of home. I could see the curvature of Earth, the beige of the desert, the white of the clouds and the blue of the sky. It was life. Nurturing, sustaining, life. Mother Earth. Gaia.”

Samantha Harvey imagines alien observers who stumble upon the International Space Station and wonder why we humans have chosen to do this, to have developed this device and sent it beyond the limits of our own sky with six souls trapped within? To what purpose?

How are we writing the future of humanity?

Let’s dwell for a moment on the methodology suggested here for carving our future: Writing, how are we capable of writing? I find it forever astounding. In 2001: A Space Odyssey, Stanley Kubrick imagined the moment when our human ancestors discovered the first tool (and promptly used it to start smashing the skulls of others), but I fantasize about when someone first recognized that a marking of some sort, an inscription, could represent something…well, something else. Accounting for a collection of grain, some think at the start.

And with time, words for ever more abstract ideas. Freedom. Justice. Solidarity.

Large language models, do they write? Yes and no. They certainly produce words. But the process they use in training, the statistical method employed to create correlations between words—or more accurately, words eviscerated into tokens—is always algorithmic and automated. The word for this process has always struck me as equally apt and harrowing: Autoregression.

A mechanistic process of stepping backwards, thoughtlessly.

Our world in autoregression.

How are we writing the future of humanity?

Here is a painting from 370 years ago, Las Meninas by Diego Velázquez :

The astronaut Shaun brings a postcard of this painting on his journey, given to him by his wife. In orbit 1, Harvey asks—who is the subject here?

Look, really look.

Is it the girl? The ladies-in-waiting? The king and queen reflected in the mirror (did you spot them)? The mysterious man in the stairwell? Velázquez himself, sizing up this strange little world?

Or perhaps, Harvey wonders: “Is the subject art itself (which is a set of illusions and tricks and artifices within life), or life itself (which is a set of tricks and artifices within a consciousness that is trying to understand life through perfections and dreams and art)?”

Do you begin to see why ever page of this novel feels precious? Do you feel how profoundly human all this is?

(Later, Pietro sees the postcard spinning in the air. After examining it for a few seconds, he declares: It’s the dog. The subject is the dog.)

How are we writing the future of humanity?

How are we writing the future of humanity?

My girlfriend is reading Orbital with me (free relationship tip: read together). Our love was borne of this newsletter; she found my writing, and so she found me. Yesterday, she told me that Orbital is really a love letter to our planet. To us.

AI will never love, and it will never write a love letter. AI will never break down crying when writing an essay over the weekend while listening to Lou Reed, feeling the ache of having lost both parents while feeling unmoored within space and time, adrift in a strange liminal state of consciousness, still capable of day-to-day functioning in all the required ways yet constantly feeling this tug, this pull, an undertow that is both of the past and yet somehow the future, sensing a newfound relationship with time that’s bound up with grief, never having expected as a child that the future would be so relentlessly worrying, and wanting so desperately to smash so many gilded pens, yet still clinging to the possibility of optimism without hope, and reaching now for words once copied from a memorial to Oliver Sacks, revisting this elegant testament to an essential humanity, to remember to care and feel empathy for those who are different or less fortunate, to have fun and love often, to find wonder and beauty and know gratitude, and unexpectedly in this moment of yearning to trace my fingers across the lived contours of my exulted lover, and I suppose I should stop now with whatever this is and luckily the neighborhood children have dropped by and they’d like to know if I can go outside and play and my answer is yes I say yes I will yes.

The earth is the answer to every question.

Is this Heaven? No, it’s Iowa.

Subscribe now

It’s an awkward time on the human-cognition-versus-AI beat, let me tell you, because:

1. The vast majority of normal human beings who use generative AI, I’d guesstimate somewhere between 95 to 99 percent, are presently doing so via written interactions with “classic” large-language models such as ChatGPT. Write a prompt, get a reply, lather rinse repeat.

2. Despite the versatility of these interactions being the very thing that’s led to mass AI hysteria broad exuberance about AI capabilities, we are now explicitly being told by the Big Tech companies that chatbots are not, repeat not, the future of AI. Prompt engineering is dead, long live Agentic AI.

3. As best I can tell, Agentic AI presently has three broad use cases, all of which give me a headache:

a. Coding. If you need to engineer some new software, you can tell Agentic AI what you want in general terms and it may provide you with something potentially useful. This, I concede, may prove to be an economically significant development, given how broadly software permeates every aspect of our society. But I myself have little need for custom-written code, nor am I interested in becoming an amateur computer programmer. I have nowhere really to go with this.

b. Convincing large businesses that they can save money. AI is really good at is ingesting vast amounts of data. As such, if you are a large corporation that possesses vast amounts of data, your leadership might be convinced that unleashing Agentic AI will discover new cost efficiencies, perhaps by rendering the need for middle managers obsolete. It’s not clear to me that this perceived use-case is in fact true, but for me personally it doesn’t matter, because I am not sitting on a trove of data to mine for cost savings. AI as Office Space is fundamentally uninteresting to me.

c. “Vacation planning.” Another use of Agentic AI is to grant it access to our personal computers and smartphones and allowing it to manage our digital affairs—vacation planning is the go-to example. (Check out this research on “Agentic AI for Trip Planning Optimization Application.”) The thing is, I enjoy planning my own vacations, it’s not a process I find the need to “optimize.” And the even bigger problem is that there’s no part of me that trusts these Agentic AI tools to invade my digital space. So, there’s little testing of them that I can do on this front.

You see my dilemma. For all my many criticisms of generative AI chatbots, at least these things provide an interesting non-example of human cognition to mess around with. But how the hell am I supposed to probe the cognitive capacities of Agentic AI?

Time for some Zork!

If you recognize this screen, congratulations, you’re old and nerdy. For the less familiar, Zork is a classic computer game created at MIT in the late 1970s and released commercially in the early 1980s by Infocom, the company that pioneered text-based interactive games. You see kids, back then our newfangled “microcomputers” couldn’t really process things called “graphics.” So instead we got text-based choose-your-own-adventure style games wherein we, the human players, would type various prompts that (we hoped) would lead to meaningful discoveries to drive the game narrative forward.

For Zork itself, this meant exploring a subterranean world known as the “Great Underground Empire” and collecting various treasures while avoiding being eaten by Grues or having your shit stolen by the infuriating Thief.1 It was fun and maddening. On the fun side, I spent many hours with Justin M., my best friend in childhood, drawing maps of the strange abstract world we were exploring.2 (As I discovered in the course of writing this essay, there’s an entire cottage industry of Zork maps, including the one I pasted above.)

The maddening part often involved figuring out how to phrase something such that the software would respond in meaningful fashion, rather than saying “I don’t know [word you just typed]” repeatedly and eventually taunting you for your stupidity:

Ok, so. Just a few months ago, Berry Gerrits, a researcher in the Netherlands, had the fun idea to test the current state-of-the-art LLMs—versions of Claude, ChatGPT, and Gemini—on their ability to play Zork.3 As he observes, Zork is a great game for comparing and constrasting human thinking with how LLMs do what they do:

This is what cognitive scientists call a “world model,” meaning, a set of mental representations that we humans form about (ahem) the world and how we expect it to function. A very big question hovering over AI is whether it’s possible for such systems to develop world models akin to what humans possess. (This is why AI gurus such as Yann LeCun, who readers may remember from my “exclusive interview” last year, is currently devoting his attention and efforts to this task, which he amusingly calls LeWorldModel. Fei-Fei Li, another OG of AI research, likewise has launched an AI effort called World Labs.)

As best I can tell, Gerrits tested LLMs in their “classic” chatbot-based format, activating “reasoning” mode on occasion, but making no use of their agentic capacities.4 And in this classic mode, their performance was unimpressive, as the models uniformly struggled to advance beyond the initial phases of the game: out of 350 possible points, none exceeded 75 (21%), and most of the time they scored much lower than that. Most damningly, the chatbots did not evidence the capacity for in-context learning. Instead, “they often continued attempting unsuccessful actions without adapting their approach, and failed to learn from mistakes across multiple game sessions. These patterns point to a fundamental distinction between pattern matching and genuine understanding.”

So classic LLMs suck at Zork—but what about Agentic AI? We’ve been told it’s both incredibly powerful and adept at solving problems with only minimal guidance. Might Agentic AI therefore be more capable at navigating the Great Underground Empire and gathering the treasures of Zork?

To answer that question, I forked over $20 to renew my access to ChatGPT Pro, which includes an “agent mode” that can be activated up to 50 times per month. I then ran 10 open-ended trials wherein I pointed the AI to the online Zork emulator and told it to “play to win.” And…and…

Reader, it did not win. In fact, Agentic Chat did very poorly. The chart below summarizes the results, but in brief, the highest score it achieved was 40 out of 350, about the same as the classic LLMs that Gerrits tested.

This is just a small test of course, hardly definitive proof of anything. But at least in these trials, I found no evidence of Chat’s agentic mode developing anything remotely akin to a Zorkian world model. It would make valid moves but with no discernible end goal in mind. Basically, there was a lot of this:

This is Agentic AI as clumsy lobotomized hamster.

More charitably, it’s impressive that Agentic Chat managed to play this Zork emulator despite ongoing interference from random pop-up ads (I chuckled when “Am I gay? Take this test!” spam started causing problems). Further, I am very curious how Anthropic’s agentic models would fare playing Zork, given that Agentic Claude is considered best in class at the moment. But I’m resistant to giving any money to Anthropic, in part because I think their CEO is an ass, in part because because I’m terrified that their per-token pricing might bankrupt me. That said, if anyone out there has a subscription and wants to investigate this together, please get in touch.

Here’s my hunch: over time, Agentic AI tools will get better at games such as Zork, just as they’ve improved on other complex, deterministic tasks. But will this improvement result from using efficient world modeling, the way that humans do? Or will they continue to use relatively brute force computing techniques that are as costly as they are inelegant?

Somewhere, Melanie Mitchell is nodding, because—from a scientific perspective—it’s not just whether AI can solve certain problems, but also how. As such, if we really are entering the “era of agentic AI” (whatever that means), it seems we still will have interesting possibilities for comparing and contrasting how we humans think to how these tools do what they do. Or so I am choosing to believe, perhaps out of professional necessity.

Next week: Can Agentic AI avoid dying of dysentery?

Subscribe now

1. Listen to the Dead Kennedys and then form a punk band and put out one banger of an anti-AI track (works best if you’re still in middle school).

I won’t be a slave to AI
AI is only imitation
It’s not the voice of my generation
Artificial intelligence can’t know the human experience

2. Assume leadership of one of the most powerful religious institutions in the world and then issue a radical anti-AI encyclical that, among other things, trolls Peter Thiel.

In the Magnifica Humanitas, under a section titled ‘We can all do our part,’ Pope Leo channels Gandalf to hammer home his point:

The twentieth-century Catholic author J.R.R. Tolkien, in the words of a protagonist in one of his novels, described our responsibility in this way: “It is not our part to master all the tides of the world, but to do what is in us for the succour of those years wherein we are set, uprooting the evil in the fields that we know, so that those who live after may have clean earth to till.”

Source

3. Write a biting satire of the many ways in which the tech industry sucks (AI in particular) that also trolls Peter Thiel.

Source

4. Hang out in libraries and local bookstores, for hours and hours.

5. Refuse, and learn about refusing from centuries of Black American resistance to white supremacy.

“When people try to sell you on the idea that the future is already settled, it’s because it is deeply unsettled. This promise of an artificial intelligence future is really just a collective anxiety that very wealthy, powerful people have about how well they are going to control us in the future. If they can get us to accept the future is already settled, that AI is already here, the end is already here, then we will create that for them.

My most daring idea is to refuse.” — Tressie McMillan Cottom

6. Deface pro-AI ads on the subways.

7. Don’t be afraid to go hard.

Source (you can buy these btw)

9. Send pro-human, anti-AI postcards to your cool AF mayor.

Source

10. Dance, dance, dance, because AI cannot.

Source

11. Remember that poetry is always punk.

Source

My friend Tomáš Baránek, a publisher and writer in the Czech Republic, was kind enough to interview me recently about human thinking and creativity contrasted with AI — and I love the title he chose, “We Need the Friction of Existence.” Friction is punk.

Subscribe now

This week’s essay runs long, so I’ve broken it into two parts—part one explores an economic theory about AI automation in society generally, part two drills into the paradoxical implications of this theory for education in particular.

The Economics of Desire

What if the AI enthusiasts are right? What if we see widespread AI-driven automation across huge swaths of the economy in the next several years? What does our future look like then—and what might it mean for education?

No small questions these. But I take them seriously and today I want to offer you some thoughts on how our world may change if AI is truly transformative to our way of life. Before we get into that, however, some background on my intellectual history with economics.

As I’ve hinted previously, I was a libertarian in my youth. It started with reading Ayn Rand in high school (me and Hillary Clinton both) before hitting its apex in my undergraduate years at the University of Washington. I’d fallen under the sway of Dr. Paul Heyne, an economist who, oddly for an academic, preferred teaching undergraduates to conducting research. As such, he proved to be a wonderful teacher as well as friend and mentor to me and many others. He cultivated a small group of acolytes who would gather with him on a near-weekly basis to talk about the ideas of “classical liberalism,” wherein we’d read Frederich Hayek, Ludwig von Mises, Frank Knight, all the people that give Peter Thiel a hard-on.

It was as intellectually stimulating as any period in my life has ever been. And Heyne cultivated not only my academic progress but also my career—after my junior year, he sent me straight into the world of the Koch Brothers through a fellowship with the Institute for Humane Studies, a think tank of sorts at George Mason University devoted to cultivating libertarian academics. (The Kochs may be evil, but unlike the vast majority of philanthropists in America today, they have always understood that political change is downstream of ideological transformation. Ideas matter.)

I’m sharing this as long-winded way of saying, there was a period of my life where I thought about economics constantly. Over time, my views have shifted significantly, and nowadays I am far more wary about our ability to reduce complex social phenomena to simplistic models, especially when such models are more theoretical than empirical. Nonetheless, I believe economics remains a useful discipline for generating insight into large-scale systems, particularly when it’s used to explain historical shifts in human behavior.

Last week, while investigating whether AI-generated writing detection tools are rapidly improving—more on that in a future post—my research led me to an essay from Dr. Alex Imas titled, “What will be scarce?: The economics of structural change and the post-commodity future of work.” According to his online bio, Imas is currently serving as “director of AGI economics” for Google DeepMind, and is academically affiliated with the University of Chicago—warning signs, I grant you, but his essay resonated immediately, for reasons I’ll now go into at some length.

Ok, so. The first prong of Imas’s two-part argument is grounded in the history of structural changes to economies, and specifically, how capitalistic industrialization leads to commodification. Prior to the Industrial Revolution, most goods were supplied in what we might call artisanal fashion—“the weaver who made your shirt, the baker who made your bread: you personally knew them, and their skill and reputation were tied to the product that they sold.” This shifted with industrial practices that mechanized the production of goods—technology plays a major role here of course, but so too does the atomization of human labor such that we become mere cogs in the goods-producing machines.1 As Imas notes, this gave Marx heart palpitations, but for others, then and now, industrial commodification was a major driver of prosperity, as numerous products became cheaper and more plentiful.

We need not pick a side in that fight here, nor in the comments please, that’s not what I’m driving toward. What’s important instead is Imas’s historical claim that, as the production of various items became commodified, and as incomes broadly rose, patterns of human demand shifted. Consider food. The automation of agriculture dramatically shrunk the proportion of the American workforce devoted to farming (from 40% in 1900 to 2% today), while making basic food more plentiful and cheaper. But because there’s only so many calories we can consume, when basic food ceases to be scarce, our demand shifts from quantity to quality—those who can afford to do so will spend more on going to nicer restaurants, buying organic fruit and (supposedly) sustainable meat, etc.2 In short: “As people get richer, they want fundamentally different things.”

So far this is all pretty standard Econ 101 stuff, but then Imas introduces his second main argument, the one that’s burrowed straight to the center of my brain: when basic needs are met, our desires increasingly become shaped by wanting what others cannot have. Imas decribes this as mimetic desire (citing French philosopher René Girard), and explains it as follows, with my emphasis added3:

Now let’s bring AI into the picture. As I’ve said tirelessly for years, crediting François Chollet (also French), AI is a tool of cognitive automation. The entire point, the trillion-dollar vision that our entire economy precariously rests upon, is whether we can automate and thereby commodify a large swath of cognitive activity involved in economic production. The CEOs of AI hyperscalers explicitly contend they can drastically lower or even eliminate the marginal “per unit” cost of intelligence. The CEO of Cloudfare just took to the Wall Street Journal to declare that, despite his company’s record profits, he’s firing middle managers and other back office “measurers” as fast as he can due to AI automation. Or as this unhinged Silicon Valley start-up called Mechanized puts it, “our long-term goal is the full automation of valuable work across the economy.”

That’s it, that’s the future they want. It’s not a secret.

What happens if they get it? What if I’m wrong that applications of agentic AI will be relatively narrow in scope, limited mostly to coding and accounting and other deterministic tasks? What if instead AI really does manage to automate a great deal of so-called knowledge work and broad swaths of our economy generally?

In such a world, Imas offers this conjecture: it will “trigger the emergence of something new: a post-commodity economy, where a growing share of expenditure goes toward goods and services whose value is inseparable from the human who provided them.” Demand for services will shift to demand for better human relationships, creating what he calls the relational sector. In such a world, “the human being is not just an input into the production process. Their judgment, attention, memory, warmth, or presence is an integral part of the value.”

Does that sound like anything we cover around here? Does that sound like…dare we call it…teaching?

That’s where we turn now.

The AI Education Paradox

Can education be commodified?

For the past several decades (and beyond, really), the answer by powerful and influential people in American society has been a resounding “yes, absolutely.” Whether in the form of test-based accountability or big-data analytics or value-add metrics or personalized learning, the explicit goal has been to quantify and automate the provision of education, and thereby improve it.4

I know this world well, because I once worked in the heart of “ed reform,” and I remain friends with many who are still committed to its ultimate vision. You’ll have to trust me when I say, most of ‘em are true believers. That is, they genuinely think that if we can just figure out the “key performance indicators” for teaching and learning, we’ll have the means to drive to system-wide performance improvement in schooling (sorry, the jargon is unavoidable). What’s measured is what matters. Welcome to life as a Gates Foundation program officer.5

AI of course is the latest iteration of this enduring effort of cognitive commodification. “The promise of personalized learning never delivered,” wrote John Bailey in 2023 (an advisor to Jeb Bush and numerous education philanthropies), but “today’s AI is different.” Maybe so, maybe so. But now let’s bring Imas’s economic analysis back into frame so as to imagine two possible future states:

1. AI is successfully used to automate a great deal of economic activity, but we discover yet again that the provision of education itself is fundamentally relational, and therefore impervious to such automation. In this future world, assuming AI generates new wealth (or “abundance” if you prefer), we would expect to see demand for “artisanal education experiences” to increase rapidly. The delivery of education becomes increasingly bespoke.

2. Conversely, suppose we discover instead that core aspects of education really can be automated by AI. In this world, B.F. Skinner’s behaviorist fever dream of “teaching machines” is finally realized, and every student is matched with an “infinitely patient, infinitely loving” robot tutor. Here’s the ironic twist: in this future world, the demand for artisanal education experiences should still increase rapidly. Remember, when something becomes commodified, mimetic desire kicks in—the yearning to have something that that others can’t possess. If everyone has access to an AI tutor, unique experiences with other human beings will become the class differentiator. Schools become restaurants.

This is what I’m calling the AI Education Paradox: The more that AI automates the service economy outside of education, the more this will lead to people to want highly humanistic educational experiences for themselves and their children. And yet, to the extent that AI automates aspects the provision of teaching within education, this too will result in parents wanting differentiated and bespoke educational experiences. All roads lead us to education at the center of a new relational economy, rife with mimetic desire.

Squint, and you might start to see signs of this transition already underway. While I have vowed to spend as little of my life as possible thinking about the pedagogical horror show that is Alpha School, it might just be a canary in the coal mine (an industrial-era metaphor I invoke deliberately). What is Alpha School promising? To parents who can afford its $40,000 to $75,000 annual tuition, they offer an automated experience of core academics, but also a highly curated set of fun stuff for kids to do. Michael Pershan clocks it perfectly here (quote slightly paraphrased, my emphasis added):

Move on to what, that is indeed the question. Wearing our most optimistic rose-colored glasses, we might fantasize that AI automation points us toward a renaissance of human-centered education. The current “techlash” might be short lived, but it might just as well be an early indication that technological intrusion into our lives has reached saturation, and parental demand preferences are shifting in structural ways. This is why, in the near future, I predict we’ll see more schools “of choice” promising they are tech-lite, or even tech-free. Luddite Charter Academy, coming soon.

But there’s a looming educational nightmare that’s just as visible, one that we’re already partially living in. For one thing, to the extent AI automation generates new wealth, we know—we know—this will be concentrated in the hands of a select few. There will be parents who have the means to place their children in artisinal schools of choice, while the remainder continue to attend underfunded public schools that buckle from dwindling student attendance. For another, the Balkanization of the American education system reflects the broader epistemic collapse manifesting across our society, where we no longer have any real shared sense of common reality—those who sacked Congress on January 6 are traitors to some, soon-to-be-very-wealthy heroes to others. These competing histories will be reflected and directed by competing curriculums. And just as democracy and public education—emphasis on public—rose hand-in-hand together over the past several centuries, so too may they fall apart in tandem. Christian Nationalist Academy, also coming soon.

The AI Education Paradox is thus a paradox-within-a-paradox (do you ever just stop and look at your hands, man?). If AI-driven automation becomes prevalent, it should lead to broadly renewed interest and demand for human teaching, while simultaneously exacerbating existing trends of institutional education stratification. Economic progress, educational progress, and democratic progress will no longer function as interlocking gears (if they ever have). Things will grind to a halt.

What then becomes of the nation-state? What then becomes of America?

Sounds grim, I know. But this essay, all 2,500-plus words of it, has been a thought experiment, an exploration of what might happen if AI-driven automation upends the economic (and educational) fabric of society, as so many predict it will. In reality, my bet remains that once the hype subsides, the tranformative power of AI will be relatively circumscribed. I often liken it to email, something ubiquitous in our professional lives yet hardly revolutionary to our experience of the world. John Gruber offers a similar and perhaps better analogy:

We can survive a new form of wifi, don’t you think?

Subscribe now

Who’s up for some new science regarding the history of ancient humans, and how it relates to a battle of wits between modern humans and ants?

(I immediately lose half of you to TikTok.)

Ok, thanks to those who’ve stuck around, and there’s some AI at the end of all this, I promise. But first: Let’s dive into some recent anthroplogical research that, conveniently for me, tee up a study of collective intelligence that I’ve been wanting to share with you for some time. Before getting to it, however, we’ll start with a question that’s surely been top of mind for many of you—why did the Neanderthals disappear from Europe roughly 40,000 years ago while our Homo sapien ancestors flourished?

It turns out this question has been the source of great anthropological tension for the past century or so. And until relatively recently, the consensus explanation was based on raw intelligence—Neanderthals, it was believed, simply weren’t as smart as us. Although they generally had larger brains, for multiple reasons—many of them weird and frankly racist—they were long considered unintelligent brutes, the knuckle-dragging cavemen of yore. On this view, we humans were simply were more individually cognitively capable, and thus more adept at surviving the changing climate of the European continent.

But new research is upending this story. For one thing, it turns out the anatomical variation in Neanderthal brain size to the H sapiens who were alive at the time parallels the variation in brain size among modern humans today. As such, according to some researchers, “the implied average cognitive differences between Neanderthals and modern humans are extremely small.” This, combined with the growing evidence that Neanderthals made tools, artwork, jewelry, and even mastered fire, strongly suggests need to update our caveman priors.

Here’s one intriguing hypothesis: perhaps H sapiens thrived because of our comparative collective intelligence. The anthropological record suggests that Neanderthals lived in much smaller groups than our human ancestors, and in more widely dispersed areas across the continent. Using complicated ecological modeling, some researchers now believe this left Neanderthals more vulnerable to environmental change and other external threats. In contrast, the bigger and networked H sapien societies possessed more collective intelligence to grapple with the changing world via cooperative, coordinated activity. Over to paleontologist and evolutionary biologist Nicholas Longrich:

Yes, let’s consider ants!

Just over a year ago, a team from the Weizmann Institute of Science published a delightful study of individual versus collective intelligence that pitted humans versus ants on a very specific cognitive challenge. The task at hand? A version of what’s called a “piano mover’s” puzzle, having to maneuver an awkward and bulky physical object through a tight space. In this study, both humans and ants had to move a T-shaped beam through a relatively narrow corridor, as pictured above. You’ll get an even better sense of the competition by watching this video:

Both humans and ants were tested in three different “solver” scenarios: (1) individual humans versus individual ants; (2) small groups of humans and ants (six to nine each), and (3) large groups of humans (around 25) versus large groups of ants (around 80). Additionally, for the group scenarios, in some trials humans were allowed to communicate talk with each other, but in others they had to wear sunglasses and masks so as to limit their communicative abilities to pushing and pulling on the beam, as the ants did.

What the researchers found will not surprise anyone who’s ever tried to coordinate a multi-family camping trip. In the first scenario, individual humans beat individual ants across the board—hooray us. But in the group scenarios, however, ant performance improved with greater numbers, whereas human performance decreased. What’s more, when human communication was restricted, ants actually solved the task faster than we did. In other words:

Alert readers may notice there’s a seeming contradiction here. On the one hand, I’m suggesting that our ancient ancestors may have triumphed over Neanderthals because of our greater collective intelligence. Yet, on the other, I’m pointing to evidence that suggests that our capacity to solve certain types of problems actually drops as more of us gather together. What gives?

Well, both of these things can be true, and that’s the point I’m driving at. Under certain circumstances, such as, say, navigating complex European ecosystems undergoing climate change, collective H sapien intelligence may have proven highly advantageous. But under very different conditions, such as, say, trying to move a weirdly shaped object through a cramped space, our individual intelligence works against us. There is no universal truth about intelligence, collective or individual, that suggests more is always better. We need both nuance and theories as to why certain types may be advantageous in particular settings.

Which brings me to my obligatory AI-related commentary. As I’ve argued previously, Big Tech CEOs such as Sam Altman and Dario Amodei explicitly conceive of intelligence as a commodity, a good to be traded on the open market and priced in per-unit costs. But even if we grant for sake of argument—for sake of argument!—that AI represents some novel form of cognitive capability manifesting in digital form, that in no way means that it will prove to be universally beneficial for any particular problem we humans want to solve. Software engineer Simon Willison neatly illustrated this tension when he recently observed that, using AI tools, he can now produced 10x as much code in a day but no longer has the capacity to review it, leaving him feeling both guilty and worried about what might happen if something goes wrong. And we’ve already seen things go wrong.

Zooming out a bit, Henry Farrell and Cosma Rohilla Shalizi recently published an excellent analysis of AI as social technology that touches on this theme. They begin by noting that various human institutions are designed to foster collective intelligence by channeling “the production and gathering of knowledge across many individually limited minds”—the birth of bureaucracy. The downside, however, is that these “vast systems [] are regularly indifferent, and sometimes inimical, to the particular fates and desires of individuals and groups.” Indeed. From this foundation they explore myriad questions of how AI might intersect with various forms of bureacracy management, highlighting the many “known unknowns” before reaching this conclusion:

There is no universal method of solving complex problems through some form of digitized, commodified intelligence. And while humanity benefits from our capacity to share our knowledge with each other, our T-beam competition with ants demonstrates that our intelligence can work against us too. So choices will have to be made. What’s becoming clearer by the day, though, is that voices of collective concern about AI in society are getting louder and louder.

I mean, just listen.

Subscribe now

The Era of AI as Chatbots is over, welcome to the Era of AI as Corporate Agents. Don’t believe me? Time for a roll call of corporate pronouncements over the last month:

Platforms and workflows and workloads and automation and agents agents agents and…and…it’s all so very soul crushing. As I mentioned last week, in theory I’m glad that the major AI companies are focusing their attention serving “enterprise” clients, in the hopes this means they’ll do less damage to our childrens’ cognition. In reality, however, the industry-wide pivot toward channeling agentic AI into corporate IT departments does suck some wind out of one’s intellectual sails, at least if you’ve spent the last several years using AI-as-chatbot as means of comparing and contrasting AI with human cognition.

This shift to agents is significant, but what does it mean for our supposed “AI future”? “Prediction is very difficult, especially if it’s about the future,” Neils Bohr is rumored to have said, so I generally steer clear of speculation about the business side of AI (the occasional OpenAI-as-Enron analogy excepted). That said, were I to be hypnotized and interviewed by consultants about what lies ahead for AI, here’s how it might go…

Are you saying AI chatbots will go away?

No. ChatGPT will still be around, and various chatbot interfaces such as Gemini and the like will be inserted (not to say jammed) into a wide suite of software tools we use, whether we ask for them or not. But the efforts of the major Big Tech AI “hyperscalers” will no longer be primarily aimed at improving the quality and usefulness said chatbot interactions with consumer users. The frontier of AI development will be on improving agents.

What about “prompt engineering”?

Not a thing, if it ever was. The fundamental value proposition of agentic AI is that it figures out how to solve problems independently using “command and control” instructions, rather than through dialogue with users. Apologies to those who tried to “Learn The Art of The Ask” because that was time wasted.

Will this put an end to promises of Artificial General Intelligence?

I think so. For the past three years, we’ve had to endure endless claims about how feeding data and adding compute chips to large-language models would lead to some sort of super-sentience, “scale was all we need,” etc etc. But look again at that quote from OpenAI’s chief revenue officer, now the focus is on “orchestration, control, observability, security, integration, and governance.” No time for superintelligence when there are TPS reports to generate.

Does this make you a little sad?

Unexpectedly, yes. While I’ve been steadfast in poking holes in claims that chatbots put us on the fast track to human-level artificial intelligence, the truth is I’ve relished the broad attention that’s been directed toward deep scientific and philosophical questions regarding cognition, consciousness, what makes us human, etc. Agentic AI feels like a pretty hard turn towards the corporate and the technical. Not my jam.

But might AI agents be a form of AGI?

No. No man. Shit no, man. What we’re seeing right now is that agentic AI is potentially useful for multi-step tasks involving large data sets and grappling with relatively objective methods to determine what’s true or false. As Nathan Lambert from the Allen Institute for AI observes:

Coding, we’re there already. Terminal tasks, to the extent that means “stuff that can only get done on a computer terminal, such as cybersecurity,” also plausible. Finance and accounting? Sure. Healthcare? Well, obviously I have concerns. Law, uh, not really a deterministic profession, as anyone who’s ever tried a case can tell you. Beyond that, who knows. Construction seems safe.

Do you expect AI-in-education enthusiasts will pivot to talking about AI agents transforming learning?

Sigh, yeah. I’ve noticed my professional “education world” lags my professional “AI world” by six to 12 months, so by the end of this year we should see EdTech companies promising us agents that will personalize learning to unlock six sigma gains and blah blah blech. This won’t be any more effective than using AI chatbots as tutors, of course, because the process of learning is just about the least deterministic thing imaginable. But I’m already steeling myself for the delusional hype about agents heading our way.

What would you say you do here?

As you can tell, I’m struggling. Going forward, I don’t want to parse the business plans of AI hyperscalers, that’s not my beat. Will there still be enough on the cognitive science side of AI to cover? Possibly, especially if Gary Marcus is right that new forms of AI are incorporating neurosymbolic reasoning methods. Still, one does have to wonder whether AI is going to end up like email, something ubiquitous in our professional lives yet deeply and profoundly uninteresting as an object of study.

Ugh. Let’s close with some cathartic release.

I recently appeared on a Canadian podcast called Eh I (gotta love it) to discuss the limits of AI in the classroom—you can find our conversation here.

Subscribe now

Well here we are, still reeling from the emotional devastation of Dan Meyer’s poignant eulogy for the recently deceased Khanmigo, only to have yet another tragic death in the AI-in-education family. Please join me, won’t you, in offering condolescences to the recent passing of ChatGPT’s “Study Mode,” and may its memory be a blessing.

What’s that, you don’t remember Study Mode? Tragedy on top of tragedy. You see, one year ago almost to the day, a relentless wave of negative press hit OpenAI as journalists discovered that students were using chatbots en masse to avoid effortful thinking, aka “to cheat.” As such, the company quickly announced a new product feature within ChatGPT that would ensure it would be used solely for educational good rather than evil: Study Mode. And OpenAI promised us that when Study Mode was activated, ChatGPT would “be engaging and interactive, and to help students learn something—not just finish something.”

There were, of course, some obvious challenges with Study Mode, as people quickly observed. For one thing…

…and for another…

Still, flaws notwithstanding, over the last year we could at least say that Study Mode…existed…as something that students seeking cognitive repentance might activate to curb their wicked cheating ways. That is to say, if they clicked the little “+” in the prompt box, there they could find Study Mode, waiting for activation such that—and here again I quote from OpenAI’s press release—students would “think critically about their learning.”

Alas, critical thinking is dead, and so too is Study Mode for all intents and purposes. Two weeks ago, I debated James Donovan, OpenAI’s Head of Learning & Cognitive Outcomes Research, regarding the role of chatbots in education. As our conversation kicked off, moderator Alex Grodd expressed some befuddlement related to his inability to find Study Mode, and Donovan confirmed it is no longer a tool that ChatGPT users can directly access in the “vanilla model” (that is, normal ChatGPT). Instead, ChatGPT will now allegedly “detect” when students are trying to study and then automatically refrain from providing answers. To which I say:

Donovan did suggest that for “B2B” users of ChatGPT in education—which apparently means countries such as Estonia that have partnered with OpenAI to embed ChatGPT in their education systems (shudder)—Study Mode will remain an option. Cool, that’s nice for Estonian kids, but not really relevant to the 400 million students worldwide who are using vanilla ChatGPT every day.1 Donovan, a smart and engaging guy, thus ended up arguing—based on secret OpenAI data—that students really are “cognitively augmenting” their education. You can judge for yourself how persuasive that is:

Study Mode was always a ruse, of course, a PR exercise masquerading as pedagogical safeguard, but while it’s admittedly fun to dance upon its grave, something bigger is happening at OpenAI regarding its education posture. Remember last year when Leah Belsky, OpenAI’s VP of Education, promised us an “education moonshot”? I think she’s still employed there, but the entire “moonshot division” at OpenAI appears to have run up the curtain to join the choir invisible as the company desperately tries to figure out how to generate revenue in advance of its suddenly somewhat dicey IPO prospects.

To wit: a few weeks ago Denise Dresser, OpenAI’s Chief Revenue Officer and de facto COO at present, issued an org-wide memo that leaked—and boy, it’s hard to see how education fits in to what they have planned going forward. Time for a brief fisking:

As we start Q2, I want to begin where we always should: with our customers. I have been spending time with leaders across our largest enterprises, most influential startups, and key venture firms. The message is clear. People are excited about what we are building, and they want a deeper view into our roadmap so they can plan with confidence and stay ahead of the market.

Enterprises, startups, and venture firms—but no mention of talking to students, who still comprise the majority of ChatGPT users (never forget this!). But students are not paying customers, and this memo is crystal clear about who really matters to OpenAI right now.

Enterprises buy business outcomes….They pay for higher revenue per employee, faster cycle times, lower support costs, and better execution.

Yep, this’ll map neatly to the culture of schools.

Our compute advantage sets us up to deliver continuous leaps in capability….Every step forward in compute lets us train stronger models, serve more demand, and lower the cost per unit of intelligence.

Oh yeah, teachers will love love love the idea of calculating the “per unit cost of intelligence.” Sign ‘em up.

The market has moved from prompts to agents. That shift is a massive opportunity for us.

Customers want systems that can reason, use tools, operate across workflows, and perform reliably inside real business environments.

Oh you don’t say! RIP, “prompt engineering,” long live AI agents, I guess. Does this mean we no longer need to provide professional development to teachers to improve their “AI literacy”?

I’ll remove the tongue from my cheek now, because I do believe this corporate reorientation “from prompts to agents” is a big deal, and I plan to write more about it soon. For now, though, I’ll just observe that while businesses may be lining up to have digital agentic hamsters scurrying about their databases, it’s not clear that normal humans share in that excitement. As teacher Stephen Fitzpatrick notes, “AI is being built for coders,” which is cool for them I suppose…but what about everyone else? Over to Elizabeth Lopatto in The Verge:

Me too, Liz, me too. And look, while I’m intellectually curious about AI, there is no part of me yearning to expose my private, personal data to AI agents. Not only do I not trust the technology, I’m not even sure what I’d want a coding agent to do. John Herrman with the Intelligencer recently ran into the same problem, with hilarious results:

Mercifully, humans are not firms, and students are not enterprises. And strangely enough, I’ll be (somewhat) relieved if OpenAI pivots to “B2B,” because it should mean less education malpractice from Belsky and team. As I note in the clip below, such harms are largely invisible in the moment, but eventually they will be made apparent. The good news is that concerned parents, students, educators, journalists, and others are observing this malpractice first-hand, and they do not like what they see.2

And they—we—are organizing to stop it.

Subscribe now

Well, hello! Nothing like a major feature in The New York Times to drive up one’s subscriber numbers, so welcome to the 600-plus new Cognitive Resonance readers—I’m grateful you’ve stopped by, and I hope you’ll stick around. For the most part, I focus here on comparing and contrasting human cognition with generative AI, with a healthy dose of ranting about dumb applications of AI in education. And on occasion things will get a tad personal and political, but I try to keep it contained.

That said, the story of my father dying because of his mis-reliance on AI blends the professional and personal, of course, and since so many of you arrived here because of Teddy Rosenbluth’s in-depth reporting on what happened to him, I want to share a bit more about the experience of the story process itself. I had no idea that when Teddy reached out to me that she intended to comprehensively investigate what happened over the past 18 months, and I suspect she didn’t either—that’s how reporting goes. But both her journalistic process, as well as new facts she uncovered along the way, continue to affect me deeply.

So, to start, being “reported on” has revealed why my many attempts at undergoing therapy have never worked out. I often say the bedrock principle of cognitive science is that we understand new ideas based on the ideas we already know, but it turns out that’s true for understanding new people, too. Or at least me.

By which I mean, when Teddy flew out to Austin in early March to interview me for three whole days—the Times is thorough, y’all—it was only after she’d spent several weeks digging into my family history. As such, when we sat together in my kitchen, she came prepared with questions about the entire arc of my father’s life, the early hopes, the physical setbacks, the set of unique experiences that comprised the story of his time on this Earth.

She wanted to know about his interests in technology, and I delighted in telling her about the time he jerry rigged our Dodge Caravan with a portable television and VCR player for our cross-country move from Long Island to California. We watched together an odd video he made in the last year or so describing his concerns about the re-election of Trump (apple didn’t fall far from the tree there). And she had an amusing fascination with the Lay-z-Boy chair that served as his de facto life nerve center in the living room of his condo in Edmonds, Washington. She was bummed I never could find a picture of it.

What I learned was simply this: Describing my father’s life to a relative stranger was cathartic. Validating. Therapeutic. Through answering Teddy’s questions about my father, I felt seen, as if she was starting to understand me based on her growing understanding of my dad’s life.

I had a story to share and someone wanted to listen. Isn’t that what we all want?

(This of course is what students so often feel with their human teachers, too. It’s why the teacher-student relationship often stands second only to parenting in its impact on who we become, and the character we form.)

The upshot is that, when Teddy called me two weeks ago to tell me the story would soon be published, my immediate sensation was bittersweetness, if that’s even a word. As I told her, only semi-jokingly, as long as the story was still being reported on, she was obligated to take my calls and talk to me about my dad. She kindly relayed that she often stays in touch with the subjects of her stories, so I’m looking forward to our future monthly check-ins regarding how the Riley family is doing. (I’m kidding, Teddy. Mostly.)

But Rosenbluth is a reporter first and foremost, not a therapist, and she reported the hell out of this story. Despite being granted a generous 3,200 words by her editors, however, hard choices had to be made about what to include and exclude, and here’s where I want to share with you something she uncovered through her reporting that didn’t make it into the final story that, quite frankly, enrages me still.

Quick recap: on July 15, 2025, after discovering that my father was relying on a “report” by Perplexity AI to reject his oncologist’s recommended treatment, I emailed two oncologists whose research was referenced therein: Dr. Adam Kittai, then at Mt. Sinai, and Dr. David Bond at Ohio State University. When I emailed them, the questions I posed were not focused on the AI report itself so much as whether my dad was at risk if he continued to forego treatment for his leukemia based on his (mistaken) belief that he was undergoing Richter’s Transformation. I didn’t expect a response, but to my suprise both emailed quickly to confirm that my dad should follow the recommendation of his primary oncologist, and nothing in their research suggested otherwise. And that’s where things were left.

That is, until Teddy had the smart idea to circle back to same oncologists to ask them to take a closer look at the Perplexity “report” in detail. So they did, essentially conducting a peer review of the output generated by this mindless tool. What they found was not an error here or there but rather a document that was basically nonsense from start to finish.

Thus, after the NYT story was published, I contacted both oncologists to ask if I could share publicly what they found, and they helpfully agreed to let me do so, which you’ll see momentarily. Even for someone like me who knows the likelihood of these tools producing false output, this is shocking to read, and I’m presenting just a sample of their comprehensive commentary.

From Dr. Adam Kittai – you can just focus on the yellow notes:

And here’s one of 10 pages of notes of errors identified by Dr. Bond, including “one nonsensical statement not supported by evidence”:

I’ve defended the use of the term “hallucination” to describe false output from LLMs for pragmatic reasons—when a phrase sticks in the public consciousness, that’s not for nothing (see also “AI skepticism”). But after reading this defenestration of the Perplexity AI output by these two human experts, hallucination feels inadequate in capturing the full magnitude of the bullshit spewed. In every section of the AI-generated "report,” in every sentence even, there are misleading or wrong or unsupported or nonsensical statements. But of course my father was in no position to know this, because he wasn’t an oncologist.

Nor was my dad’s experience an outlier. In the comments to the NYT story, you’ll find many reports from doctors and patients who have discovered the unreliability of AI as tool for medical diagnosis: For example, one doctor said this:

“As a primary care physician, I have used AI tools to explore possible causes of difficult to diagnose patient symptoms. While helpful in bringing forth diagnosis that might have been missed, as one drills down on how to manage a particular disease I have seen it confidently assert erroneous conclusions. When I point out the scientific inconsistencies of what it has asserted it reverts to sycophantic praise for my intelligence and abruptly changes its recommendations. What worries me is that I was only able to spot it’s inaccuracies because of my depth of knowledge and experience in the subject, something that the average user does not have.”

And one patient shared this:

“I found real comfort and value in the AI responses. You can’t just message your doctor at 11 p.m. and get an immediate answer as to why you’re experiencing some new and confusing symptom. So I get how AI can become a trusted medical ‘friend.’ However, at the end of the day, AI was completely wrong about my diagnosis. In fact, out of probably 20 different possibilities it listed, my real condition was never an option it presented.”

This goes beyond anecdote too, as there’s actually robust empirical evidence that AI should not be used medically. Gary Marcus recently published a helpful essay summarizing four recent research studies on this topic, and concludes: “LLMs don’t know what they don’t know; they work decently well with the information they’ve got but don’t know how to conduct clinical interviews, and in the hands of the lay public can easily give bad advice because the proper questions never get asked, either by the patient or the LLMs.”

How is any of this legal? Good question, and it might not be. Of course, the NYT story includes a bland quote from some PR flack at Perplexity saying the company was “deeply saddened” to learn about my father’s death. Sure, right, thanks. Not mentioned in the story is that Perplexity was already aware of what happened my dad prior to Teddy contacting them—so while I doubt anyone there feels any genuine emotional burden for the role their product played in killing one of my parents, no doubt their lawyers are keeping tabs on things. I’ve long joked that “dangerous when used as intended” is a trial lawyer’s favorite phrase, and here, Perplexity AI was more than dangerous. It was lethal.

With this essay, I will conclude my public grieving of what happened to my father, though I will remain forever grateful for the kindness and support from so many of you. The truth is, I’ve found this experience restorative. My father was a remarkable person who lived his life with dignity and curiosity, but he was somewhat of a hermit, so it’s meant a lot to share his story with the world, even in tragedy. There’s just one thing I wish had make the final cut of the NYT story, and that’s the solitary comment he ever shared on this Substack, in response to my essay about his failing health. I didn’t tell him my plans in advance, so I was nervous when that newsletter went out—would he be mad for my having shared something so personal?

He responded with this:

What you wrote (with considerable elegance) goes to the heart of who we are and how we are different from any AI model of the human mind: emotions. I don’t mean simplistic descriptions (e.g., oxytocin “causes” feelings of attachment) but it is an essential part of defining how we experience the world, from love (eros and agape) to feel an enriched response to a poem or piece of music and so on. This is why computational models are inherently inadequate; they do not arise except as add on subroutines to mimic the emotional depth of our experiences. And it is, to use the words of Victor Frankl, the last freedom, our freedom to choose our attitude.

I have nothing but gratitude for how my life has gone. To love and be loved, to fight the good fights, to live with the contradictions of sorrow and joy and never fail to appreciate a good joke is for the living alone.

That was my father, Joe Riley, a good man who loved his children deeply, and who I loved so very much.

Last Saturday, I was interviewed by Paul Henley of BBC Newshour regarding what happened to my dad. You can listen to the four-minute segment below.

Subscribe now

(Slide created by Jane Rosenzweig, Director of Harvard College Writing Center)

2. Ask the AI-in-education enthusiast if they are familiar with recent research indicating that generative AI leads to widespread “cognitive surrender.”

Shot:

“Conceptually distinct from cognitive offloading, which involves strategically outsourcing a discrete task to an external tool (e.g., using a calculator), cognitive surrender represents a deeper abdication of critical evaluation, where the user relinquishes cognitive control and adopts the AI’s judgment as their own….

Across our studies, we observe that when System 3 [i.e., AI] was available, people readily engaged it and frequently adopted its answers. This shift reflects a reallocation of cognitive control rather than mere effort saving. System 3’s fluent, confident outputs are treated as epistemically authoritative, lowering the threshold for scrutiny and attenuating the metacognitive signals that would ordinarily route a response to deliberation. In the case of cognitive surrender, there is a shift in the locus of control, with an external system (System 3) occupying the default position.”

(Paper here https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6097646, with my emphasis)

Chaser:

(Paper https://arxiv.org/abs/2505.01106, BlueSky thread here)

3. If you feel the need to pile on with research, consider citing to this recent report from Stanford showing the complete lack of empirical research to support the use of AI in education.

“Research on how AI impacts K-12 students and educators is still extremely limited.

“As of October 2025 the AI Hub for Education Research Repository contained over 800 academic papers relevant to AI in K-12 education. Our review found that only a small subset (20 papers) produce strong causal evidence. Causal evidence provides the strongest basis for estimating how a tool impacts students and educators. The current causal research is still very limited: we did not identify any high-quality causal studies in K-12 settings in the U.S. for students and very few for teachers.

Students

• Immediate gains with access: AI tools significantly improve student performance on math practice, programming projects, and writing tasks while students have active access to the technology.

• Short-term boost, uncertain transfer: AI improves performance with access but when assessed independently without AI support, effects are mixed.

• Easier doesn’t mean better: AI tools can alleviate students’ cognitive burden and foster positive experiences in learning, but can be at the expense of deeper thinking.

• Pedagogical design matters: Tools designed with pedagogical guardrails (such as AI chatbots for tutoring that provide step-by-step reasoning instead of direct answers) show more promise than general purpose AI tools.

(source, my emphasis added re lack of causal evidence)

4. Ask the AI-in-education enthusiast if they are familiar with any of the recent efforts led by students pushing back hard on the intrusion of AI into their education.

a. In Pennsylvania

“Through its countless new programs and AI-centered events, Penn has positioned AI as an inescapable future that we all must accept in order to achieve success. There is no doubt that AI is part of the current occupational landscape, and we will certainly encounter it long after we graduate. Nevertheless, we attend this institution to develop hard skills, question the world around us, solve problems, produce new ideas, and the ability to think for ourselves. With the University forcing AI into our learning every chance it gets, do we end up gaining knowledge or cheat codes?

“The irony is that as Penn pours endless money and energy into AI advancement in its attempt to get ahead, the University is only quickening its own demise. AI cannot coexist with education — it can only degrade it. As technology advances and workers are replaced by machines, schools are some of the only places we have left to explore and wrestle with human thought. With our own university leading the charge, AI is now corrupting those few sacred spaces and leaving us with nowhere to engage in true scholarship.”

(Source, with my emphasis added)

b. In Colorado

“In an online survey conducted by Flynn Zook, a CU Denver student, nearly 300 respondents weighed in on the agreement. Fewer than 10 expressed clear support, while a small number said they were undecided. Many respondents raised concerns about environmental impact, intellectual property and the potential use of tuition dollars to fund the initiative.” (Story link)

c. In Ohio

(Source)

5. Politely point out that Sal Khan, perhaps the most prominent advocate for the capacity of AI to “revolutionize“ education, has recently changed his tune.

“‘For a lot of students, it was a non-event,’ Khan told me recently about his eponymous chatbot, Khanmigo. ‘They just didn’t use it much.’”

“Kristen Musall, a geometry teacher at Hobart High, gave Khanmigo a try when it first rolled out. Musall appreciated its encouraging, teacher-like tone, but she found that students didn’t really care for the bot….Musall no longer uses Khanmigo in her class. She says there’s been more enthusiasm for the product among administrators than teachers in her school.

“Kristen DiCerbo, the organization’s chief learning officer, said AI can only respond to students based on what they ask. And it turns out, she said, ‘Students aren’t great at asking questions well.’”

(From Chalkbeat’s Matt Barnum)

6. Direct the AI-in-education enthusiast to the PureGenius website to see if they get the joke.

(source; feel the future)

7. Ask the AI-in-education enthusiast if they are familiar with the broader pushback against the intrusion of education technology into schools led by educators and parents.

“McPherson Middle School, about an hour’s drive from Wichita, is at the forefront of a new tech backlash spreading in education: Chromebook remorse.

“Schools in North Carolina, Virginia, Maryland and Michigan that once bought devices for each student are now re-evaluating heavy classroom technology use.

“Now children’s groups and educators concerned about screen time are turning their attention to school-issued laptops and learning apps. Parents are flocking to support efforts, like Schools Beyond Screens and the Distraction-Free Schools Policy Project, to vet and limit school tech.

“Sarah Garcia, also 13, said spending less time online had prompted students to talk more. ‘Since we don’t have our Chromebooks in front of our face,” she said, “most people now interact with their, like, peers and stuff.’”

(source)

8. Gently remind the AI-in-education enthusiast that we have evidence in our own lifetime that highly addictive products marketed to children that cause serious harm are something we can address through policy and norms.

(source)

9. If the AI-in-education enthusiast has the audacity to cite f***ing AlphaSchool as counterexample and “proof of what’s possible,” liberally reference any or all the myriad reasons this is one of the most embarrassing possible arguments they could make.

“Former Alpha School employees told me that the company’s increasing reliance on generative AI in every aspect of its operation, as well as the constant monitoring and tracking of every student’s mouse movements, is making students anxious and does not always provide the quality of education Alpha School advertises to parents.

““All educational content is obsolete. Every textbook, every lesson plan, every test, all of it is obsolete because gen AI is going to be able to deliver a personalized lesson just for you,’ Joe Liemandt, Alpha School’s ‘principal’ and the founder of Trilogy, the company that owns many of the apps used by Alpha School, said in a podcast interview published last year.

“When a student requires help with additional questions, the chatbot fails to identify which specific question is being addressed,’ an internal Alpha School document outlining issues with AlphaRead says. ‘Accuracy of the content provided by the [AI] tutor is a concern. There are instances where it not only delivers incorrect answers but also provides convincing yet flawed justifications. Despite raising multiple queries about a particular answer, the chatbot erroneously confirmed an incorrect option as correct.”

(source)

From teacher Michael Pershan:

“Alpha is not trying to provide the best, most ambitious math or ELA education possible according to conventional understandings of that term. If they were, they’d keep studying ELA/math in the afternoon. Instead, their goal is to minimize the time spent on core academics while maximizing skills.

“This is unusual! This is not what most schools are trying to do!

“What’s most novel about Alpha School and Math Academy is their fundamental orientation towards K-12 schooling. The goal, quite expressly, is to minimize it and move on. Move on to what?”

(source)

From teacher Dylan Kane:

[Dylan explains how AlphaSchool measures its “success” by demonstrating how much faster students improve on the NWEA MAP, a mediocre-to-shitty assessment of student learning, compared to national average of students. In 2024-2025, using mediocre-to-shitty ed-tech product iXL, AlphaSchool students outgained the national average by 2.6x. Dylan reports on how students are doing this year using AI: on math, 2.5x, and on reading 2.8x]

“Again, let’s take these numbers at face value for a minute. Last year, the program was mostly iXL + a culture laser-focused on motivating students. There was little to no generative AI involved. This year, Alpha School has overhauled their academics, released their own platform, and incorporated generative AI throughout. They are finally doing what they say they are doing: AI-driven schooling. And the results are…more or less the same?

“This is completely fascinating to me. 100 million dollars, tons of hype on the internet, grand claims about the future of education. And the results haven’t budged from bribing kids to try hard on iXL?”

(source)

10. If all else fails, try appealing to the poetry of human existence. But don’t hold your breath.

(Source)

Our days are indeed precious on this earth. Today, the The New York Times published a long story about what happened to my father due to his reliance on AI for medical guidance. I am very grateful to reporter Teddy Rosenbluth for sharing his story with the world, and to all of you for your enduring support.

Subscribe now

Unexpectedly and unintentionally, I’ve created a three-part mini-series (very mini) on the evolving landscape of AI R&D. To recap parts one and two:

1. Two weeks ago, I wrote about the push for specialized AI models in lieu of developing artificial general intelligence.

2. Last week, I shared my interview of AI guru Melanie Mitchell, a conversation that concluded with her thoughts on “AI agents.” Mitchell observed that AI coding agents were perhaps different in degree and kind from other forms of more general AI agents that could, say, go out and make purchases on our behalf. I then shared that my own mental model for “AI agents” has all but dissolved as a result of this confusion.

This week, please join me in going from mental-model dissolution to modest resolution! For part three, we’re going to dive into the weeds a little—just a little—on coding agents, relying on this very helpful essay that LLM engineer Sebastian Raschka dropped over the weekend. Warning: There will be some technical parts up front, but I promise we’ll soon get to some interesting questions about AI more broadly.

Claude Code! Codex! Cursor! If you work in software engineering of any sort, perhaps you’ve been using these tools to “vibe code” or “codemaxxx” or whatever. On the other hand, perhaps if you’re like me, you’ve thought to yourself, “huh, I wonder how those things work, too bad I know almost nothing about coding,” and gone about your day. Perhaps unlike me, you have not felt overly worried about this, since (unlike me) you don’t feel a professional obligation to at least have some basic mental model of what the hell is happening with the latest AI tools. But over here at Cognitive Resonance headquarters, lemme tell ya, it’s been the source of some anxiety—something significant is happening, and I’ve been a little lost trying to figure it out.

Enter Raschka’s timely and helpful essay, which you might want to read in its entirety. But the quick-and-dirty version is that coding agents involve three basic components: (1) the core large-language model; (2) a process for crafting and testing computer code (an “agent loop”); and (3) a cultivated environment of information and tools that the coding agent can draw upon (“runtime supports”). The term of art for this is the “coding harness,” and Raschka diagrams it as follows:

Before continuing, a brief interjection: I think a great deal of the present disconnect between AI enthusiasts and AI skeptics around AI model capability is made visible via this diagram. When software engineers and other AI enthusiasts criticize the “stochastic parrots” crowd nowadays, they are often saying that critiques of LLMs that fail to take into account these nifty new developments around agent loops and runtime supports are, well, just living in the past, man.

And they have a point. It may be rhetorically fun to refer to LLMs as, say, “synthetic text extruding machines,” but if that “synthetic text” is being “extruded” into really powerful and not entirely understood forms of recursive coding tools (and it is), and if this is undeniably proving to be useful to software development (and it is), it becomes very easy to ignore such rhetoric as divorced from present AI engineering reality. It’s noteworthy that Colin Fraser, a data scientist who’s been criticizing the limits of LLMs for years (including around these parts), now says that “AI tools have gone from completely useless for me to highly productivity enhancing.”

At the same time, LLMs still play an essential role in the vibe-coding process, as they serve as the interface between the human user and the AI coding model. Thus, the problems arising from the inherently stochastic-y and parrot-y nature of LLMs do not just magically go away. Indeed, as we’re about to see, a great deal of the progress in the capability of coding agents is due to the methods developed to control or limit—to harness —the naturally unpredictable and often inefficient computational processes that underly LLMs.

But how? I’ll summarize here in broad brushstrokes lest all of our minds melt, but it includes: (1) a repository of information that the coding agent will continually draw from when prompted—you might think of this as its domain-specific long-term memory; (2) a set of tools that are useful for various aspects of coding—you might, controversially, think of this as the “cognitive capabilities” that have been embedded in the coding agent; and (3) a set of methods for limiting the coding agent from just running wild with inefficient code production—you might, if you so desire, imagine this as a deliberate means of creating a limited “working memory” for the coding agent.

Again, broad brushstrokes. The upshot is that you get complicated diagrams that are pretty damn interesting, such as this one (which I’ve modified with my provocative cognitive-science overlays):

There are a few things to point out here. First, note just how much engineering is going on in the background to each prompt, work that is largely invisible to the user. Remember Alison Gopnik describing LLMs as stone soup, meaning, involving a huge amount of human labor that we misattribute to the ingenious qualities of AI itself? Coding agents seem very congruent with that metaphorical story.

Second, while the core LLM remains the method for calling up various coding-specific tools, you can see here that the majority of resources attached to the “prompt assembly” are not LLM-specific. Indeed, Raschka offers a provocative thought related to this point: “Since, in my view, the vanilla versions of LLMs nowadays have very similar capabilities, the harness can often be the distinguishing factor that makes one [coding agent] work better than another. This is speculative, but I suspect that if we dropped one of the latest, most capable open-weight LLMs…into a similar harness, it could likely perform on par with [powerful versions of] Codex or Claude Code.”

Well! Quite a bit of the AI economy, which is to say the entire economy, may be riding on the ultimate answer to this speculative hypothesis. Because if it’s really true that core LLMs have commodified to the point that even open-weight models can function on par with those from the Big Tech hyperscalers, then the frontier of AI is no longer really about advancing toward artificial general intelligence, it’s about engineering more effective means of harnessing the existing technology. It’s about developing more specialized AI, in other words. This explains why OpenAI, having made bet big on creating one giant do-everything LLM tool, suddenly appears vulnerable to Anthropic’s focus on coding and “enterprise” (business) facing tools.

But whatever Big Tech company ends up capturing the market for computer coding, the bigger question that looms is simply this: How many domains of human activity can be “harnessed” in similar ways? I find it genuinely impressive that AI engineers have figured out how to nestle the deterministic process of code compiling inside the probabilistic next-token prediction of large-language models. But how many other realms of human behavior are susceptible to this?

Consider, oh I don’t know, teaching and learning. It’s impossible to imagine creating an “education harness” that looks anything remotely like a coding harness—for one thing, you can’t repeatedly run recursive loops on little kids, they’ll get annoyed quickly. My suspicion is that “deterministic” tasks, the sorts of things that can be summarized by spreadsheets (such as accounting), will be more vulnerable to agentic AI intrusion, but most activities aren’t like coding or accounting. In fact, when I asked Raschka what domains he beyond coding he could imagine harnessing, he said he currently uses a similar process to organize his notetaking. That’s useful to some degree, no doubt, but hardly the stuff of economic transformation along the lines of, you know, electricity and the steam engine and whatnot.

Of course, I recognize that the process for harnessing coding agents need not portend the future for all AI development. There remains the more “classic” version of agentic AI, whereby a model is free to call up various tools without hard restrictions limiting the scope of its activities. But to make that fly, you have to place an aaaaaaaaaaawful lot of trust that the model will do what you want it to do (properly book a flight somewhere, say) versus what you don’t (sell off all your financial holdings and invest in the proceeds in Trump digital trading cards). And right now, people generally do not even like AI, much less trust it.

And why should they? Remember, Anthropic’s CEO has promised us “Powerful AI” that will deliver us from death itself, yet somehow ‘lil ‘ol human error continues to plague their own cybersecurity, given the leak of Claude Code’s code harness last week. What’s more, it appears that said harness may largely be crap code, at least according to this fun analysis from software developer Denis Stetskov. Among other things, Stetskov notes that the leaked code contained hardcoded words to pattern match for user frustration (“wtf/shit/fuck/horrible/awful/terrible”), rather than using the Anthropic LLM to interpret the natural-language input of users. “That’s like a trucking company using horses to haul parts,” Stetskov quotes one critic noting, before he himself highlights the paradox of Anthropic mindlessly using AI to code everything it commercially deploys:

“This is the ouroboros. The snake eating its own tail. AI writes the code. AI reviews the code. AI checks the deployment. When it breaks, the answer is more AI. The loop has no exit condition….The company that sells AI coding tools cannot build a quality product with its own AI coding tools.”

The loop has no exit condition, the harrowing mantra for our times. Wtf/shit/fuck/horrible/awful/terrible.

Subscribe now

Well, it’s time for my annual tradition of heading to the ed-tech Death Star that is the ASU+GSV conference in San Diego. I’ll be speaking on Monday 4/13 at 9am regarding “What does the AI bubble mean for education?” (details here) and on Wednesday 4/15 at 11am on whether “AI is Amplifying Cognition or Dismantling It?” (details here). If you’ll be there please swing by, and if you’d like to meet to take a break from the relentlessly annoying AI enthusiasm, hit me up at at info@cognitiveresonance.net.

Well, this is quite the treat. At the beginning of my intellectual exploration of generative AI and large-language models, I happened upon the work of Melanie Mitchell at the Santa Fe Institute. If you aren’t familiar, she’s been thinking about and researching AI and human cognition for the past four decades, and she makes deliberate effort to translate complex ideas to lay audiences. As such, finding her articles and books proved invaluable in helping me to build a mental model of these strange tools.

We’ve exchanged light correspondence over the years—more than a few of you found my Substack via her recommendation—and to my delight, she recently agreed to let me interview her for this newsletter. We start by discussing how she’s processing the current wave of AI enthusiasm, then move to a wonky discussion of an AI test we both used to like (and now have mixed feelings about), briefly touch on the unique structure of the Santa Fe Institute, and then conclude with a mild disagreement over Oliver Sacks and quick thoughts on “AI agents.”

Benjamin Riley: You have been thinking about AI for a long time. You’ve been through the ebbs and flows of excitement around it, things like the “AI winter,” and obviously interest in AI has exploded in the last several years—for better or worse. I’m curious, how has all this been for you, emotionally? Has it been awesome and uplifting, or really frustrating, or somewhere in between?

Melanie Mitchell: It’s been surprising, in many ways, how fast this stuff has taken off. People who go way back [in AI research] are a little bemused to see the amount of interest, and how many people now want to talk to us.

It’s also disturbing, because a lot of the stuff that people were worrying about a long time ago have indeed come to pass. There are die-hard AI haters, but I’m not one of them. I work in AI and I’m fascinated by it. I think it’s an incredible way to understand intelligence. But there’s a lot of downsides to what’s going on right now, and I worry quite a bit about that.

When you say that there are these things that people were worried about prior to this current moment, what do you specifically have in mind?

Joseph Weizenbaum wrote quite a bit about his worries about AI back in the 60s and 70s, and he was worried about people anthropomorphizing these systems and trusting them more than they should be trusted, giving up control to them in very harmful ways. He worried about the cognitive debt that people talk about now, where we’re outsourcing a lot of our cognition to AI. His book, Computer Power and Human Reason, is incredible and it’s very prescient, and what the things that he worries about are all still live today.

What about the power of “Big Tech” and Silicon Valley? It feels like they have become more powerful than the nation-state in some ways. Was that on your radar back in the day? Or did the concerns stay more in the theoretical realm?

In the older decades prior to generative AI, most of AI was funded by the military, at least in the US. That was the main worry of people working in the field, that their work would be used for autonomous weapons and other uses in warfare. That’s come back into the fore recently.

But I don’t think there was as much worry about the concentration of power in big companies so much, just because there wasn’t a lot to commercialize yet, right? There was some. We saw things like IBM’s Watson. And Google, they very much have been kind of an AI company from the very beginning, but they were never making that much money on AI until recently.

That’s super interesting. I’m still curious, though, now that people want to talk to you more frequently, whether that’s exciting for you? Or is it stressful because there’s so much confusion about how AI functions?

It’s some of both. It’s fun to give a big talk at NeurIPS, and to be in the public eye to some extent, but also it’s frustrating in many ways. One is that it’s just overwhelming, I have to say no to a lot of stuff. And it’s also frustrating to see some of the bigger voices, the more famous people…some of what they’re saying is, to me, just wrong. But it gets much more play, and it’s hard to break through.

Are we thinking of Geoffrey Hinton here? Are we going to name names?

There are many people who are pushing that AI systems are sentient, and that they’re devious and they’re scheming. People are using all this anthropomorphic language to describe their behavior, and politicians are kind of believing them and making policy around that.

It’s interesting, I’ve had this debate with a friend of mine that someone such as Eliezer Yudkowsky is not someone taken seriously in cognitive-science circles. At the same time, he’s prominent and his books get attention. So it does seem like there’s different levels of debate taking place around AI, and the intellectual one seems relatively small compared to what people find sexy in the broader sphere of ideas.

Yes, I think we’ve all been primed by science fiction to take those kinds of [extreme] scenarios more seriously than we should perhaps. But it’s very polarized right now in the AI world, what people believe and what they’re predicting, and it’s just shows kind of how uncertain things are.

What are the poles you see?

Well obviously there’s the “doom” pole and there’s the “abundance/utopia” one as well. Then there’s question of what kind of risks should we focus on, more near term or more speculative? Take something like human-level intelligence: Do we need to just scale up what we have now, or is there some new thing that we need to find? What are the likely timelines, and what are the likely effects on society? People are very much disagreeing on all of these things.

Well that brings us to the theoretical question I want to pose to you. I recently wrote about a new paper that Yann LeCunn co-authored that argues we need specialized intelligence, and that humans aren’t all that generally intelligent.

And I’ve been wondering where you are with this. For a long time, you’ve been talking about the power of analogies as a means for applying abstract ideas to novel situations. Just in the last several years, we’ve seen new capabilities be sort of grafted into large-language models in interesting ways.

So as you sit here today, do you think these models are our best bet for getting to something that looks like human intelligence? Or are you still examining and probing them to find both their promise and their pitfalls?

That’s a great question. I agree with Yann that in many cases, we humans aren’t that general with our intelligence. We have certain things that we’re good at, and our generality, I believe, comes from more of our social embedding than it does from our individual intelligences.

And I also agree that you need something like a “world model,” or collection of world models, to make sense of the world and deal with novelty. Analogy is very much part of that. I don’t think large language models yet have that kind of capacity.

There’s also an embodiment camp that says you can’t be intelligent without having embodiment in the world. I’m agnostic on that, but I think that embodiment certainly makes learning more efficient. We humans are driven towards causal understanding as opposed to associative modeling of the world. And that’s maybe a drive that evolution programmed into us, but we gain that understanding by intervening in the world, not just passively learning from it, right?

To this embodiment question, I keep thinking that life itself might be essential for intelligence, the need to preserve oneself and therefore having to be able to adapt. We might need to be motivated to learn to preserve our own survival.

But back to analogies. For a long time, you’ve been working with the Abstract Reasoning Corpus (ARC) test developed by Francois Chollet. [Editor’s Note: As initially conceived, the ARC tests presents little reasoning puzzles in the form you’ll see below, using grids of colored shapes. The puzzles are meant to be solved by humans and AI alike.]

I used to point to the ARC test frequently because it demonstrated that AI tools could not match human cognitive capability. But then AI results improved, and so Chollet created ARC 2, and now even ARC 3 which is more game-like. It’s been hard to track where things are.

What’s more, you’ve been conducting research to unpack that even when an AI model is solving an ARC puzzle, it’s doing so in an uncool way—meaning, it’s not actually getting at what ARC was originally designed to assess. There’s a construct validity issue.

So how do you feel about ARC today? Is it still a useful test, or is it becoming more problematic than helpful?

There’s a famous maxim called Goodhart’s Law: If a measure becomes a target, it ceases to be a good measure. And that’s happened to ARC. Chollet’s original paper was called On the Measure of Intelligence. So ARC was a measure, but then it became a target. And once it becomes a target, with a large monetary prize and prestige attached to models that perform well, then people lose sight of what it was originally meant for.

As Chollet noted in his paper, ARC is mean to test human-like reasoning using human-like priors, where the priors are things like “core knowledge” concepts.1 But it turns out AI models can solve these tasks by a kind of brute force.

For that reason, ARC in a way has lost its usefulness, unless we go back and embrace what it was actually meant to test. We should disallow what’s been allowed, which is training AI systems on these test tasks, or doing a huge amount of test-time compute that is essentially training on lots of these tasks. That’s been a little disappointing to me.

Sometimes I worry if I am shifting my own goalposts. To play devil’s advocate, there’s an argument that if AI models can get there by brute-force mechanism, that’s sufficient to show they are as capable as humans are. That’s very debatable of course, but if we presume costs and time to compute will come down over time, some say all that matters is whether the models cash out in performance. I think we both disagree with that, but I do get the argument.

I understand that, and in some cases, the brute-force approach doesn’t necessarily matter. Like, for instance, protein folding. It’s a very non-human approach, it’s not using the laws of physics, it’s really just a kind of brute-force comparison between one protein structure and a huge number of stored structures. And if it works, that’s great.

But that’s not what ARC is trying to test. Obviously we don’t care if a machine can solve ARC in and of itself, it’s meant to be a proxy for other real-world abilities that humans carry out. So we need the concepts to be aligned, and we need AI models that can deal with the world without huge amounts of test-time computation and training.

You said something briefly a moment ago about the social aspect of human learning. Have you spent much time thinking about cultural evolution and the ways in which cognitive architecture might be shaped through cultural practices?

So this is not my area of expertise, but it’s an interesting topic and I think that we are very much shaped by our culture. People talk about embodiment where we’re shaped by our bodies, our cognition is shaped by our bodies, but we’re also shaped by our cultural embodiment. And they’re kind of invisible processes to us, but they’re really important.

I completely agree. I’m a huge fan of Celia Heyes at Oxford and her cognitive gadgets theory, and I’m told she’s writing a new book on this, which I really hope is true. Because I think that set of ideas around how culture shapes some of the most fundamentally basic things about our cognition is not well understood.

Yes, and there’s another dimension, which is how we outsource thinking to artifacts. We use writing and computers, and we interact with these artifacts and they affect our thinking. When writing was invented, people were saying this is going to ruin everyone’s memory, and maybe it did. And then when calculators were invented, people worried kids weren’t going to be able to do arithmetic anymore. I think every new technology brings these kinds of fears.

But the question is whether AI, which seems more overarching than just a calculator, will have a more dramatic impact. Or in 50 years, we will look back and say, “oh we were so worried about it, and that was just a dumb thing to worry about”?

Well I don’t know if you were referring to Plato when it comes to worries about writing, but the story is more complicated than many realize! And I sometimes wonder whether calculators have been beneficial as an education tool.

But I’ll use education as my transition to ask you about the Santa Fe Institute, because it’s an unusual institution and you have an unusual perch. You are conducting original research, but you are also engaged in the transmission of ideas—your Substack of course, and also the wonderful podcast series you did around AI.

Do you feel like your efforts are part of the ethos of SFI, or is that more a Melanie Mitchell thing?

It’s more me. I’m driven to do that for whatever reason, not everybody is devoted to science communication. But I do feel it is in some sense our obligation as scientists to communicate our work to the public. The public ultimately funds our work, and they will be the “consumers” of it in some way or another. So it’s important that they understand it, and that they also understand the scientific process.

I don’t think people understand that process very well, which is why we’re now seeing so much skepticism.

Do you think that’s the problem, that people don’t understand the scientific process?

I think people don’t get the role of uncertainty. People were very angry back during covid, when scientists were saying “you don’t need masks” but then “oh, wait, you do need masks.” Scientists changed their minds. And people thought, well, that just means they don’t know anything. And people also don’t know who to trust.

So I think it’s really important to communicate as much as possible.

I think that it’s not just that people don’t know who to trust, but that there’s different forms of trusting that are now taking place, largely driven by social media and digital ecosystems. Demonstrating proof of your affinity to groups with shared identities becomes the way in which truth is established. The scientific method is still alive, but as a piece in the larger socio-cultural puzzle, it feels to me like it’s getting smaller.

From the outside looking in, it seems like the emphasis at Santa Fe Institute on “complexity” is a helpful move away from reductionist models of science, and taking a broader ecosystemic perspective. Do you think that’s been vital for you in your work?

Yes absolutely. Intelligence is a kind of complex system, it’s not just a brain in a vat, which is often how AI treats intelligence. Also, much of AI is focused on presenting models as individual intelligences, but I think complex systems gives us alternative ways to view all of this. And the interdisciplinary interaction is super interesting at Santa Fe Institute. There’s people like me with a background in computer science interacting with psychologists, neuroscientists, philosophers, and what have you. It’s incredibly enriching.

It seems cool! And sometimes I fantasize about trying to recreate the Santa Fe Institute, but just on cognition, and have that be the complexity unto itself.

Speaking of complex, let’s talk about Oliver Sacks. We had an interesting dialogue on social media about him in the wake of Rachel Aviv’s story in The New Yorker. And many were reposting the story and saying “oh my, Sacks was falsifying his data, we need to cancel him at least as far as being a researcher goes.”

But when I read the story, I found it more nuanced than that. I am biased here—I am a big fan of Sacks, who I think was an extraordinary person, albeit flawed in interesting ways. But the fact that he was intermingling facts from his own life into his narratives did not traumatize me. And then Vaughan Bell, a clinical psychologist in the UK, pointed out that clinicians who publish case studies have to intermingle false information so as to protect the anonymity of their subjects.

So this raises some deep questions around what we expect of certain forms of “science,” and where different standards may apply. I’m curious what your view is of Sacks, whether you think we should no longer credit what he’s written or whether—dare I say—it’s more complex than that?

Well, what you quoted from Bell, I don’t find that convincing at all. De-identifying people, changing their name or where they’re from is one thing, but adding details that never happened but that were key to the narrative, that’s totally different. It’s hard to write down rules for how you should do this. It’s very context dependent.

But I was very disappointed because, reading Oliver Sacks books growing up was one of the things that got me interested in the mind and intelligence. But some things that really stayed with me from his books, according to this article in The New Yorker, turned out to be sort of fiction that he was presenting as fact. So I feel like there’s a difference between the kinds of fictions that Bell was talking about and the kinds of fictions that really changed the whole story.

Maybe so, maybe so. Bell also pointed to a New York Times article that argues that clinicians have to do more than just change the name, because if the story tracks too closely to the subject, they will know it’s about them. But I take your point.

Interestingly, this is what AI calls into question, right? How do we know nowadays what is the product of an actual experience in the world, versus what’s just been generated?

And I think about this vis-à-vis the role of fiction. I think one of the most brilliant cognitive scientists of the last 200 years was Jorge Luis Borges. And his work is fiction and labeled as such. But what he’s talking about, to me it’s in the same range with Oliver Sacks, those books sit side by side on my bookshelf in terms of exploring the psyche. If Sacks had labeled his work fiction and written it as a fictional novel, the same insights into the human condition would still be there.

Maybe, but I think it’s still dishonest to publish it as nonfiction.

Fair enough. Ok, we’re almost out of time, so here’s an absolutely impossible question to end on—can you describe a mental model of what we should be thinking about when we think about AI agents, in four minutes or less? Go!

People use that word agent differently. Unlike chatbots that just output language, I think of AI agents as outputting actual actions, meaning they can go out on the internet and do things for you, like fill in web forms, charge things to your credit card…

…delete all your email accounts…

Yes, that too. And so that’s going beyond just outputting language or images. People sometimes talk about “coding agents” like Claude code, which output code but also they can run the code. They can go out to the internet and grab libraries and do all kinds of stuff. I’m not totally sure that’s the same thing as an agent that goes out and buys things for you, though.

That’s a good point. Maybe that’s why my mental model for “agents” has dissolved, because people are talking about the coding apps as agents. And that’s not what I thought we were talking about with agents as recently as just six months ago.

I think the term is a little overused. The word agent is also too closely connected to the word agency, which is something entirely different.

Yes, and ironically in the law an agent is someone or something that has very little agency, because it’s controlled by a principal!

We’re out of time, but thank you again for joining me today, and for the intellectual kindness you’ve shown toward my efforts. It’s been important and helpful—I’m grateful.

Happy to do it!

My thanks again to Melanie Mitchell for sharing her thoughts with me, and for being such a thoughtful thinking human!

Subscribe now

Contention: Whether you find our current “AI moment” at least somewhat intellectually stimulating depends largely (if not entirely) on your comfort with metaphors.

I know this will bother some of my more rabid anti-AI readers, but for all many misgivings about the role of AI in society at present, I am firmly and unapologetically curious about this new technology. More than curious, really. I am intellectually excited by the profound questions that AI poses regarding what we mean by intelligence, cognition, learning, abstraction, understanding—the list is long. What’s more, AI demands that we—and by we, I very much mean we humans—really grapple with the metaphors we use to make sense of ourselves, and those we use to help us understand these strange new tools we’ve created.

Recently, I wrote a mini-defense of the metaphor of “stochastic parrots” as an accurate and useful metaphorical description of the core function of generative AI in the form of large-language models. That’s a claim I’m making about generative AI as it exists today. But what about where things are headed? What metaphor should we use as our “north star”—hey look, another metaphor—to guide where this technology should be going? That’s the question I want to explore this week, using an interesting pre-print article published just a few weeks ago as my intellectual foil.

The article: AI Must Embrace Specialization via Superhuman Adaptable Intelligence, lead authored by Judah Goldfelder (currently a PhD student in the “Creative Machines Lab” at Columbia University), with Philip Wyder, Yann LeCun, and Ravid Schwartz-Ziv listed as co-authors—longtime Cognitive Resonance readers may remember my “AI insider spills secrets!” essay that featured LeCun’s many criticisms of LLMs as currently constructed. In any event, this new article is very readable, not overly technical, and offers up a very useful landscape graphic of the many metaphors and definitions hovering around the aspirations of the AI research field at present. To wit (and please pinch to zoom on mobile, sorry it’s so small):

Stefon voice: Adaptive generalists, cognitive mirrors, economic engines, this landscape HAS IT ALL! Jokes aside, I think it’s very helpful summary of the competing “metaphorical visions” within the AI research world about what exactly the goal of the field should be oriented around. On the Y axis, you’ve got the split between those who want to see AI models cash out in the real world on specific tasks (performance) versus those who want to develop AI models that can adapt to new, unforeseen circumstances (“learn” or adaptability).1 Then on the X axis, you’ve got the scope of tasks that AI models might reasonably address, ranging from literally anything to the somewhat more narrowly defined set of things humans are capable of doing.

Reader, I debated spending the next 1,200 words unpacking each of these “north star” definitions of AI, but I’ll forego that for now—if you’re insatiably curious about any of them, hit me up in the comments, and I’ll do my best to provide a pithy summary. Instead, I want to focus here just on “superhuman adaptable intelligence,” the new term offered by Goldfelder et al as to where the field of AI ought to be headed. More specifically, I want to unpack their definition of “SAI” because I both agree with the general thrust of their argument and think it rests on a somewhat myopic understanding of human behaviors and activities.

So what is “superhuman adaptable intelligence”? It’s “intelligence that can learn to exceed humans at anything important we can do, and that can fill in the skill gaps where humans are incapable.” Note my emphasis on learning: At heart, the argument here is that we should give up on trying to create a singular and generally superintelligent (godlike?) AI model, and instead focus on developing more specialized AI systems that can adapt and improve on specific tasks over time. I won’t go into all the technical details here, but to summarize the argument in a sentence, “the AI that folds our proteins should not be the AI that folds our laundry!”

I’m nodding, I’m nodding. And I particularly appreciate the brief burn of large-language models nestled late in the paper: “Homogeneity kills research. Autoregressive large-language models have become the dominant architecture in the state-of-the-art ‘general’ AI space…[but] they have many flaws; their errors diverge exponentially with prediction length, [making] long-horizon interaction brittle.” Here, Goldfelder et al are echoing something Gary Marcus has been arguing for several years now, namely, that for all their impressive linguistic capabilities and the billions (trillions?) being spent on them, LLMs are largely a detour on the road to developing more advanced AI models and systems.

So on the whole, I’m a fan of the “direction of travel” for future AI research that’s offered here. At the same time, and me being me, I am compelled to highlight one of the more amusing passages contained in this paper that, to my reading, is indicative of a significant blind spot that plagues the AI research community, if not all of cognitive science. It arises in the context of their argument that human beings are not actually “generally intelligent” or all that cognitively capable, and it (yet again) involves chess:

Magnus Carlsen is widely regarded as the greatest chess player of all time, and as such represents the pinnacle of human adaptation when it comes to playing chess. But this begs the question: Is Magnus actually any good at chess?

When compared with the best computers, the answer is clearly no. Even more damning is that with modern day computers, creating a program that plays chess at a much higher level than Magnus is not particularly difficult.

Our perception of his ability is colored by the limitations of humanity. Humans in general are bad at chess; Magnus is much better than most humans. The conclusion, that Magnus is good at chess, is a perfect illustration of our own human centric biases. Magnus Carlsen is not objectively good at chess, he is good at chess with respect to human performance levels.

By any objective metric, playing chess at a much higher level is not difficult from a computational perspective, but it is something that humans are incapable of.

It is undeniably true that chess engines today are more powerful than any humans at calculating chess moves, and thus can generate “solutions” that humans are incapable of. But invoking the “objective metric” of computational prowess as so-called proof that Magnus Carlsen is “not objectively good at chess” is a category error—humans play chess because it’s fun and interesting, it’s the sport of human thinking. As is true with all human games, nothing of any real importance happens as a consequence of winning, the satisfaction instead arises from the experience itself.

What’s more, and I say this as someone who’s spent way too many hours watching Daniel King analyze games on his Power Play Chess channel (he’s great), the capabilities of chess engines are often treated as a semi-interesting sideshow to the far more absorbing endeavor of exploring human-versus-human competition. When commenting on a game, King will often observe, “the computer thinks this is the best move but good luck to any human at seeing that”—and then he’ll go right back to his thoughtful and very human analysis of the game at hand. That’s what we humans like to do, be curious about our fellow humans.2

In my view, AI as a research field, to say nothing of all of cognitive science, suffers from obsessive commitment to what my late-in-life mentor Jim March described as the “consequentialist paradigm,” the view that “our beliefs require that action be justified in terms of ambition for its consequences.” Sometimes that perspective is warranted, no doubt—but not always. Sometimes, we pursue things not because of an outcome we hope to achieve but because the pursuit itself generates meaning, in ways that are entirely subjective, often ephemeral, and we might even say irrational. Why does Don Quixote tilt at windmills, March asks? Because “he embraces the foolishness of obligatory action. Justification for knight-errantry lies not in anticipation of effectiveness but in an enthusiasm for the pointless heroics of a good life. The celebration of life lies in the pleasures of pursuing the demands of duty.”3

Artists, poets, musicians, those who spends time dwelling in the depths of the humanities—and please do note the root of that word—they all naturally intuit this, I think. There is a reason that so many in the artistic community are horrified by AI, and it’s not solely because of the misappropriation of human labor and creativity (though this must not be overlooked). It’s because they see AI as part of a continued technological erosion of the celebration of life that arises from poetic pursuits. The pursuit of artificial intelligence, whether superhuman or adaptable or specialized or whatever else, could be part of the humanities, broadly construed, but only if researchers can resist slapping metrics on everything and declaring that all that matters is what’s measured. Life is not a data set!

I recur to March once more to close us out, and please do take a few minutes to ruminate on this:

We have grown to accept the modern idea that human actions, institutions, and traditions are justified by expectations of their consequences; and we have come to view optimism without hope as an unfortunate surrender to Pollyanna. But education unconditionally celebrates life. It is an arbitrary assertion of optimism. It echoes an ancient conception by which I do what I do because that is what is appropriate to my nature; and my nature is not simply a tautological summary of what I do but an understanding of the essence of my destiny, an interpretation of my history, and an assertion of my humanity.

Speaking of art and assertions of humanity, several months ago Tomáš Baránek, a publisher in the Czech Republic, reached out to me in hopes of translating my Large Language Mistake essay into Czech—there are few things more flattering to a writer than someone wanting to share it with an audience in another language (unfortunately, The Verge’s legal team put the kibosh on that plan).

Sadly, Baránek’s mother Vlasta Baránková recently passed away, but through our correspondence he shared with me her book of gorgeous and haunting artwork—she was an accomplished artist and illustrator of many childrens’ books, including some I’m sure I read as a child. Tomáš also shared with me that she had to maneuver around communist-era restrictions in order to publish her work. As he observes, “this is how real artists and creative people live and die. They cannot help themselves, they enjoy the creativity and its painfulness.”

Indeed. And you will see that creative beauty and pain reflected in the work of Vlasta Baránková, who’s work I share here with permission. She may be gone, but her exploration of the human spirit remains with us. I grieve with you, Tomáš, but I am so grateful to know of your mother and the contributions she made to our shared humanity.

Subscribe now

Last week, I appeared on the What’s the Big Idea? podcast hosted by Dan Kearney, a practicing teacher in California. Somehow, we recording this on Tuesday morning and then Dan went and taught for the rest of his day—teachers are unreal.

I’ve probably appeared on 50 or more podcasts in my lifetime and I’m grateful for each and every invitation, but if I had to pick just one that encapsulates my reverance for human cognition and the role of human teaching in fostering its development, this is the one. Dan graciously granted me wide latitude to tell Helen Keller’s cognitive story, not just the famous moment wherein she learned the word for water from her teacher Anne Sullivan, but the entire arc of Keller’s childhood cognitive experiences, her (in her words) “plunge into unconsciousness” after becoming blind and deaf, and the “awakening of her soul” through human language and dialogue. It’s a wondrous human story, and one that stands in stark contrast to the pale imitation of cognition we find in large-language models.

You can find the podcast, titled We Should Be Skeptical of AI but really covering so much more, via these channels:

• Spotify

• Buzzsprout

• Apple Podcasts

Plus, Dan’s got one of the best podcast voices you’ll hear, and he’s the one who oberves that, “as teachers we dream of the a-ha moments,” gifting me with the headline above. Indeed they do, and what an elegant testament to fostering human imagination and flourishing through dedicated pedagogical practice. By humans.

Just one more thing. Kearney’s broad range of questions allowed me to give a few shout-outs to some friends and intellectual influencers of the Cognitive Resonance project. In rough order of appearance, you’ll hear me pay my respects to:

• Michael Pershan (teacher in NYC)

• Eryk Salvaggio (artist-scholar)

• Murray Shanahan (scientist and science advisor to the film Ex Machina)

• Gary Marcus (godfather of AI Skepticism)

• Paul Cisek (neuroscientist)

• Alison Gopnik (cognitive scientist)

• Stephen Fitzpatrick (teacher in NY state)

• Chanae Bond (teacher in Texas)

So I hope y’all fire up the podcast—and if you’re pressed for time, Helen Keller’s story starts at the 18-minute mark. Thanks for reading, and potentially listening too.

Subscribe now

Here’s a peek behind the Cognitive Resonance curtain: I’m currently engaged to provide strategic advice to England’s National Institute of Teaching as it launches a new center— or centre—that, broadly speaking, aims to help educators understand and address challenges to student learning and wellbeing that are posed by digital technologies. This effort is led by Pip Sanderson, the NIoT’s director for teacher education, who I’ve worked with extensively over the past two years. She’s become a good friend.

Last week, Pip participated in an event hosted by Eton College titled Fulfilling Potential: Reframing Cultures for the Digital Age in post-16 Education, and she prepared five minutes of opening remarks for her panel. At my unexpected invitation and with her permission, I’m now sharing her speech with you.

As you’ll soon read, Sanderson offers powerful testimony regarding the role schools and teachers should be play in fostering shared knowledge and understanding as digital technologies proliferate. Full disclosure: I provided an outline for these remarks, sourced the (very real) anecdote about the boy wearing the t-shirt, and suggested the rhetorical framing that appears in the final question.

Apart from that light involvement, however, these are entirely Pip Sanderson’s words. I hope everyone working in education systems, in whatever capacity, takes heed of what she says here.

A lot of today is about the future. But I’d like to start by looking backwards.

Across history, when our information systems change, we change: psychologically, socially, emotionally—and in profound ways. The invention of writing did this. The invention of the printing press did this. And over the last 30 years, digital technologies have done this, and are doing this still.

As Naomi Alderman puts it, we are living through a Third Information Crisis. Not a moment, but an epoch. We’re already in the middle of it, and it will shape the rest of our lives. Naming it matters, because it helps us see this isn’t just about new tools.

Instead, it’s a systemic shift in how knowledge itself is formed and trusted. And that has huge consequences for us all, and the very notion of democracy. We are all navigating a radically altered information environment.

But some things remain the same. From cognitive science, we know something simple but powerful: we understand new ideas using the knowledge we already have. And building that knowledge takes effort.

So we know we must protect against the temptation of cognitive offloading that new technologies offer us. We must preserve the cognitive grapple that leads to real knowledge.

We also must nurture our own identities, our young people’s identities and our social norms, and—just as crucially—our sense of what feels believable against the onslaught of digital information. They don’t just change what we know.

They change how we decide what is worth believing at all.

Eliot Higgins, founder of Bellingcat, describes this as “epistemic collapse”: a breakdown in the processes that allow societies to establish shared truth, reason together, and hold power to account. In other words, a breakdown in verification, deliberation, and accountability.1 And because those processes aren’t innate, because they’re learned behaviours, we may have taken them for granted as we move from one information age to another.

Let me root this back into schools, with an example someone recently shared online. It’s a non‑uniform day. A boy arrives wearing a t‑shirt that reads: “Male by birth. Alpha by choice.”

That slogan comes from an online ecosystem that rewards certainty and identity signalling, often by framing the world as a struggle between winners and losers. That moment is one example that captures what teachers are telling us. It’s not just that pupils believe the “wrong” things and treat others differently or poorly because of what they’ve absorbed online. It’s growing cynicism about whether any source of knowledge deserves any trust at all.

And teachers are left feeling deeply unsure about how to respond. How would you expect teachers to respond to the boy in that t-shirt?

Now crucially, we all may share extreme or misleading ideas, adults and children alike, not because we fully believe them, but because they signal something to others, be it belonging, humour, outrage, or identity. From a socio‑psychological perspective, that’s just being human.

But what’s changed is the scale and speed. Ideas that once sat at the margins are now being algorithmically rewarded, normalised, and of course monetised. And that is rapidly eroding shared ways of knowing and understanding.

That’s why this moment, and what we do in response, is better understood as a crisis of epistemic infrastructure. If schools and colleges are to build positive, connected cultures in a rapidly evolving digital and AI‑influenced world, where our epistemic norms are crumbling, we have to start with how information actually functions in social groups: how identity, relationships, and status shape what young people trust and share.

So I think our greatest endeavour in the next few years is to work out how we create classroom cultures on mass scale where verification, deliberation, and accountability are normal; where pupils learn not just what to know, but how knowledge is built, challenged, and revised.

There is some good news here.

Even in the highly fragmented information environment we live in, pupils consistently report that they trust their teachers more than almost any other authority. That gives schools and teachers a unique role, and it’s not to be the arbiters of truth in every debate, but rather to serve as places where young people learn the habits that make shared knowledge possible. So I’ll conclude with two questions:

• What durable knowledge and habits can teachers draw on as technologies change faster than curricula ever can?

• And how do we replace what can feel like cognitive anarchy with something closer to cognitive solidarity?

You must admit that’s one hell of a speech! My thanks again to Pip Sanderson for agreeing to share it here, and prepare to read more about this notion of cognitive solidarity in the weeks ahead.

But not right away, as I suddenly find my professional cup overflowing with projects—file this under “happy problems to have.” As such, I’m going to be dialing back the frequency of my essays. You’re devastated, I know. But if you have thoughts to share via a guest post related to some medley of cognitive science, generative AI, and education, now’s a good time to reach out—my email is info at cognitiveresonance dot net.

Thanks for being part of this strange but satisfying Cognitive Resonance endeavor.

Subscribe now

Over the weekend, my friend Greg Toppo of The74 asked me to comment on Google's new hijacking of—er, I mean new partnership with—ISTE+ASCD, an umbrella ed-tech organization, that willl purportedly train every teacher and higher ed faculty in the US to become "AI ready." Toppo’s comprehensive story is here, and I’m quoted at some length.

But, not enough length! With Toppo’s understanding, I am now share with you the full context of my remarks. I have met the enemy, and it is ISTE+ACSD. See you soon-ish at ASU+GSV, Richard, let’s see if you can look me in the eyes!

It is completely unsurprising, though no less infuriating, that ISTE+ACSD continues to double-down on its leading role as shill for Big Tech.1 I’m sure it’s nice to unlock Google’s funding, but what so profit an allegedly nonprofit education organization if it shall loseth its soul? All the while doing damage to teacher effectiveness and student learning in the process.

But that of course presumes there’s any soul at ISTE that’s left to be lost, which seems unlikely. Richard Culatta, who worked in the Obama Administration and is someone I once respected, seems blissfully unaware that ‘AI literacy’—a stupid and largely incoherent term, but surely one that if it has any meaning suggests a modicum of critical thinking—might result in teachers resisting these tools of cognitive automation. To be sure, ISTE says resistance is futile, because “the future” will demand “AI readiness.” I’d credit their clairvoyance just a tad more if ISTE had not claimed, as recently as just a few years ago, that “the future” would also demand that everyone learn to code.2

Oops!

If we can bracket the odiousness of ISTE’s grift, I admit I’m fascinated to see the major Big Tech companies competing so vigorously to control “the education market.” OpenAI is giving away their premium model to teachers (until they won’t), and now Google is doing whatever this is.3 Of course, Google is also the company that once ran ads suggesting children should use AI to outsource the drafting of their letters to their heroes, so I confess I’m just a tad skeptical of their commitment to “student agency” and “critical thinking” nowadays. As a company, they’ve tried to erode the former, and shown little collective capacity for the latter.4

I also must briefly note the contradictory, and fundamentally very stupid, paradox sitting at the center of the AI hype right now. On the one hand, we are told AI models are on the the verge of ravaging the entire global economy and will soon destroy the need for knowledge work of any sort. That’s bullshit of course, but if it were true, then why in the world would teachers need any training to understand how to “use AI responsibly”? For that matter, why would we even need teachers at all, if AI will obviate the need for any human thought expended toward any productive enterprise? The robots will do all our work for us, as we become robots ourselves.5

The good news is that all of this will end up being a waste of teachers’ time and Google’s money and ISTE’s relevance. That’s my prediction about “the future,” and I’m confident in it because I know a thing or two about the science of learning. Human beings have evolved to learn from each other in the context of our relationships. This is the superpower of our species, and the kids who’ve grown up in the past 20 years are increasingly disgusted by what tech has done to them personally, and society more broadly. They are not happy about the world we’ve given them, and their voices are growing ever louder. There’s no hope that Google or ISTE will listen to them, of course.

But their teachers will.

Subscribe now

Here is an overly simplistic yet defensible story of how large-language models do what they do:

Training phase

1. A model is built by feeding it strings of text. The back end of these strings are hidden from the model. For example, it might be fed “Four score and ______,” with the word “seven” deleted from what it can perceive.

2. The model makes a guess as to what word comes next. If this is time zero in training, meaning, if this is literally the first piece of text the model has ever been fed, its guess will be completely random.

3. Every time the model makes a wrong guess, it makes a note in its statistical word ledger, a little database that it manages. And then it keeps guessing.

4. When it finally correctly guesses the next word should be “seven,” the word that’s been hidden from it, it makes another note the ledger.

5. Now it’s fed the next piece of text in its training. For example, “Four score and seven ____”, with the word “years” deleted from its view. It repeats the same process it used before, but this time, its guess will be almost but not completely random, because of the tiny adjustments it made to its statistical ledger a moment earlier. The model will keep guessing until it finally hits on “years.”

6. Repeat this process for every piece of text that has ever been digitally encoded in all of human history. The model keeps adjusting its ledger until the humans building and testing it decide the output is generally acceptable. The model has been trained. [See footnote if you’re ginning up to debate this point in the comments. I urge you to chill.]1

Commercial deployment

7. You, the human user of the large-language model that we just trained, sit down and feed some text to it. This is your prompt.

8. This text may be identical to text the model has been fed before. For example, you might type, “Four score and seven ___.”

9. Here’s where it gets a little weird. Despite how it may appear, there is no single objective response the model can provide to this inquiry. It might plausibly respond with text that corresponds with we humans know to be the continuation of Gettysburg Address, such as “years ago.” But it might produce the entire speech, or perhaps the number 87, that would make sense too. And who knows, it might even add an unexpected bonus, such as “Lincoln had bars” plus a little hat emoji and US in smallcaps.

10. In other words, whatever text the model produces in response to your human input is a statistical guess as to what should come next, but what should come next is not objective. It’s a guess, but an informed and probabilistic one.

11. You, the human user, can also feed the model completely novel text that is unlikely to sync up with any specific text string it was fed during training. For example, you might type, “What is a splurge minus a gibbons?”

12. You’d think this splurge-gibbons prompt would be more problematic for the model, but wrong, the model is completely indifferent. That’s because it still does what it always does, which is take the text you’ve fed into it, consult its statistical word ledger, and then—this remains key—makes a probabilistic guess as to what text should follow.

    

13. Once the model has been commercially deployed in this fashion, there is no objective measure that can be used to universally adjudicate the quality of its output. None.

14. Instead, as a practical matter, what’s important is simply this: do you, the human user, find the model’s responses useful in some way?

Got all that? Sound vaguely familiar? Good, hold that thought, let’s pivot to some etymology.

The word stochastic originates in Ancient Greek, στόχος (stókhos) meaning “aim, or guess.” In its initial usage, stochastic was used as an adjective to describe a conjecture of some sort, but it has since evolved to describe something random in the particular yet quasi-predictable in the aggregate. A statistical term. Imagine a starting poker hand in Texas hold ‘em: whether you’re dealt pocket aces or 67 (six seven) is random for any one hand, but run repeated trials of the same experiment—that is, keep dealing the cards—and you will arrive at definable distribution. You’ll know the general probability, in other words.

As a word, stochastic really took off in the 1950s, right when computers and cognitive science were beginning their Renaissance—not a coincidence. Apparently music also played a role in its uptick in usage, for reasons I don’t understand. In any event, here is the data, and I’d bet cash money that use of the word stochastic has rocketed up more recently:

Now it’s time for one final pivot, to ornithology—specifically, parrots!

You know what a parrot is, you know what it means to “parrot something,” I trust I need not elaborate. But I’ll now share with you the strange fact that the only pet I’ve owned as an adult was a Senegalese parrot named Chiku, Swahili for “talks a lot.” In reality, Chiku did not talk a lot—the only words she (?) ever mimicked were “Chiku bird”—but she sure did shriek, every morning at the crack of dawn. That, combined with Chiku’s habit of biting my then-girlfriend in the face, truncated our time together in the mid-2000s, and I’d like to think she is still living happily with the guy in San Jose who purchased her (Chiku that is, not my ex-girlfriend.)

Despite that ordeal, I still think parrots are incredible creatures—the African Grey is my favorite—but I hope we can all agree that while they are amazingly capable of learning to mimic human language, parrots obviously do not understand what words mean the way that we humans do. [See footnote if you are contemplating mounting an impassioned argument that birds are smarter than us—they are not.]2

Stochastic parrot. Arguably the most famous metaphor used to describe what a large-language model is, despite so much metaphor-versus-metaphor competition that there’s a push to create an AI metaphor observatory (please hire me). The term comes from this famous paper published in 2021 and co-authored by Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and confusingly, Shmargaret Shmitchell. Among other things, the article resulted in Gebru being fired from her job co-leading Google’s AI ethics team, and years later Sam Altman declared via tweenage tweet, “i am a stochastic parrot and so r u.”

I’m not going to get into any of that. Instead, I’ll now try to tie this essay together with a simple question that vexes so many: Are large-language models stochastic parrots? Is this a useful metaphor for describing how they do what they do?

Yes! The answer is yes, and hopefully you already see why. Stochastic is a near-perfect description of the statistical approach that’s used to both to train and deploy LLM models. And, because there is no conscious living creature nestled inside the server racks that these models need in order to function, we can describe them as, yes, parroting the text they produce.

They simply squawk.

Now for some controversy: Describing LLMs as stochastic parrots does not necessarily mean expressing disapproval of them derogatory. I think many people get tripped up here because they know that the originators of the metaphor, all of them prominent AI skeptics, meant it that way. Fine, agreed. But the cool thing about being sentient humans is that we can independently think about whether the metaphor is apt as a description of what’s happening under the hood of an LLM (yet another metaphor). Again, yes! It’s great, as close to perfect as we might find in two words for describing how these weird new things operate. It’s math, not magic.

Even more controversy: Stochastic parrots can be useful. Please recur to point #14 above. Again, we as sentient humans with agency get to decide whether the output of LLMs are useful or not. Clearly LLMs are useful to many millions of people! I say this with love to all my fellow AI skeptics out there, if you take the position that these tools are incapable of ever producing useful output, you’re not only wrong, you are implicitly suggesting that you are in better position to decide what others find useful than they are themselves.3 As democracy continues to crumble worldwide, I hope you at least reflect on this, and whether there are some dots to connect.

Controversy on top of controversy on top of controversy: Just because something is useful, that doesn’t mean it should be used. On first blush, it may seem like I’m contradicting what I just wrote in the prior paragraph, but I’m not, I swear to f’ing deity-of-your-choice that I’m not.

To demonstrate this point, allow me a digressive analogy: guns. I’ll never forget the time I was standing in line in a hunting and fishing shop here in Texas, buying a new fly rod, while the man ahead of me—who looked straight out of central casting for Mad Max: Fury Road—purchased the single biggest handgun humanity has ever produced. “I wonder who he plans to kill with that,” I thought quietly to myself. And then he told the person behind the register that six rattlesnakes had entered his house the previous weekend, and he was acquiring this new tool to solve that problem.

For Immortan Joe-Texas, that gun was useful! Justifiably needed! I can acknowledge that and argue nonetheless that we should completely abolish firearms in the United States, for reasons too many to count. But again, that involves—nay, requires—a democratic process to resolve. There’s no one right answer, both snakes and mass school shootings exist in America. We must decide what to do about them. This is the premise of the social compact, or so we hope.

The paradox hovering around large-language models, of course, is that many of their use-cases demonstrate the very reason why they should not be used. Take, oh I don’t know, how about…education! There is zero question that students find LLMs useful—half of all ChatGPT users are students, OpenAI absurdly boasts—for the obvious reason that these tools alleviate students from exerting cognitive effort they’d rather not undertake. We thus know, or at least all of us who understand the basic enterprise of education to be building knowledge through effortful thinking should know, that these tools pose extraordinary hazards and generate myriad harms towards that end. Many students even sense this. Yet, for reasons that continue to infuriate me, many educators are rushing to embrace the use of AI despite the obvious damage they are doing to actual learning. Indeed, over the weekend I had an ostensible educator and self-professed “AI evangelist” crash my mentions on BlueSky to boast that he’d persuaded his AI-skeptical students that they had to learn to use AI because of “the future.”

Don’t do this, educators! Stop it!

I’ll conclude with a restatement of what I’m contending: We need metaphors to help us understand the unfamiliar. The stochastic parrot metaphor is particularly appropriate one for understanding LLMs. These digital squawking things are obviously useful to millions of people, and AI skeptics should not argue otherwise. They, or rather we, should instead make the case that we should resist the intrusion of these tools into society broadly, and schooling in particular, precisely because of their seductive but harmful utility. The end.

Oh Chiku, I miss petting your little head!

Update and correction, February 16, 2026:

Well I said there’d be controversy on top of controversy, and boy has there been! Not long after hitting send on a BlueSky post promoting this essay, Emily Bender and Margaret Mitchell—two of the co-authors of Stochastic Parrots—responded with vehement criticism of my characterization both of them as AI skeptics and their intent with their seminal paper. I understand but disagree with them on one point, but agree with the other, and so have made the edit reflected above, which we’ll get to momentarily.

Many people don’t like the term “AI skeptic,” and to a degree I understand why—it’s an imperfect term. “Vaccine skeptics” are anti-science idiots, of course. Sadly, however, there is no perfect term to describe the broad coalition of people who are concerned about AI intrusion into society. Nonetheless, words matter, etymology matters—a la stókhos—and the classic definition of skeptic means “inquiry, doubt.” That not only feels apt, the reality is that “AI skeptic” has become the commonly used term in public discourse to describe the still all-too-small general movement to push back on what Big Tech is currently ramming down all our throats. Indeed, over a year ago, I wrote a well-received piece that tried to unpack and explore just how diverse the “AI skeptic” movement really is, pushing back on Casey Newton’s drive-by description at the time. I remember him responding to me saying, “you need to drop into my threads to see what I mean” or words to that effect, and now I see why. Congratulations to the anonymous assholes on social media I’m having fun blocking, you’ve fostered me feeling (thin) empathy for Newton. Hard to do.

In any event, should our movement coalesce around some other description, I’ll happily climb aboard, but for now and the forseeable future, I’ll continue to use AI skeptic to describe it and people I think are part of it. Now might be a good time to rewatch this scene from Life of Brian.

The other main complaint, at least the one that reached me before muting people a-plenty, has more force, namely, that I mischaracterized the nature of the argument they were making in their article. Here is the truth: recently I had a conversation with someone who likewise expressed admiration for stochastic parrots as metaphor for LLMs, and stated that to describe them as such is not necessarily—their word—pejorative. Writing this yesterday, I initially wrote exactly that, but then didn’t want to appear to be plagiarizing their specific statement, so I swapped in derogatory.

My mistake! Again, words matter, and while derogatory carries many meanings (as so many words do), I think it’s commonly understood as “slur.” So let me as clear as I can be: at no point in On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? are there any slurs. The article instead is an important—again, seminal—scholarly exploration of the myriad risks posed by LLMs, the harmful impact on society they may have, and—helpfully and very underappreciatedly (not word)—they paint a vision of what a different AI future might look like. Unfortunately, their advice has generally not been heeded, certainly not by Big Tech companies, and so here we are.

To fix this, I thought I might pop pejorative back in up there, meaning—in one usage—”to express disapproval of,” which is very much the tenor of Stochastic Parrots. But since that word too has multiple meanings and synonms—including derogatory—I’ve just written, “Describing LLMs as stochastic parrots does not necessarily mean expressing disapproval of them,” so we can hopefully all move on with our lives.

Now back to writing a speech that ends with a call for human solidarity. Social media, you are not helping!

Subscribe now

This past Saturday, I wrote a 1,500-word essay explaining why I haven’t been posting for the last month, which I’ll now summarize in a sentence: the omnipresent fascism in America is really getting to me, and thus it’s hard to focus on—or even care about—the newest AI-related developments. It was a confession in the form of an apology. After finishing the draft, I pondered whether I should even publish it, given its frankly depressing tone and my ongoing wondering about whether I should continue to use Substack as a form of therapy.

Then on Sunday—just yesterday—my beloved Seattle Seahawks (I’m a die-hard fan, didn’t you know) won the Super Bowl. Joy. Elation. Sportsball rapture. All of the emotions that flow when your team brings home a championship: that feeling of having invested one’s energy and devotion in something larger than oneself, and then feeling that pay off in a moment of pure ecstasy. This is why I’m an unapologetic football fan. Go Hawks.

As football games go, though, Super Bowl XL will not be remembered for the gameplay itself. Real NFL aficionados can appreciate a good defensive battle, but let’s be real—it makes for lousy viewing for the casual fan. At some point, my fellow Seahawks-loving friends and I started discussing whether our field-goal kicker would be crowned Most Valuable Player. That’s not an indicator of an entertaining game.

What people will remember about yesterday’s Super Bowl is Bad Bunny’s halftime performance. I am not going to pretend to be a profound cultural critic, so let me just say: holy shit, his performance was amazing. I knew that he’d made anti-ICE statements at the Grammys a few weeks ago, so he’d already won my political affection, but I had no idea what to expect from the actual spectacle that Super Bowl halftime shows have become. For that matter, I had no idea he’d be singing exclusively in Spanish.

As musical entertainment, Bad Bunny’s performance was thrilling, sexy, fun—all the things. But I’m writing about it because of its political importance. It’s only hitting me this morning that it was the first national moment of cultural pushback against the fascism of the Trump administration. Trump’s name was never mentioned; the word “ICE” was nowhere to be found. Instead, Bad Bunny delivered two simple messages: Together, we are America and The only thing more powerful than hate is love. Unity and love—two sentiments that might seem anodyne in normal contexts—were unmistakably intended as indictments of the division and cruelty sown by the fascists currently in charge of the USA.

Can I pivot from Bad Bunny to the philosopher Richard Rorty? Let me try. The opening paragraph of Rorty’s book Achieving Our Country contains a syllogism that’s haunted me for years:

“National pride is to countries what self-respect is to individuals, a necessary condition for self-improvement… Emotional involvement with one’s country—feelings of intense shame or of glowing pride aroused by various parts of its history, and by various present-day national policies—is necessary if political deliberation is to be imaginative and productive. Such deliberation will probably not occur unless pride outweighs shame.”

It’s been a rough year to take pride in America. This is no secret: I am firmly placed on “the Left” of the American political spectrum, and in many ways, I think the existential struggle for leftists like myself is to be self-aware of the many, many historical horrors that have been—or continue to be—perpetrated within one’s country, and to feel moved to remedy them. Building this consciousness, however, takes its toll. Having knowledge of injustice, and bearing witness to its ongoing manifestation, can be radically dispiriting. (That was the theme of the essay I wrote Saturday.)

But yesterday, for 15 glorious minutes wedged between my Seahawks manhandling the Patriots, I felt national pride. I feel it still this morning. Bad Bunny represents the best of the American ideal: youthful, cross-cultural, playful—again, promoting unity above division, and love rather than hate. A nation of the Indigenous and the immigrant, where human decency and self-creation can flourish. Together, We are America, with an emphasis on WE.

(And the MAGA counterprogramming to this is a flabby Kid Rock singing about the glories of statutory rape? Get the fuck out of here.)

I believe the next 12 months in America will be our most consequential since 1860, when the election of Lincoln led to our first civil war. As much as is humanly possible, I’ve tried to steel myself for the horrors to come—the murders in Minneapolis were mere prelude. The central question is whether we will have free and fair elections, whether the democratic process that we have (imperfectly) relied upon for 250 years will endure or collapse. The last year was rough; this year will be rougher.

But I love America, and I am proud to be an American. However improbably, Bad Bunny has reminded me that, in the end, we will triumph over the fascist bastards—so long as we fight for the best of American ideals. So long as we fight to achieve our country. Which means I’m chucking that depressing essay I wrote on Saturday into the digital trash can and getting back to work.

Next week: in defense of stochastic parrots!

Subscribe now

In December, I was thrilled to interview Jessica Riskin, a historian of science based at Stanford University. As you’ll read, an essay she wrote in 2023 about Alan Turing and AI played a non-trivial role in my decision to launch Cognitive Resonance, and I’ve broadly become a fan of her efforts to challenge existing scientific paradigms using the lens of history. To that end, she has a new book coming out in March, The Power of Life, that reexamines the scientific theories of Jean-Baptiste Lamarck—it’s a fun and provocative read (and available for pre-order here).

We touched on many subjects so I’ve broken the interview into six sections—feel free to skip around, but the whole conversation is worth reading!

1. Life, Intelligence, and AI

2. Determinism and Free Will

3. Eugenics and Behavorial Genetics

4. Lamarck and The Power of Life

5. Cultural Evolution

6. Fighting Authoritarianism in America

1. Life, Intelligence, and AI

[Ben] Let’s start with your intellectual project as I see it. Having read your forthcoming book The Power of Life, and your previous one The Restless Clock, as well as many of your essays, I feel in your writing a real celebration of natural life. Across your efforts, I see you resisting the reduction of life to mechanistic descriptions, both in the present and the past.

You highlight that we’ve come to think of science as this mechanistic process, but it didn’t have to be that way. Historically, it took a turn in that direction, but it wasn’t inevitable.

And thus the people who nowadays claim the mantle of science as this rational project, they miss the fact that the current approach is actually grounded in very biblical, even divine notions—God is the watchmaker who puts life into motion.

You want us to stop doing that and reembrace the complexity of humanity. I wrote down this quote from your most recent book, “life at its essence is creative agency.”

So I feel you saying that to be human is to have a will. Does this resonate?

[Jessica] That’s beautiful. I like that very much. I would add, it’s not just humans, but I think all living things. Life in its essence is creative agency, that’s Lamarck’s idea, but I really admire it and like it. I think with living beings in general, there’s been this kind of reductive mechanist approach to nature in general. It seems to be wrong headed.

So, I would also add my meta-level interest in science, what science is, and why people think certain kinds of explanations are science and others are not science. How did it get to be that this very narrow, reductive mode of explanation is the only one that people will accept?

And people even feel they have to kind of prove themselves by carrying it to crazy extremes, these kind of crazy extremist positions like Robert Sapolsky’s.

Oh we’re going to get to him. But let’s first talk about AI.

I became one of your biggest fans when I read your essay on Alan Turing and ChatGPT in the New York Review of Books in June 2023. You shared an anecdote about a conversation you had with a student who raised the idea that perhaps his thinking was no different than the pattern-matching and statistical correlations that generative AI makes.1

I think that was a career-changing moment for me, because I realized this tool was going to be used both as mirror and searchlight into the way we think about how we think, and what we are. And this made me freak out that we’re heading toward a place where we further devalue the beauty in our essential humanity, and our agency.

So I’m curious if you’ve kept in touch with that student, and how many more conversations like that have you had? It’s been a few years, what’s the vibe?

On the first question, no I haven’t kept in touch, because that was actually a student at Reed College, not Stanford. I had gone to Reed to give a lecture and two-day seminar there.

But I have had hundreds of conversations with students since then about that kind of thing. And what’s interesting is that my students seem to be getting less open to it. Certainly, the graduate students seem to be more skeptical, and feel it’s really insidious, and want to hold it at bay more and more. Which gives me hope.

If anything, I see a more entrenched view that there is an essence to human thinking that is not at all what these AIs do. Of course, nobody really knows how the AIs work, so there is a level of mystery.

I always think back to Alan Turing’s 1950 paper, where he says it would be possible to have a process of training so complex that the designers and trainers themselves wouldn’t understand how it works. And he says this would necessarily be the case because if the trainers did understand it fully, they would have no further sense that it’s intelligent. But the same goes for a human brain, if you knew everything that was going on inside, you’d have no illusion of intelligence.

So that brings us to Turning and his test, which is really the main topic of your NYRB essay. There’s something you write about that has stuck with me, which is the conversation Turing had with Max Newman on the BBC, where they were talking about what you just described. And Newman said, well, perhaps intelligence is like a mosaic: if you look at the constituent things, all you see are little bits of colored stones, but pull back and the beauty is there.

I can’t tell if you agree with that or not?

No, I don’t at all agree with that. I don’t agree with that reductive idea that if you can break it down into tiny moving parts the intelligence evaporates.

And if you think of his analogy, it’s not even true, right? The mosaic isn’t just lots of little colored dots with cement in between. It has a design. They’re arranged that way for a reason.

It often seems when people want to insist on some kind of an irreducible essence to life or cognition—or intelligence, thought, mind—that they are forced to do it in religious terms, as “the soul,” or in some dualist way.

And what’s interesting to me about Lamarck is that he absolutely rejects that. He thought it was fine if people wanted to believe in God, but it had no bearing on anything, from his perspective. He was a materialist, he thought there was only one material, natural world. And so he’s unusual in that regard.

That’s partly what I find attractive about his approach. I really do think that most reductive modes of scientific explanation are just not persuasive, and there’s no reason to believe in them. They are their own kind of faith, in fact.

It feels to me like you are in the vicinity of those who argue that we should pay more attention to cells rather than neurons. AI and cognitive science are focused on neurons an information transformation, this is where Alan Turing and Claude Shannon cut their teeth, the manipulation of symbols. But it feels that the Lamarckian view, and maybe the Riskinian view, suggests we should look at the genesis of cellular life and how it builds upon itself. Is that at all true?

I would say there is something kind of dualist in believing you can do anything in silicon that you can do in a living brain. As if the matter doesn’t matter, the substance doesn’t matter.

Why would you think that the actual substance has no bearing on whether the thing is alive or intelligent? I mean, certain electrical phenomena only happen with conductors, you wouldn’t think that you could make those things happen using pieces of wood.

So why would it be that intelligence or cognition or life could be transferable to another kind of matter? It only works if you’re a dualist on some level, and you think that there’s some immaterial essence to it.

The pushback I get on this, which I’ll now use to play devil’s advocate with you, take two forms. First, and I chuckle at this, I’ve been accused of “meat chauvinism,” that because we humans have bodies, we presume they are required for intelligence. And people point to AI and even computing more broadly as evidence that there are ways to emulate aspects of intelligent behavior in non-corporeal forms.

And then the other argument is the “wing-flapper defense” as I call it, which is that we looked at birds and how they fly and thus mistakenly tried to build mechanical wings for a long time. It was only when we turned to theories of aerodynamics and lift that, boom, the Wright brothers build airplanes. So maybe we should examine our human intelligence and develop theories about it, but try to recreate it artificially in some other manner.

What would you say in response?

Well, the meat chauvinism, that really sounds so Silicon Valley.

100%.

But that’s exactly the point I was making. That is a very dualist person! He probably thinks—just hazarding a guess it’s a he—he thinks he’s a hardcore scientific type, very rationalist and sober, but he’s really a dualist.

You could do a computer simulation of water and it won’t be wet. Why would anything be transferable into a computer simulation and be considered the same thing as whatever it’s simulating?

Now, the flying example is one answer to that. Airplanes really do fly. They aren’t simulating it. But they aren’t computer simulations either, they are heavy objects traveling through air.

I might have thought you’d point to the lack of agency in AI, that when two humans communicate, they both have thoughts they are trying to express using words, and that absent this, we can only go so far with AI.

I don’t want to be absolutely categorical, a priori, and claim there will never be an artificially produced, intelligent, sentient machine. I mean, maybe there will be of some kind. Maybe there will be. But I don’t think any sane person thinks large-language models are that.

They don’t have a kind of inner autonomous agency, right? They’re not engaging with you. They’re a tool that you either use or you don’t. I do use them now like a search engine. But it’s not an intelligent machine that has lived experience in the world.

I would say that having a kind of physical, embodied lived experience in the world would be a prerequisite to true artificial intelligence.

2. Determinism and Free Will

Earlier you mentioned your colleague Robert Sapolsky, and you wrote a review of his book Determined where you pulled no punches.2 I was in a book club where we discussed his ideas for several weeks. It was painful.

I don’t think it his book is worth that level of engagement. It’s not serious.

Agreed, but these are smart people in my book club, I’d like to say, and some found his work very seductive. And what makes it seductive is that he makes it seem if you don’t agree with him, you’re invoking some spooky or even Biblical unknowable force in the Universe.

And this is where your book The Restless Clock comes in, because it traces the history of science viewing nature as a clock, created by a divine Clockmaker, echoes to the present day with people like Sapolsky. So I’m wondering if you can trace that history briefly?

The irony is that somebody like Sapolsky presents his totally mechanist, totally determinist model as the only scientific model, and suggests that if you reject his model, then you’re going over into spooky woo-woo kind of metaphysical stuff, when in fact, it’s the other way around.

That mechanistic clockwork model grew out of the argument from design, that somewhere out beyond our mechanical world, there must have been a rational designer.

And yet we’ve had this weird reversal where people like Richard Dawkins and Robert Sapolsky, they’re heirs to William Paley, who in the early 19th century wrote the definitive version of the argument from design. If you stub your toe on a rock, you don’t ask how the rock got there, but if you stub your toe on a clock, there had to be a clockmaker somewhere around, it couldn’t just be there.

Of course, then comes Darwin, who replaces the external clockmaker with evolution by natural selection. But what I’m interested in in The Restless Clock is the way in which Darwin really had to wrestle with his model because he was pulled in two directions. He was pulled toward the Paley direction, of there being a clockmaker out there somewhere, but also in the Lamarck direction, which gets rid of the external clockmaker by giving organisms agency.

So if you were to strike your foot against something and it yelps and scuttles away, you might conclude it arrived there by its own agency. Which is very interesting to think about.

3. Eugenics and Behavioral Genetics

This brings us to the ugliest extreme of Darwin, namely eugenics, and the notion of determinism through our genetic code. You and Mark Feldman wrote a review of Kathryn Paige Harden’s book, The Genetic Lottery, that also pulled no punches.3

It felt like there was real anger in that review, at the revival of arguably the worst manifestation of modern science. Harden would say she’s trying to recast things, and that understanding genes is not inevitably evil, and in fact, if you’re of the progressive left, you need to have an understanding of the role that genes play in making us who there are.

And you and Feldman are basically saying it really doesn’t tell us all that much.

Genes obviously play an important role in living beings. Obviously, I wouldn’t deny that; it would be kind of crazy to deny it.

But I think it’s a category mistake to think that they do some of the things that she and others in sociogenomics and behavioral genetics ascribe to them. Things such as grit or educational attainment. Those are social and cultural categories that are hard to measure in themselves.

I’ve just been looking at one study discussing polygenic indices for educational attainment in children, and environmental influences involving how their parents and caregivers treat them. And the study reported finding that children’s polygenic indices for educational attainment make no difference to whether the parents sing or read to the child, but only to whether they play with a child.

This is totally preposterous: if you’ve ever sung or read to a child or played with a child, you know that these are not separable. It’s just crazy.

DNA plays a crucial role in the production and disposition of proteins. And that’s important in all kinds of ways. But there need to be plausible causal mechanisms. In the production of melanin in the iris, there’s a very clear causal pathway from DNA to dark-colored eyes.

But there are no such pathways from DNA to the Harvard Admissions Office, they don’t exist. They couldn’t exist. It’s like saying that you could use Euclidean geometry to do the weather report.

It took me a while to figure this out. It doesn’t necessarily occur to people that this realm of behavioral genetics is more social science than it is natural science. As soon as you use the word genetics, you think that what’s happening here involves people in white lab coats conducting experiments.

But actually, it’s just statistics and correlations. We take a trait that we see in the world, we put a metric on it, and then we match it up to some aggregated genetic data. It feels to me that in the last five years, on the scientific frontier we’re starting to realize this has many limits. Yet, at the same time, we’re also seeing the revival of expressly eugenicist efforts to manipulate your baby’s DNA to make them smarter.

I’m curious how you see things trending?

Behavioral genetics uses these very enormous and easily accessible databases. So the social scientists are not running gels in a lab. They’re accessing these databases and doing a kind of astrology with them.

Sometimes they seem invincible, but I think it’s worth continuing to make the counterargument. I have a colleague who has been making it since 1970. The current version with this alphabet soup of GWASs and PGIs and EAs, the people in this field repeat the old adage that correlation is not causation.

And yet, their whole field is founded on a conflation of correlation with causation. They never make the slightest gesture at a causal mechanism. How do you get from the creation and disposition of certain proteins all the way to life outcomes?

The riddle wrapped inside an enigma is when you take something like intelligence. By design, intelligence tests create the bell curve, but then we treat that as if that is a naturally occurring phenomenon, when it’s not. You have to pick and choose what questions to use to get to that distribution. And so you actually have social science wrapped inside of social science, all which is largely hidden from view.

Exactly, exactly. And even on the correlation level, putting aside the absence of causation, the correlations are vanishingly tiny.

And then there’s the whole problem with population stratification that keeps coming up again and again. One example I’ve heard is using multiple forks. You could do a polygenic study to determine who uses multiple forks at their meals, and you’d find an association with aristocrats, since they pass down both their DNA and their cutlery practices.

But DNA is not causing people to use a different fork for their salad.

We shy away from this, but this is clearly what happens with race. We have clearly created institutions with a long historical track record that have punished and kept people in a state of subservience because of differences around perceived race. People see disparity in life outcomes cutting along racial lines and assume there’s some X% we can attribute to genes.

But when it comes to nature and nurture, you invoke the metaphor of a coastline, which I love. How much of the coastline is caused by the ocean and how much the shore? It’s just a nonsensical question.

4. Lamarck and The Power of Life

The Power of Life, your new book on Jean-Baptiste Lamarck, will come out in March. I don’t want to give away the ending, but it ends with three words: “Lamarck was right.” What was he right about?

I’ve thought about this a lot. I think that Lamarck was right that living organisms shape the course of evolutionary development, by their behaviors and responses to the environment on various levels. They shape the course of evolutionary development.

They are not solely the passive objects of outside forces.

Of course, Lamarck died in 1829. So he didn’t know about anything that’s happened since. I don’t mean that he was literally right each of his statements, but overall, his picture of the living world was one according to which living organisms are continually actively, creatively engaging with their surroundings on many levels. And that shapes their development from generation to generation.

I think I can firmly say he was right about that.

My only memory of Lamarck from my own education, prior to reading your book, was being told in my biology class that Lamarck though giraffes stretched their necks and then passed this trait along, and boy wasn’t he so dumb to think that.

But was he wrong about the giraffe?

So funny you should ask that, because I am now involved in the first science experiment I’ve ever been involved in since high school. It’s a study of epigenetics in giraffe inheritance, with colleagues at Stanford and other schools, and people at the Giraffe Conservation Foundation in Namibia.

They actually collected cheek swabs, if you can believe it, as part of their conservation efforts. My daughter thought was the most hilarious thing, she imagined people climbing up on ladders to take these samples.

I think some people hear the phrase epigenetics and get a little lost. Can you just briefly say what you mean by that?

What I mean is that the DNA molecule is kind of wrapped and packaged with other things. And these other things, these molecular structures, make a difference to gene expression.

Scientists have shown there can be inherited epigenetic effects from generation to generation, changes that are not in the DNA, but are in the structure surrounding the DNA, that cause differences, phenotypic differences that can be inherited from generation to generation.

And then these feed back into natural selection.

So, for example, if a giraffe stretched its neck, underwent epigenetic changes—changes in the packaging of its DNA —and then passed those on to its offspring, natural selection would then be acting upon offspring with an inherited longer neck.

So this is really very preliminary this research. It’s just to see whether there are epigenic structures that differ among species of giraffe and between giraffes and okapi. And if there are epigenetic differences between the species, along with the phenotypic differences, then we would design some more experiments to see what those differences might mean.

5. Cultural Evolution

All this makes me wonder if you’ve thought about the role of cultural evolution and humanity, because I think we can tell the biological story of our evolution that will take us up to some point at around, say, 100,000 years ago.

But at that point, we start to bring in all sorts of things that are clearly not rooted in biology, such as language, right? And other cultural practices that become bound up with who we are, and the ways in which we express our agency.

And I have become a huge fan of Celia Hayes at Oxford, author of Cognitive Gadgets, a book that explores how much of what we consider our cognitive architecture comes through the cultural transmission of practices.

I’m curious what your reaction is to that?

That’s super interesting. Of course, people involved in the Extended Evolutionary Synthesis have been studying culture in other animals. Whales apparently have these, like, whale conventions in different parts of the ocean, and work out their song for the year, and they change them annually.

I think the study of bioacoustics, of animal languages is also a study of culture.

And so I’m not sure that I see human culture as discontinuous or distinct from the rest of the animal world and other sorts of animal cultures. Obviously humans are very different and our culture has the appearance of being more complex and elaborate. But if you believe in evolution and you believe that humans are the product of evolution, then it would just make sense that our capacities are continuous with other animal capacities. Communication and language and culture included.

And I’ve always been sort of surprised about Chomsky and the linguistic idea that there’s this absolute discontinuity on the level of language. I think it took hold of the field of linguistics for a long time. Maybe now it’s loosening.

Well I think Chomsky’s ideas are on the ropes, both from the cultural evolutionists, and frankly from AI. The stimulus isn’t all that impoverished.

But lucky for you, we’re almost out of time, because I would push back and say that while it’s cool that whales get together and sing,that’s nothing like what humans do in terms of the institutional and cultural practices that we have developed. There’s nothing like formal schooling that animals go through.

That’s absolutely true. But there is teaching and learning and knowledge that’s handed down, in birds and other organisms, primates. I think it’s a growing area of research and it’s an interesting one.

6. Fighting Authoritarianism in America

My last question touches on something you wrote, I believe it was the review of Harden’s book, wherein you observed that the history of recent social justice movements have been grounded in the recognition of personhood and self, and in our agency to change our circumstances.

And now we are undergoing a moment in the United States of just incredible, horrific fascism aimed at certain populations, and the scientific enterprise itself, and I’ve been frustrated that more people aren’t just jumping into the fray.

But you are. Tell me about how that’s going and how you’re handling that?

Yeah, oh God, yeah. I’ve been doing a lot of that.

So we have an AAUP chapter, American Association of University Professors, at Stanford. I did not know the chapter existed until a little less than a year ago. It had four members. It never met. It had no officers or bylaws or anything.

And then in January of 2025, a bunch of us started trying to reawaken it.

So now I’m a co-president with a colleague in the medical school, Alyssa Burgart, and we have five officers and almost 200 members, and we meet every month, the executive committee meets every week, and we’ve been doing a bunch of things. We’ve done open letters and petitions and rallies and teach-ins, and we’ve been writing op-eds in the Stanford Daily. I think one of the most satisfying things we’ve done so far is to sign on to various AAUP amicus briefs in lawsuits against this Administration.

Our two causes are (1) academic freedom and autonomy, and (2) shared governance of the university. Those are the two things we can all agree on.

The thing that has been most dismaying, almost, since the beginning of Trump 2, is how his administration cuts through the whole country like butter. The law firms collapse, the media companies collapse, the universities by and large collapse, though not quite all of them yet.

And it just felt like even if nothing we do has an effect, at least it feels better than doing nothing. We’re bearing witness, and we’re manifesting our presence, at the very least.

You’re proclaiming a will. And I admire you for it.

My thanks again to Jessica Riskin for sharing her thoughts on such a wide range of topics. The Cognitive Resonance newsletter will be on brief hiatus for the next week or so due to travel.

Subscribe now

Again I begin with gratitude.

When I hit send last Monday on the essay about my father’s death, I genuinely thought I would lose more than a few subscribers, and understandably so. I assume that many of you have signed up here to learn about AI and human cognition, and while those themes were somewhat present in my public grieving, let’s be real, this newsletter is presently serving as a form of therapy for me. Yet once again, somehow so many of you responded with words of kindness and empathy, and often while sharing with me your own vulnerabilities and tragedies.

Last week, I felt a wave of human decency directed my way that gave (and gives) me strength. I felt seen, heard, and loved. Saying thank you feels woefully insufficient to the magnitude of support you’ve collectively provided, but that is all I know how to say, over and over: Thank you. Thank you. Thank you.

That’s how my week started. Then on Wednesday, agents of the fascist regime in control of the United States murdered Renée Good, a queer poet who had the audacity to bear witness to their cruelties.

There have been many hard days since the second Trump Administration assumed power in January 2025, but for me, last Wednesday was the hardest, the absolute nadir. The despair that enveloped me as I watched the videos of Good’s killing, as I looked at the picture of the car with blood-stained seats and stuffed animals overflowing from the glovebox, I felt my brain fog over. I began sobbing uncontrollably throughout the day, seemingly at random moments, overwhelmed by the violent sadism of my own country and its purported leadership. All but immobilized by trauma.

Here again I need to share with you a bit more about my relationship with my father. As mentioned last week, we talked about AI frequently over the last year, but sitting right alongside at the top of our conversational agenda was Trump—or “T*” as my father used in texts, always refusing to name him fully—and this moment of tyranny in America. I would like you to know that my father was passionate about politics from an early age, he did volunteer work on the John Kennedy’s campaign when he (my dad) was only 11 years old (careful readers may connect this to him later becoming an assassination “conspiracy buff,” a story that again will have to wait). At every stage of my life, we talked about current affairs, through political lenses we largely shared, apart from my brief libertarian phase that scared the shit out of him (rightfully). These conversations intensified over the past year for obvious reasons, and I often asked about what life was like in the 60s, the last period when it seemed the US might dissolve, hoping to gain perspective and insight into how we might navigate the current moment. He had much wisdom to share, coupled with a gallows sense of humor that helped us both to cope.

Last year, I told my father I felt guilty about feeling so energized by the current moment. “It’s ok to feel purpose,” he told me.

But he’s gone now; Death has finished singing my father’s song, just as it has Renée Good’s. I wrote in advance of my dad’s passing how I sensed the coming unmooring from the last of my human creators, and that is exactly how I felt this past week, unmoored, like a tiny broken boat adrift at sea, battered by a violent storm and powerless to do anything to change the course of fate.

This is heavy and depressing and I won’t fault anyone for stopping here. Some close friends are worried about me, and I’m worried about me too. I’m worried about everyone living in America, if not the world, as no one is safe from this country’s violent power. But several months ago, someone shared on BlueSky a post that read, “too much ‘we’re cooked’ energy here right now, not enough ‘fuck this shit,’” and it’s that latter sentiment I want to harness. More specifically, I want to talk to you about courage.

I have never been a New Years’ resolution person, but many years ago, I started to pick “themes” for the year, meaning, I choose a topic and then endeavor to read, watch, listen, and learn about it in some greater depth. Sometime pre-pandemic, I chose courage as my theme, human courage. In particular, I wanted to learn about the people in history who faced overwhelming oppression but nonetheless fought back.

I read a lot about the Civil Rights Movement, including Reverend Fred Shuttleworth, the leader who vowed to kill segregation or be killed by it (I’ve shared my admiration for him with you previously). I read about Mamie Till-Mobley, the mother of Emmett Till, who demanded an open casket at her son’s memorial, and urged the pictures of his beaten body be made public so as to bring light to the terror of white supremacy in America. An act of courage that galvanized a movement.

I also read about Sophie Scholl. Her name may be unfamiliar to you, as it certainly was to me. She was a leader of the White Rose effort in Germany in the 1940s, an anti-Nazi resistance group comprised of university students that distributed pamphlets calling for active opposition to Hitler. Here is an excerpt from their third missive, one that was later air-dropped by the Allies all over Germany:

Prior to becoming a radical anti-fascist advocate and resistor, Sophie Scholl had taught kindergarten. At age 21, she and other White Rose leaders, including her brother, were captured by the Gestapo and, after a show trial for treason, these “domestic terrorists” were executed by guillotine.

These are a few of the stories of historical courage that I read. Now I want to share one with you about the here and now.

Over the past year, I’ve formed new friendships arising from my advocacy work in resisting the intrusion of AI into public education, including one with a veteran high-school teacher—I won’t say where, for reasons that will be clear momentarily—who is active in the “AI for Teachers” Facebook group. This place functions as my personal intellectual hell, a repository of literally hundreds of thousands of educators largely falling all over themselves to find ways to automate their own cognition and that of their students. But this teacher has taken it upon himself to speak up and push back.

We talked on Friday, or just yesterday as I’m typing this. We asked each other how we are doing, and we both agreed, not well. And it was then he told me about one of his students, a 13-year-old girl who’s mother has been snatched by ICE and is now facing deportation. This student is looking for somewhere to stay so she can continue her schooling.

Moments after telling me this on Zoom, this student entered his classroom. She walked straight to him and hugged her teacher, she hugged him so very hard, a desperate hug, the hug of someone overwhelmed and in need of love. This brief embrace, even recalling it now, well my fingers are shaking. I hear my friend Audrey’s voice again, crying out, What are we doing, what are we doing to children? To mothers, to fathers, to people without children, to innocent humans who simply want to live without fear?

I later asked my teacher friend for permission to share this story, which he granted, and he told me this:

That hug was the first of many this morning. She came back and we talked for a good half hour. She came across the border two years ago at age 13 from Venezuela--which means if her mom gets deported back there, she’ll end up in jail for pretty much ever unless there’s true regime change. They were separated for four days when they crossed--didn’t speak any English--and thrown into holding cell. Didn’t know what was happening, but when reunited, mom told her they would never be separated like that again. So incredibly sad and infuriating at the same time as she told the story with tears in her eyes.

Her mom basically has a couple weeks to appeal the deportation order, and so me and another teacher have been working frantically to get her an attorney to take the case, and we were able to get one about an hour ago. So that’s great news. I’ll keep you posted.

This is not fair. No educator should have to take time out of their day to search for legal representation for their students’ families. It is emotionally exhausting and it is time consuming. But I am so very grateful to those who bear this burden nonetheless. I am in awe of their courage. (My teacher friend also wanted me to note here that none of this should be interpreted as his supporting the Trump Administration’s actions in Venezuela.)

Every human being has a claim to a useful and just state. I had to pause the writing of this post to attend a protest on Saturday. I carried an American flag with me, as I typically do at these rallies, but for the first time, I mounted it upside down. Someone asked me why, what does it mean. It’s a distress signal. America is in distress.

So I return to the question of courage. In the wake of Renée Good’s killing, Nikole Hannah-Jones, a personal hero who has long called us to account for America’s long history and ongoing practices of violent subjugation of non-white communities, she wrote on social media, “this is just the beginning.” She is right, I have no doubt about that. Those of us who still believe in the best aspirations of American democracy will be tested by even greater horrors than what we’re witnessing at present. We must find the courage to bestir ourselves to fight back. We need “fuck this shit” energy and lots of it.

That will come to me, I trust. And maybe next week, I’ll return to writing about cognition and AI, we’ll see. Right now, though, I hope you will permit me to share with you a poem I’ve just written. It’s not very good, but it’s all I’m capable of.

Blood of White Roses

they are called stuffies nowadays, at least that’s what my nine year old daughter calls hers
she presides over a vast empire, a coterie so large as to put the lie to Noah’s story
no boat could be so big
a few years ago her mother declared a moratorium on further acquisitions
one of the most useless and flaccid policy pronouncements of this century

there are axolotls and capybaras and a pink silky chicken-like creature that reminds me of David Bowie
that one being the last present my mother ever gave her
Puppy is her favorite, a ragdoll of a dog, but I always ask about Rainbow Goat
Which I made for her during the pandemic and drove from Texas to Florida to deliver
accompanied along the way by a baby yoda that I took pictures of in odd places
which delighted her in transit
but upon arrival she proclaimed him “creepy”
now he sits with me in the passenger seat when I roam about America
so I am not alone

my Puppy I named Barney, a panda bear pillow with a face I slept upon for who knows how many years
(there’s no one left to ask)
at some point his eyes wore away, leaving a trail of synthetic cotton wherever I dragged him
one day my mother—she didn’t know, she couldn’t possible have known—threw him away
it’s my earliest memory of screaming

(we rescued Barney after digging through the trash, or so I choose to remember)

I am going to do the hard thing now
and look at the picture again
the one with the blood-soaked airbag, a yellow line of police tape in the foreground
as I type this so many are looking at the other pictures and videos
trying to figure out where the gun was
which way the wheels were pointed
but I am staring at the glovebox with the stuffies crammed into it
a unicorn I’m almost sure of, with a mane like a mohawk
what I first thought might be a brown platypus
now I think, perhaps a bear's butt
a little blue octopus, I don’t know
I have to stop looking now

news reports inform me that these three little creatures may have belonged to a six year old boy
I keep thinking, he’s somewhere out there
his father died years ago
and the mother that brought him into this world
she who wrote poetry that asked us to make room for wonder
she is gone too
so that is one immutable fact shared between us
newly orphaned in our common country

only he is so much younger

Subscribe now

First, thank you to everyone who reached out to me over the holidays to offer their condolences on my father’s passing. One touching consequence of writing a newsletter is receiving words of support from people I’ve never met in person, a connection with those who might otherwise be strangers but for having read what I’ve shared here. I am deeply appreciative for your kind words, they’ve meant a lot during my grieving.

Along those same lines, I hope you’ll forgive me if I spend one more week telling you about my dad, because as you’ll soon learn, AI played a bizarre role both in fostering our relationship and, perhaps, in hastening his death. To understand what happened will require that I share with you some intimate details about who he was and who I am, things he struggled with and the struggles between us, and how the intersection of his curiosity with this new technology proved to be one of my most vexing challenges over the last year.

So, to start: My father was trained as a neuroscientist. He received his PhD in 1977 from the University of Florida, which explains the odd fact that I was born in Gainesville, a city I’ve yet to ever visit (my parents moved away when I just a few weeks old). After he completed his post-doc in southern California, my family moved to Long Island so he could join the newly formed department of neurology at SUNY Stony Brook as an assistant professor. Over the next several years, he published around 50 research articles, predominately on the effects of sustained drug use on the brain.

Then, circa 1983, something happened that would upend the course of my family’s future. For reasons that remain shrouded in some mystery, my father stopped working, and he would spend the rest of his life on disability. I was only seven at the time, but I remember my dad was “sick,” and was making frequent visits to see specialists in New York City. But sick with what exactly, you might wonder, and me too, for all my life. Because the doctors could never quite pinpoint anything specifically wrong with him, at least physically. According to medical reports that I only found a few weeks ago, buried deep in my dad’s filing cabinet, neurological examinations suggested he possessed an extraordinarily high degree of verbal capability, but struggled with relatively simple logical tasks and problem solving, a “highly unusual combination,” as one examiner put it. Some of his doctors speculated he might have some form of encephalitis, an inflammation of the brain, but others found no evidence of that.

Whatever was going on, the upshot is that my father never worked again in his life. This, as you might imagine, was the source of considerable stress in my family, especially since my mother worked as a school librarian, which of course is not a lucrative career choice. I remember being shocked, and frankly deeply resentful, when I figured out in high school that our family of five (I have one sister and one brother), was technically living below the poverty line. I loathed this state of affairs and blamed my parents, my dad in particular, for the limitations that resulted. Later in my life, I would come to recognize that growing up in these circumstances helped shape me to become self-sufficient and independent in ways I’m now fiercely proud of. Indeed, once I started moving into the world of the rich and privileged and realized how fucked up so many people seem to be when they live a life without constraints, I even became grateful. But in my 20s and into my 30s, well, my relationship with my dad was strained, to say the least.

Although my father was unemployed for nearly all of his adult life, he never lost his intellectual curiosity, and thus he developed an unusual medley of interests to keep his mind active. One of these was the JFK assassination, he became part of the “conspiracy buff” community—we’ll save that story for another time. But another enduring area of interest was technology, where he was remarkably prescient in seeing where things were headed.

To wit: My family was the very first on the block to have a “microcomputer” in our home in the form of a Commodore 64 (and later the Amiga, which I think my dad felt genuine affection for). He also got active on what were called electronic “bulletin board systems” or BBSs, the precursor to the Internet really, where people connected their computers to their phone lines to send files back and forth at rates so painfully slow it defies modern comprehension. What’s more, back in the day the monopolistic phone companies charged exorbitant rates for making “long distance” phone calls, which meant that some BBS users, including my dad, looked for workarounds by hacking (or “phreaking”) phone codes to make free dial-up connections. This was illegal, of course, and when I was in eighth grade my dad was arrested and charged with multiple felony counts of theft by computer intrusion—we’ll save that story for another time, too.

My point is that my dad was always interested both in the brain and technology, so when AI in the form of large-language models were dropped into our world, he was absolutely fascinated—and so was I. As such, when I began my own efforts to understand how these models were doing what they were doing, he was right there alongside me for the intellectual journey. And for all my critiques of this new technology, I will forever be grateful that AI created a path for my dad and I have to have so many rich conversations over the past two years about its functioning. It’s fair to say it helped restore our relationship, and created a new bond between us, as we together tried to figure out just how similar (or not) its processes compare to human cognition. Cognitive Resonance exists in part because of these conversations, as I realized the exploration my father and I were mutually sharing might be of broader interest to the world. (A hypothesis I’m still testing.)

Which is why it’s such a strange, tragic irony that AI played a non-trivial role in the health crisis that led to my dad’s death.

As I’ve shared with you previously, about 18 months ago my father was diagnosed with lung cancer, kidney disease, and Chronic Lymphocytic Leukemia (CLL). Upon receiving the news, he quickly addressed the lung cancer via radiation treatment, and—after some false starts—was eventually able to successfully treat his kidneys as well. But the CLL, well, that’s a more complicated story, and it’s where his use of AI likely hastened his declining health, and magnified the pain he endured.

Here’s what happened: Not long after his CLL diagnosis, my dad’s oncologist recommended he start “Venetoclax-Obinutuzumab” treatment (Ven-Obi), a relatively new approach to addressing CLL that’s proven remarkably effective both at extending patient life expectancy while reducing physical suffering. I did not know his doctor was urging this, however, because my dad did not tell me or my siblings. Instead, my father became convinced that he was undergoing something called Richter’s Transformation, a rare complication of CLL that is particularly painful. There was no evidence of this, medically, but my dad nonetheless believed it was happening to him, and that as a result he should refrain from treating his CLL by Ven-Obi because it would only make things worse.

And my father believed that because that’s what Perplexity AI told him.

It was a shock when I discovered what was happening, as you might imagine. I only discovered what was going on when my father gave me access to his online medical record, allowing me to peer into his long-running correspondence with his oncologist. From that I learned that my dad had used Perplexity to self-diagnose his condition and had sent the Perplexity report, if it can be called that, along to his very perplexed and frustrated doctor. Given that I’d spent the better part of a year talking with my father about the unreliability of factual statements made by AI, you can only imagine my extreme frustration discovering that my efforts had utterly failed within my own family.

AI enthusiasts, whether in education or more broadly, will often try to cover their asses from responsibility for non-factual statements by AI models by saying, “well, you always need to check their output.” As a general matter, that’s a ludicrous claim, since the whole value proposition of these tools is to spare us cognitive effort—but in this instance, it’s exactly what I did. I contacted the doctors who led the study that Perplexity cited in support of its statement that refraining from Ven-Obi was the proper course of action for someone with Richter’s. Much to my surprise, both doctors replied straightaway, and confirmed what I already knew to be true, that Perplexity had misstated the conclusion of their research, and that my father should follow the course of treatment his oncologist was recommending.

Of course I immediately passed this information along to my dad, desperately hoping to appeal to his scientific and empirically oriented belief system. But he didn’t respond at all. I was yelling into the void. It was only after several more months passed, and after his physical condition continued to worsen dramatically, before he finally agreed to start the Ven-Obi treatment his oncologist had recommended a year prior. It didn’t seem to matter at that point, sadly. Although the treatment immediately reduced his white blood cell count, his pain endured, and culminated in his death just a few weeks ago.

I am obviously still grappling with all this, and I don’t want to overstate my case. I don’t think AI killed my father. I think it’s possible, perhaps even likely, that in a world without AI, he would still have latched on to some other piece of research to support his disposition against medical treatment, as he had deep misgivings—fear, really—about spending time in hospitals. Nonetheless, the fact remains that AI does exist in our world, and just as it can serve as fuel to those suffering manic psychosis, so too may it affirm or amplify our mistaken understanding of what’s happening to us physically and medically. (OpenAI claims to be limiting the use of ChatGPT to provide “tailored” medical advice that requires a license, but the head of its medical research team maintains “it will continue to be a great resource to help people understand legal and health information” —sure, if you say so.)

In the course of discussing AI with my dad, I am fairly certain he moved toward becoming more generally skeptical of it. Toward the end of his life, he started sending me articles and YouTube videos about the limitations of these tools. Still, I will forever wonder whether my efforts came too late, and whether he might still be with us if I’d been more effective in undermining the authoritative tone AI strikes when generating its tokens. There’s nothing I can do to change the past, of course. But I can for damn sure keep working to raise the consciousness of others.

A fire has been lit, even while my heart still hurts.

My father created a playlist he labelled “Wake,” which doubles both as testament to his great taste in music and elegy for what’s happening in America. You’ll be doing me and his memory a great honor if you give it a listen. He’d like knowing it found an audience.

Subscribe now