Humans understand what AI cannot

In classrooms and Chinese Rooms

Source

Understanding is a Big Juicy Word, like justice or freedom, that is powerful and important yet perilous to define. No definition could possibly encompass the entirety of what we mean by understanding, yet we all have a general sense of it means; when we ask someone, “do you understand?”, this is not a puzzling question (most of the time, anyway). Why is that? What are we talking about when we talk about understanding?

These questions sit at the conceptual juncture where I spend most of my time nowadays, the intersection between human thought and behavior and education and whatever it is AI models are doing. So this week, we’re going to explore what we mean by understanding by looking at it from a few different angles: the pedagogical, the philosophical, and the profound.

Teachers are the professionals we task with imparting knowledge to students—it is literally their job to help kids understand things. Recently, my friend Michael Pershan, a veteran teacher in NYC who writes an excellent newsletter, appeared on a podcast wherein he vehemently advocated for the importance of building student understanding. This in turn led to a flurry of text messages between us wherein he shared his perspective that “understanding is when you know explicit principles, or generalizations if you prefer, that can guide your activity….it’s knowledge of generalizations that can support procedural learning, and it’s why procedural and conceptual knowledge have to happen in tandem. You can’t first learn all the procedures and then learn the concepts later.”

Viewed through this pedagogical lens, understanding entails a certain level of abstraction, what cognitive scientists call schema. Building this is not easy. A negative times a negative equals a positive is (relatively) easy fact to remember, but why is this true? Can you explain it to eighth graders in ways that will make sense to them, so that they will understand the concept?

Relatedly, Dan Willingham describes understanding as “remembering in disguise” because it involves connecting new ideas to existing “old” ideas and then “rearranging them—making comparisons we hadn’t made before, or thinking about a feature we previously ignored.” Effective teachers help students make this happen by facilitating this mental rearranging, using their own knowledge of their students to foster the connecting of the new to the old.

In contrast, large-language models at present are completely useless at this, in part because they lack any depth of understanding of what is happening in the minds of their users. The failure of AI chatbots as tutors for kids was entirely predictable. I predicted it. And here’s a newsflash, AI agents as tutors will fail just as spectacularly, for exactly the same reason.

Let’s turn now to the philosophical question of understanding, and I’m afraid we must peer into philosopher John Searle’s infamous Chinese Room. For the unfamiliar, back in 1980 in an article tellingly titled, “Mind, brains, and programs,” Searle attacked the idea of “strong AI,” the idea that we should treat a computer as a mind. To do this, he offered the following thought experiment (my emphasis)¹:

Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese.

This is the argument that’s launched a thousand philosophy PhDs, but we’re not going to dive deep into its weeds today. Instead, I’m citing Searle for the relatively simple and at least cogent proposition that understanding involves a mental state that exists between mere input and output. You might be thinking, well yeah, duh, but here I must remind you that for a long time within psychology it was considered uncouth to treat mental states as something worthy of scientific inquiry or even discussion, largely due to B.F. Skinner’s pernicious influence suggesting ”mentalizing” was for the weak.

In any event, Searle believed that it was important not to dance for too long on the head of the understanding pin. As he put it: “One finds a lot of fancy footwork about the word ‘understanding.’ My critics point out that there are many different degrees of understanding; that ‘understanding’ is not a simple two-place predicate; that there are even different kinds and levels of understanding…To all of these points I want to say: of course, of course. But they have nothing to do with the points at issue. There are clear cases in which understanding literally applies and clear cases in which it does not apply; and these two sorts of cases are all I need [for the Chinese Room argument].”

Did John Searle ever read Helen Keller’s autobiography? I wonder. Because whatever might be said about the philosophical merits of his Chinese Room argument, I submit to you that Keller’s account of her own cognitive development accords with the fundamental claim he was making.² It is a story both profoundly haunting and inspiring.

As you may know, Helen Keller was not born blind or deaf; instead, at 18 months, she suffered a mysterious illness that “closed my ears and eyes and plunged me into the unconsciousness of a newborn baby.” For the next five years of her life, her communication was limited to a small set of “crude signs” for basic needs. But she was aware, however dimly, that other humans were capable of conversing in ways she could not comprehend:

I noticed that my mother and friends did not use signs as I did when they wanted anything done, but talked with their mouths. Sometimes I stood between two persons who were conversing and touched their lips. I could not understand, and was vexed. I moved my lips and gesticulated frantically without result. This made me so angry at times that I kicked and screamed until I was exhausted.

Can you imagine the sheer agony of this experience? The horror of sensing that other humans are talking to each other, but realizing that you might forever be denied the same possibility? “I felt as if invisible hands were holding me,” Keller wrote, confining her to world “still and dark.”

As Dan Meyer once said to me: Hell is silent.

But things did not stay that way, and of course today we know about Helen Keller’s incredible life because she ultimately learned to communicate with the outside world, largely through the persistence of her very human teacher, Anne Sullivan. The love between the two of them is worth longer exploration, and perhaps I’ll do that in a future essay, but right now I want to focus instead on Keller’s emergent cognition.

It happened quickly, but not immediately. When Sullivan started fingerspelling words into Keller’s hands, initially Keller treated the experience as a game—she could repeat back the movements Sullivan was making, but there was no meaningful cognitive connection formed:

The morning after my teacher came she led me into her room and gave me a doll…When I had played with it a little while, Miss Sullivan slowly spelled into my hand the word “d-o-l-l.” I was at once interested in this finger play and tried to imitate it. When I finally succeeded in making the letters correctly I was flushed with childish pleasure and pride. Running downstairs to my mother I held up my hand and made the letters for doll. I did not know that I was spelling a word or even that words existed; I was simply making my fingers go in monkey-like imitation.

In other words, Keller was capable of matching this particular input (the finger movements of d-o-l-l) to its particular output (the doll in her hand), but she lacked knowledge of the generalized idea of words, that words have meaning. But then the breakthrough came:

We walked down the path to the well-house, attracted by the fragrance of the honeysuckle with which it was covered. Someone was drawing water and my teacher placed my hand under the spout. As the cool stream gushed over one hand she spelled into the other the word water, first slowly, then rapidly. I stood still, my whole attention fixed upon the motions of her fingers. Suddenly I felt a misty consciousness as of something forgotten—a thrill of returning thought; and somehow the mystery of language was revealed to me. I knew then that “w-a-t-e-r” meant the wonderful cool something that was flowing over my hand. That living word awakened my soul, gave it light, hope, joy, set it free!

The living word awakened her soul—what a beautiful testament to the power of human language to express human thought. And as Keller later came to understand the meaning of other Big Juicy Words, word such as love, she felt they created “invisible lines that stretched between my spirit and the spirits of others.” Philosophers and scientists are uncomfortable with talk of souls and spirits, but theologians less so, which may help explain why communities of faith have thus far been more resistant to AI intrusion than, say, institutions of education (though that is starting to change).

But that’s just my pet theory. What is indisputable is that once Helen Keller understood the idea of words, she was able to engage in a discourse with the world, and with us. Her thoughts are still here today for us to marvel in. Do you understand how precious this is? We want to know the world, and each other. We can mentally slide between the practical realities of the classroom and the abstract playroom of philosophy percisely because we are human, which is to say, we are alive, curious, and social.

AI understands none of these things, and defames all of them.

1. Technically, this particular statement of the Chinese Room argument comes from Searle’s 1999 book; the 1980 article has a more jumbled version.

2. I’m delighted to report that two philosophers got into a heated linguistic squabble regarding Helen Keller and Searle’s Chinese Room, starting with this essay,”How Helen Keller used syntactic semantics to escape from a Chinese Room,” followed by this response, “Helen Keller was never in a Chinese Room,” which in turn lead to this rejoinder, “Yes, She Was!: Reply to Ford’s ‘Helen Keller Was Never in a Chinese Room.’” Great stuff, fellas.