Pedagogy of the Omnimodal

ChatGPT-4o is not the perfect teacher. It's not even mediocre.

This is an unfair screenshot, sorry not sorry

One thing I’m hoping to avoid here at Cognitive Resonance headquarters is reacting to the latest breathless announcements from various Big Tech companies about improvements to their chatbot products. You will lose your mind quickly playing that game. (And by you, I mean me.)

That said, I’m suspending my rule to offer some thoughts on OpenAI’s release of ChatGPT-4o yesterday – the “o” is for “omnimodal,” which we’ll explore momentarily – and the surrounding hype related to its purported transformational educational value. 

Our starting point is this video of Sal Khan and his son Imran using GPT-4o to learn about calculating the sine of angles in right triangles. The Cool New Trick is that interacting with GPT-4o can now take place in multiple modalities, meaning, you can talk to it and share your screen with it as you work through math problems or whatever. Neat! Go ahead and watch the full 3.5 minutes, if you haven’t already.

If you suspend your critical thinking skills for a moment and just let this wash over you, it sure seems impressive. Indeed, some folks on social media are so wowed that they declared that, with this new product release, OpenAI has permanently transformed education by creating the “perfect teacher for any imaginable topic.” I mean…big, if true.

But of course the goal of Cognitive Resonance is to foster critical thinking about generative AI, and to not suspend our reasoning faculties when thinking about this technology. And a careful parsing of this product-demo video suggests that the pedagogy practiced by GTP-4o is, well, a mess. Let’s turn to the play-by-play:

1. To kick things off, we learn Sal Khan has a new book about AI in education that he’s decided to title, and I am not making this up, Brave New Words, an allusion to the dystopian novel by Aldous Huxley that is explicitly meant as a warning against unbridled technological optimism and social engineering that leads to the erosion of the essence of what makes us human. I love irony myself, but c’mon man.

2. Blink and you’ll miss it, but at 30 seconds in, GPT-4o starts talking in a seemingly confused manner, and Sal just…interrupts and talks right over it. Not the sort of behavior we typically want to encourage in students. It’s worth pondering what sort of educational norms will be fostered if young children can interrupt their AI teacher-tutors whenever they so choose.

3. At 1:22, the younger Khan lets out an audible “hmmm” as he thinks – or more likely, pretends to be thinking – about what the hypotenuse of a triangle is. This is the sort of productive struggle that effective educators readily recognize, and typically they will give students the time and space they need think things out. GPT-4o, in contrast, seems eager to fill the silent void, thus interrupting Imran’s train of thought. But all this is rendered moot by Imran interrupting GPT-4o himself – like father, like son.

4. At 2:40, listen carefully. After Imran correctly identifies the formula to calculate the sine of angle alpha, GPT-4o asks him to apply that formula to calculate the “side alpha,” with the letter d. There is no side alpha, it does not exist in this problem. This is a hallucination. Imran blows right past this – he already knows how to do the math here, I’m sure – but imagine a student who was actually learning this content for the first time. The sudden switch from “angle alpha” to “side alpha” could be confusing, and a novice thinker might struggle to understand what was being asked.

5. To top things off, at 3:10 we hear GPT-4o refer to “sine alpha” rather than “the sine of angle alpha.” Minor mistake, sure, but note that “alpha” has now been used as designation for an angle, a side, and a sine – again, confusing! And we’ve yet to see what happens when GPT-4o is confronted by a student who is genuinely learning about these mathematical ideas for the first time, rather than play-acting for a product demo.

Is this overkill for such a short video? Yes of course. But we haven’t even taken up any of the larger philosophical issues here, such as Silicon Valley’s continued insistence that education predominately consists of teaching basic concepts and correcting students when they make mistakes, or what stereotypical gender roles are being reinforced by the seemingly ubiquitous use of female voices for audio chatbots – to name only two. My point instead is simply that, on its own terms and as deployed in the most favorable educational environment we can possibly imagine – namely, instructing the son of one of the country’s most well-known educators on a relatively straightforward math concept – GPT-4o’s pedagogy is far from perfect. It’s mediocre at best.

Thus the question: What will happen when the conditions are far less amenable that what we see in this demo? Throughout the video Imran Khan helpfully explains his thinking to GPT-4o, but trust me when I tell you that many students will hesitate to do this. Since the chatbot has no ability to theorize about what’s happening in the minds of the humans it interacts with, what happens when its students clam up? For that matter, what happens if GPT-4o is asked a larger conceptual question that goes beyond the boundaries of the task at hand?

“One believes things because one has been conditioned to believe them.” That’s a quote from the novel that Sal Khan would do well to read and reflect upon, rather than just misappropriate its title. We must not be conditioned to believe that chatbot tutors are the future of education.

Don’t believe the hype.