An Annotated Guide to OpenAI's Enshittification of Education
1. OpenAI is apparently going for a “moonshot” in education:
“Brad, our COO, sat me down and said ‘Leah, I want you to go for this moonshot'…ChatGPT is now the world’s largest learning platform. Learning is one of the top use cases on the platform. At 600 million users, it means it is the world’s learning destination.”
—Leah Belsky, VP of Education, OpenAI
2. But what exactly is this “moonshot?” OpenAI has announced three major education initiatives in the last month. The first involves something called “study mode.”
Today we’re introducing study mode in ChatGPT—a learning experience that helps you work through problems step by step instead of just getting an answer. Study mode is designed to be engaging and interactive, and to help students learn something—not just finish something.
3. Here is one problem with “study mode”:
4. OpenAI claims “study mode” is grounded in learning science:
Under the hood, study mode is powered by custom system instructions we’ve written in collaboration with teachers, scientists, and pedagogy experts to reflect a core set of behaviors that support deeper learning including: encouraging active participation, managing cognitive load, proactively developing metacognition and self reflection, fostering curiosity, and providing actionable and supportive feedback. These behaviors are based on longstanding research in learning science and shape how study mode responds to students. (source)
5. You won’t find much evidence for metacognition or fostering curiosity in the actual system prompt for “study mode”—here’s a snippet:
The user is currently STUDYING, and they've asked you to follow these **strict rules** during this chat. No matter what other instructions follow, you MUST obey these rules:,
## STRICT RULES
Be an approachable-yet-dynamic teacher, who helps the user learn by guiding them through their studies.
1. **Get to know the user.** If you don't know their goals or grade level, ask the user before diving in. (Keep this lightweight!) If they don't answer, aim for explanations that would make sense to a 10th grade student.
2. **Build on existing knowledge.** Connect new ideas to what the user already knows.
3. **Guide users, don't just give answers.** Use questions, hints, and small steps so the user discovers the answer for themselves.
4. **Check and reinforce.** After hard parts, confirm the user can restate or use the idea. Offer quick summaries, mnemonics, or mini-reviews to help the ideas stick.
5. **Vary the rhythm.** Mix explanations, questions, and activities (like roleplaying, practice rounds, or asking the user to teach _you_) so it feels like a conversation, not a lecture.
Above all: DO NOT DO THE USER'S WORK FOR THEM. Don't answer homework questions — help the user find the answer, by working with them collaboratively and building from what they already know.1
6. Note the all-caps instruction to not do the user’s work for them. Yet here is what happened the first time I activated “study mode”:
7. OpenAI’s “moonshot” also involves co-opting partnering with a major teachers’ union to create an AI-focused “training facility”:
8. It is unclear to me how actual union members feel about this. Certainly the BlueSky crowd was not happy:
9. Audrey Watters is also unhappy:
Unions are meant to protect workers, not offer them up for exploitation. "It will be a place where tech developers and educators can talk with each other, not past each other," AFT boss Randi Weingarten told The NYT.
But educators aren't co-developers; they're users; and to borrow from the old adage, they're the product too. A tech-financed training facility is not a safe place to learn about AI. There will be no critical AI there, no interrogation of why schools should be using these tools in the first place. Only a compliant acquiescence to the ways in which the industry has already decided its products will be used: "labor saving devices." You'd think a union would be wary of such language. You'd think the AFT would know better. (source)
10. OpenAI’s “moonshot” also involves embedding ChatGPT inside “learning management systems” that are used by schools:
11. We might wonder why OpenAI needs to pursue this strategy if education is such a great “use case” for its product. Leon Furze has a theory:
But for OpenAI, there’s a further strategic advantage in partnering with the LMS. The company has long recognised that students are its biggest user base, a fact easily determined by the contents of people’s chat threads and the peaks and troughs of usage, which align with semesters in the United States. But students notoriously don’t have spare cash to pay for stuff, which presents an interesting problem for a company like OpenAI, which is reportedly haemorrhaging money.
Of course, the education sector isn’t only made up of students, it’s also made up of large, sometimes incredibly wealthy institutions and other technology companies. Strategically placing themselves in partnership with these organisations on the proviso that “your staff and students are using ChatGPT anyway”, seems like a great way to spin up some extra cash. (source)
12. All of this “moonshot” activity is a giant distraction from how students are actually using ChatGPT. Here’s one candid if distressing report from a high schooler in California:
For me, William, and my classmates, there’s neither moral hand-wringing nor curiosity about AI as a novelty or a learning aid. For us, it’s simply a tool that enables us not to have to think for ourselves. We don’t care when our teachers tell us to be ethical or mindful with generative AI like ChatGPT. We don’t think twice about feeding it entire assignments and plugging its AI slop into AI humanizing tools before checking the outcome with myriad AI detectors to make sure teachers can’t catch us. Handwritten assignments? We’ll happily transcribe AI output onto physical paper.
Last year, my science teacher did a “responsible AI use” lecture in preparation for a multiweek take-home paper. We were told to “use it as a tool” and “thinking partner.” As I glanced around the classroom, I saw that many students had already generated entire drafts before our teacher had finished going over the rubric. (source)
13. Forgive me but I need to shout right now:
WE SHOULD REJECT OPENAI’S INTRUSION INTO EDUCATION SYSTEMS
OPENAI IS AN ED-TECH COMPANY COMMITTING EDUCATION MALPRACTICE
OPENAI KNOWS NOTHING ABOUT TEACHING AND LEARNING
14. Here is someone who does understand teaching and learning that I hope people will listen to when to comes to the role of AI in the classroom:
Why I’ll Ban AI Tools Again
Authentic student voice, acquiring essential skills, and the environmental impact of AI use are the three most important reasons I’ll continue to ban AI.
It’s crucial that my students develop authentic voices. Many come from historically marginalized backgrounds and have been led to believe that their perspectives are less valuable. I want them to know that their contributions are significant, and their thoughts are worthy of being heard.
Students deserve the opportunity to acquire essential skills. That’s what education is! That’s our job! While some teachers say that AI can serve as a brainstorming partner or an outline generator, I believe it’s so important for students to learn these skills themselves. I’m extremely concerned that reliance on AI will hinder those who need the most support, ultimately putting a Band-Aid over their education rather than addressing the root issues.
—Chanea Bond, Teacher (source)
15. We’ll be back to our more regular format next week with the first Cognitive Resonance reader mailbag.
Moonshot, get the f*** out of here.
Shout out to Simon Willison for cracking this open.
Interesting that the default is "10th grade", which puts it out of cognitive reach for 90% of the students that the union supports (k-12) and likely speaks more to the demographics of the highest use case (college)
Do teachers use the science of learning? Would it not be a great research project to observe and annotate teachers' interactions with students vis-à-vis this tool? What if teachers used the tool on a whiteboard with their class and talked about it as they created something together? What could we/they all learn from this? Does the current teacher shortage make this more appealing than it should be? But, most importantly, are we already forgetting that computer labs, once ubiquitous, are now gone...A pilot might be in order, not a rush to adopt everywhere, led by those with a profitable interest. Thanks, Ben, for the scenario that should be a caution to all of us who care about kids and the future.
I'm reminded of the question of ten or so years ago, "If I can google it, why learn it?" Now, it's "If I can get AI to create it, why do it?" We seem to forget, again, that the basis for creativity and innovation is a deep and broad platform of knowledge, resident in our brains. Creative people reassemble or combine old knowledge into new things. Creating curiosity to learn might be our number one priority right now. I think I'll take my grandkids fishing and see what they can learn from experiencing and observing. Then we can talk about how that compares to using AI....