Do we need a new scientific paradigm to understand AI?

A sprawling conversation with cognitive scientist Sean Trott (Part One)

Sean Trott is a cognitive scientist at UC San Diego who studies human cognition and generative AI, topics he covers in delightful detail via his newsletter The Counterfactual, which I highly recommend. A few months ago we began a long conversation about the scientific study of human minds and large-language models, and with his permission I’ll be sharing our discourse here at Cognitive Resonance. We kicked things off with some musings about whether the existing frameworks of cognitive science will be sufficient for understanding how generative AI works…

Ben to Sean:

Sean, one reason I've gravitated to your writings on generative AI is that you are clearly wrestling with some pretty deep questions involving how we should conceptualize our approach to understanding what they are, and what this might tell us (if anything) about how humans think and learn. You've coined the provocative if slightly hard to pronounce term "LLM-ology" to describe your wonderings in this area, and I see you as ultimately in pursuit of a science of LLM-ology. I think this is awesome.

This also raises a billion questions, and I have a big one for you, but first a little framing. Thomas Kuhn, in his seminal book on scientific paradigms, described "normal" science as consisting of (1) facts that are "particularly revealing of the nature of things"; (2) facts that "can be directly compared with predictions from the existing paradigm theory"; and (3) "empirical work undertaken to articulate the paradigm theory" to resolve ambiguities and solve existing problems. He argues that scientific revolutions happen when there is a crisis that demonstrates "that an existing paradigm has ceased to function adequately," and new metaphors and theories – a new paradigm – comes into conflict with existing normal science. Importantly, Kuhn argues that resolving this conflict is akin to the process of political revolution:

"Like the choice between competing political institutions, that between competing [scientific] paradigms proves to be choice between incompatible modes of community life. Because it has that character, the choice is not and cannot be determined merely by the evaluative procedures characteristic of normal science, for these issues depend in part on particular paradigm, and that paradigm is at issue."

Enough Kuhn, back to Trott. My sense is that sometimes you think of the science of LLM-ology as essentially a normal science, wherein we should study these AI systems using the tools of existing cognitive science, computer science, linguistics, psychology, etc. Yet I also sense that you sometimes feel these are insufficient for the task at hand, and that the creation of LLMs has created the sort of crisis that can lead to scientific revolution and a new paradigm.

So, finally, here's the big question – as we sit here today, are you more Normal Scientist or Scientific Revolutionary when it comes to LLM-ology?

Sean to Ben:

I really like this question!

My take on whether LLM-ology ought to be a normal science or shift the paradigm depends on the day. You're right that LLM-ology as I've framed it previously could be seen as belonging to the normal science phase insofar as I'm basically proposing to extend existing paradigms within cognitive science to the study of a new "model organism"—large language models. This approach views LLMs as cognitive systems that can be probed and analyzed using at least some of the same tools we use to study human cognition. This includes "behavioral" tests of specific cognitive capacities and "neurophysiological" tests that aim to figure out which layers or circuits perform which computations. More generally, the paradigm adopts a fairly "computational" theory of minds (both biological and artificial). 

I do think this perspective is a gamble: LLMs may not be the right "kind of thing" to study using the tools of cog sci. It's possible that we're deluding ourselves into thinking they're cognitive systems because they're called "neural" networks and because the kind of input/output they operate on is language, which we think of as intimately connected to cognition. LLM-ology as I've framed it may not be a productive research direction—just like, unfortunately, those methods may turn out to not be a productive approach to studying human cognition.

But there are also a few other things I think are worth noting here.

First, there are multiple ways in which cog-sci could lend a hand to the study of LLMs. A lot of my writing (and current research) focuses on how theories of human cognition, or specific human cognitive capacities, do or don't apply to LLMs (e.g., things like Theory of Mind, etc.). But more than anything I actually think it's the methodological perspective you find in cog sci (or maybe experimental science more generally?) that ends up being really valuable. That includes an emphasis on things like construct validity (am I measuring what I think I'm measuring?), external validity (what population am I studying here?), and so on.

One of the things that frustrates me most in discussion of LLMs these days is an over-reliance on single examples (showing an LLM either succeeding or failing on some task for some prompt). I'd like to see more discussion of what this example is supposed to represent and what this means both about this specific LLM and the more general population of LLMs. For example, what can we learn, inferentially, about GPT-4 by studying GPT-3.5 or Claude or some other LLM? We need a theory of generalization.

Second, cog-sci is in many ways itself a science of multiple paradigms. Historically, what became "cog sci" was kind of a reaction to behaviorism, the idea that we can understand organisms in terms of learned stimulus-response associations. Key assumptions included: 1) the idea that "mental representations" were useful in our scientific explanations; and 2) the mind can be usefully understood as a kind of computer or information-processor which "operates" over those representations.

This "computationalist" paradigm is fairly dominant within cog sci and computational neuroscience—memories are "encoded" and "retrieved", meaning is "computed", language is "parsed", and so on. But it's not the only game in town. There's another strain of cog sci that stems from J.J. Gibson's "ecological psychology", which emphasizes the role of the body and the environment in cognition. We see this show up in modern views like enactivism or embodied cognition. These views are sometimes presented as theories (and contrasted with information-processing theories). But in some ways I think they're also different paradigms with different ontological assumptions, different research agendas, and different notions of what counts as evidence. Proponents of radical embodied cognition, for example, think we should do away with mental representations as much as possible and treat cognitive processes as dynamical systems. And then there's distributed cognition, developed in large part here at UCSD, which applies the information-processing view to larger, more distributed systems—like a group of people making decisions together, or a pilot and their cockpit. Maybe we'll see analogous splits happen with the study of LLMs.

And third, I think there's a very interesting sociological shift happening with LLMs these days. Historically within cog-sci, one debate was always between the "symbolists" and the "connectionists"; and there was another debate between the people who argued cognition was "amodal" and the people who argued cognition was very much "grounded". Usually (though not always) these positions were aligned: symbolists/amodalists vs. connectionists/modalists; sometimes this debate also folded in the generativist vs. usage-based debates of linguistics. Text-only LLMs feel kind of like a wedge issue: they're very much connectionist models, but they're also not grounded in many of the ways that the modalists (and enactivists, etc.) think are critical for understanding cognition. They're also usage-based in the sense that they're trained on the use of language and don't come pre-loaded with formal rules about grammar – though you could definitely argue, as people such as Tal Linzen have, that different architectures are in some sense equivalent to different innate, inductive biases. I'm not sure what comes next with this shift or whether it's important, but it's certainly interesting to me.

Also, if you think LLM-ology is hard to pronounce, wait until you hear about the study of multimodal large language models (MLLM-ology).