Neurosymbolic AI—not with a bang, but a whither?
A visit to the frontlines of a contentious and longrunning battle within cognitive science
A basic scientific story told about human cognition goes like this: What makes humans unique is our extraordinary mental ingenuity. We are quick learners, we easily combine ideas in our head, and we express novel thoughts. We’re also capable of forming abstractions and generalizations, which allows us to transfer our knowledge to new and novel situations. In other words, we humans are capable of reasoning, and this is our special superpower as a species.
Do you like this story? Does it resonate with you? And do you think that artificial intelligence, if that term is to have any coherent meaning, must somehow also be capable of emulating these capacities that we humans possess? If so, congratulations, you might just be a neurosymbolicist. That’s the technical term, which I totally did not just make up, for someone who believes that because humans use symbols to represent ideas in our minds, and that the only path to true “AGI”—meaning, an AI as cognitively capable as humans are, something that is generally intelligent—will likewise require AI models to incorporate symbolic reasoning.
The most vocal neurosymbolicist nowadays is Gary Marcus, who’s been arguing stridently for years that “deep learning” in the form of pattern matching using artificial neural networks was and is inherently limited, because, well, such tools do not make use of symbols the way that we humans do. Further back, cog-sci heavyweights such as Jerry Fodor and Zenon Pylyshyn (among others) also planted themselves firmly in the neurosymbolicist camp, and contended that pursuing “connectionist” models of the mind, including artificial neural networks, was essentially a waste of time.
We have arrived at a strange crossroads in this debate. On one hand, the current tulip craze excitement surrounding generative AI is evidence that connectionist models are broadly capable despite their lack of symbolism. Geoffrey Hinton was clashing with Fodor four decades ago, and now he (Hinton) has a Nobel Prize, so that’s something. On the other hand, the giant meh of a response to the recent GPT5 release is perhaps evidence that, as Marcus predicted, deep learning has finally hit a wall. As Dave Karpf recently observed, we may be reaching the end of the “naïve AI futurism” era, the belief that generative AI tools would forever increase in capability if the companies deploying them are given more money, data, and compute power. The economic fever dream may finally be breaking.
But that still leaves the scientific question of whether and how we might emulate human intelligence artificially. And a new paper exploring the relationship between neurosymbolicism and connectionism points to a promising way forward (albeit with significant caveats that we’ll get to).
The paper is titled Whither symbols in the era of advanced neural networks? and is co-authored by Thomas L. Griffiths and Brenden M. Lake (Princeton), R. Thomas McCoy (Yale), Ellie Pavlick (Brown), and Taylor W. Webb (Université de Montréal). This collaboration is noteworthy because many of these researchers have produced some of the most important scientific critiques of AI capabilities over the years.
For example, Brendan Lake in 2017 co-led a study of artificial neural networks wherein he observed they were “still not systematic after all these years,” meaning, such models were very limited in their ability to recombine linguistic information in new ways (unlike humans). More recently, two years ago Tom McCoy authored Embers of Autoregression, one of my favorite research papers, wherein he identified a large range of tasks that LLMs struggle with because of their probabilistic approach to problem solving. (I’ve discussed Embers on frequent occasion.) The point again is that these are serious scholars who have been unafraid to point out the cognitive limitations of AI tools in the past.
But nowadays they find generative AI, well, far less limited! Here’s their argument:
Neurosymbolicists have long pointed to a set of capabilities that artificial neural network-based AI models have struggled with. This includes “compositionality,” the ability to flexibly construct thoughts based on existing ideas; “productivity,” the ability to generate entirely new thoughts based on prior observations; and “inductive biases,” the ability to learn quickly from sparse data.
But generative AI models have pulled even with humans in all three of these categories. As a result, “these advances undermine historical arguments that human minds use symbolic systems in ways that neural networks do not. Neural networks display comparable compositionality, productivity, and inductive biases to humans when evaluated against the same empirical standards as humans. Therefore, further progress in understanding the similarities and differences between human cognition and neural networks will require new methods for evaluating such claims.”
At the same time, these models have themselves been trained on data that is itself symbolic, such as human languages and compute code. “Thus, for both humans and machines, symbolic systems are useful in characterizing the abstract computational problems that need to be solved.”
As such, we should move beyond the neurosymbolic-versus-connectionist debate by embracing a “a new, more cognitively-oriented research agenda focused on understanding the details of human behavior.”
Still with me? I know, we’re deep in the weeds here, but I think this is cool. The frontiers of cognitive science are pushing into new territory. So what would a research agenda centered around this convergence look like? They suggest four avenues to pursue:
More informative diagnostic tasks. By this, I take them to mean we’ll need to develop unusual and clever ways to assess how AI models and humans tackle complex and novel reasoning tasks.
Mechanistic understanding. Also sometimes referred to as “mechanistic interpretability,” this refers to investigating the internal processes that LLMs use to generate their output. Sean Trott wrote a great primer on this here.
Developmental models trained on human-like inputs. We know that LLMs are trained on a massive amount of data that dwarfs what any human is exposed to in their lifetime. So how is it that children learn so much so quickly from such sparse data, and might we use insights from that very human process to inform generative AI models? As an aside, there’s a research institute to be built around this involving experts in learning, aka teachers, that philanthropy could make happen overnight if they weren’t focused instead on pushing dumb AI tutors upon kids. But I digress.
Cognitive models for predicting and explaining human behavior. Basically, all of the above should roll up into one unified model of cognitive architecture that explains all human behavior. One model to rule them all and bind them. You can’t say these researchers lack ambition!
I promised caveats, and here they are. First, although I’m perfectly willing to grant that LLMs have improved in noteworthy ways on complex cognitive tasks, I think they remain brittle and subject to ongoing taunting by clever humans (cue Colin Fraser). In other words, while they may have improved at linguistic tasks involving compositionality and productivity, it’s not clear to me whether such improvement generalizes much further than that specific domain. And if that’s true it means the connectionist paradigm remains rather narrow. To be fair, in their concluding commentary the authors note that they are “not claiming that AI systems based on neural networks match human cognition in coherence, veracity, or inductive biases—there are clear gaps for each,” but to my reading that comment doesn’t match the overall thrust of their claims.
I was surprised in particular to see Tom McCoy, he of Embers of Autoregression fame, endorsing this view, given friendly correspondence we’ve shared regarding the limitations of LLMs. So I reached out to him ask if his position on LLM capability had significantly changed. Here’s what he told me:
“This paper (Whither Symbols?) and Embers are operating at different levels of granularity. There are certain broad types of capacities that we always thought required symbols, e.g. compositionality. And now we have good evidence that artificial neural networks can possess those capacities,” McCoy said. From this, it follows that “artificial neural networks are promising candidates for things that could emulate human cognition. But then once you zoom into the details—not just having these abilities but using them in human-like ways—then lots of differences crop up…It’s plausible we might emulate human cognition, but there remains much work to be done to achieve that.”
That relieved me, to be honest, but also brings me to my second caveat—I worry it will be difficult to pursue the research agenda outlined in Whither Symbols? when the most powerful AI models, and many of the most capable computer scientists, are working for private commercial actors. Our progress on understanding cognition, both human and artificial, has already been impeded by our not knowing what data these models have been trained on, or how they’ve been “fine tuned.” At the same time, many of the capacity improvements of LLMs are no doubt due to the gobs of money that major developers have raised and spent in hopes of birthing digital sentience. The end result is that researchers end up studying these tools to try to figure out what’s gone in to their construction, when that information is already known to the developers. I find this annoying and so do many researchers in this space.
I’ll conclude with this: I believe AI skepticism in its best form hews closely to the classic Greek definition of skepsis—to inquire, consider, and investigate. I will be very interested to see if future scientific inquiry continues to bring the neurosymbolic and connectionist camps closer together, and if this comes to pass, whether it helps us to better understand ourselves and these strange tools we’ve built.
Interesting stuff, Ben. I appreciate the guide to work so far from my disciplinary home.
It seems to me you are describing a group of researchers who are skeptical about the intelligence of the latest LLMs, yet are nonetheless excited about the fact that transformer-based models are producing such impressive cultural outputs. They seem to think that the jump we say in 2022 may provide insights useful to the four avenues you outline, but not a meaningful step in that direction. Gotta say, #4 seems more dangerously silly than ambitious.
My window into this is Cosma Shalizi's musings, "You Can Do That with Just Kernel Smoothing!?!" and "You Can Do That with Just a Markov Model!?!!?!" at http://bactra.org/notebooks/nn-attention-and-transformers.html. These questions would seem to be a framework for putting aside some of the assumptions made by AI researchers who don't understand their inventions, and examining the capacity of GPTs to manipulate symbols.
So, treat questions of how GPTs do what they do as a set of interesting empirical questions about how they manipulate cultural data (symbols). The fact that in place of a research program working on this, we have a bunch of overcapitalized get-rich-quick schemes run out of Silicon Valley is going to make this harder than it needs to be.
I have become convinced that Charles Peirce is important, and that process philosophy (I've started using Whitehead's term instead of pragmatism, thanks to Kevin Munger), rather than logical positivism and its offspring, provides a better theoretical framework for thinking about all this.