Thank you for writing this. I recently designed and conducted my first DIY AI experiment, and the process reminded me that the framework of the scientific method is to disprove one's own beliefs. All that pre-work of theorizing and researching is meant to lead up to the experiment, and the experiment should be designed to independently prove whether or not that theorizing holds up in the real world. It is so fundamentally absurd that AI evangelists follow the inverse of this, it's mind-boggling to even begin correcting their logic. Definitely printing this one out to come back to when I need a sanity check to pull myself out of the churn ...
I was digging into ChatGPT's GeoGuessr skills in response to Scott Alexander's post (https://www.astralcodexten.com/p/testing-ais-geoguessr-genius) that Sam Altman retweeted as a sign that superintelligence is pretty much already here. From Alexander's data, I noticed ChatGPT seemed to do better on nature photos than urban photos, and wondered if it was because the natural environment has more truly unique features than the built environments do. I ran a larger data set to test my theory and ultimately found out I was thinking about ChatGPT's skills right -- it may be counterintuitive to the untrained eye, but there is far more variation in the natural world than built environments, so ChatGPT had an easier time deducing a photo was taken in rural CO vs. rural WY than it did if a photo was taken in Detroit MI vs. Buffalo NY -- and it's an incredible deduction and synthesis program when it has the available data, but it isn't more intelligent than what it has to work with. Definitely a fun way to test its limits!
That's so interesting, I hope you write that up somewhere! I'm struck by the fact that when I watch certain movies, I usually can tell when they were filmed in a different city than what's purportedly on screen. Be really interesting to run an experiment side-by-side to see if my intuitions there are correct, and how humans generally stack up at this task compared to AI.
Thank you -- I'm working on my second draft right now :) It's a very cool experiment for exactly that reason: It's natural for humans (and I assume other animals) to want to orient ourselves, so we naturally pick out what's unique about our environment and infer what those features might mean about where we are. ChatGPT is remarkable at identifying those features, which is the impressive part to me, and then provided there's information somewhere in its knowledge base (the prompt I'm using doesn't allow it to search the Internet on the first pass), it can put forth a guess that's very valid, either because it's actually correct or because it's at least a well-reasoned guess. Like, even if it can't tell the difference between Detroit and Buffalo in some photos, it's at least guessing cities in the same region with similar architectural features, not throwing out random guesses like Phoenix Arizona that couldn't possibly be right.
And there is a benchmarking site the GeoGuessr community has put together, and right now, it seems like LLMs beat the average player, but professionals who spend a lot of time developing their own knowledge bases still have an edge: https://geobench.org
I just wrote a long note on the affect heuristic, a bias that I think is pretty prevalent in AI discourse. I’m not necessarily accusing anyone in particular of anything, I just think it’s good to keep in mind. I’m just gonna paste it here:
“The psychologist Paul Slovic has found that when people have positive feelings toward a technology, they tend to list a lot of reasons in favor of it and few against it. When they have negative feelings toward a technology, they list a lot of risks and very few positives. What’s interesting about this is that when graphed, these results show an incredibly strong inverse correlation, even though this doesn’t make sense for how technologies work in the real world. Many technologies are high risk AND high reward, and many things are low risk AND low reward.
To show the strength of the underlying bias, Slovic took the group of people who disliked a technology and showed them a strong list of benefits from that technology. He found that after doing so, people began to downplay the significance of the risks of using that technology, even though their knowledge of the risks stayed the same. If they were acting fully rationally, their assessments of the risk should have stayed the same. Our minds are not designed to hold incredibly nuanced understandings of things.
I feel like I see this happening in AI discourse sometimes. People who are pro AI might downplay environmental costs and the dangers of piracy. People who are anti AI might dislike it because of its effects on the art industry, but they also might refuse to ask GPT questions because of the “environmental impact,” despite the fact that conversations with chatbots only take up 1-3% of the total electricity cost of AI, and streaming Spotify and YouTube for a non-insignificant amount of time are both much more environmentally costly than talking to GPT. In both cases, people create inaccurate assessments of the risks and benefits of using AI.
The takeaway from this note is to understand your biases and be careful about how you evaluate things. Even if something might have strong negatives, it also might have strong positives, and unless you’re careful, you might be evaluating things based on emotion, rather than reason.
(A summary of Slovic’s work here is in Thinking, Fast and Slow, by Daniel Kahneman, which is where I found it. The name of this bias is the affect heuristic.)”
Thanks for the comment. My argument here is contra Slovic insofar as I'm saying the methods of science are designed to be adversarial to unproven claims.
Totally spot on. I think the affect heuristic is at play in every other post you read about AI - it's really tough to be in awe of certain aspects of the technology while at the same time recognizing the massive risks and significant ethical compromises that are going into its development. At times, you almost feel like the skeptics and pessimists are rooting for the new models to get worse or fail while boosters keep pointing to the future where all the current issues will be solved. We all know deep down the reality will likely be much more complicated.
I think this is totally fair. I think part of Harry Law's post was a reaction to the larger wave of professors in academia who are still trying to ban AI out of existence in their classrooms and claiming that it cannot be used effectively to help students learn (jury very much still out on that issue despite recent studies). But I agree with your central premise - even the most avid AI boosters need to acknowledge that current AI is nowhere near AGI but I think that's a red herring anyway. Whether or not AI reaches what anyone will eventually agree is actual AGI, what we have right now, hallucinations and all, is still extremely powerful. Will be interesting to see how far GPT-5 advances the ball when/if it drops this summer.
I don't know what "powerful" means in the face of apparently ineradicable 'hallucinations' (i.e., confabulation of statements, claims, information, etc.) with regard to *intelligence*. I can see all sorts of powerful user incentives -- look, it's free/cheap; it can write my papers for me; it can write legal briefs for me (oops); we'll look cutting edge if we tell everyone we're using it; think of the money we'll save if it really could replace our employees/some of our employees. All of these can be very powerful user incentives, but I don't see them as having anything to do with intelligence.
I see Benjamin's argument here as a variation on a basic staple of critical thinking: you test claims rigorously. Offering confabulations is not a sign of intelligence, nor is accepting them uncritically.
Thanks for the comment. I'm glad *you* see AGI as a red herring, but as I'm typing this Mark Zuckerberg is announcing his new "superintelligence" unit, staffed largely by computer scientists poached from OpenAI, so achieving/surpassing human intelligence very much remains the stated goal of these Big Tech companies. And that's why I'm going to continue to delight in pointing out their many foibles!
Point out away - you misunderstand me. Regardless of the tech companies stated goals, arguing over what is or is not AGI is a waste of time while we already have powerful enough AI to wreak havoc throughout the standard education model.
I like this comparison - evaluating LLMs should be a lot like determining whether Koko the gorilla really could use language or not (she couldn't) .....
Thanks. Is that true? My hazy understanding was that Koko did develop a very rudimentary understanding of concepts that she could represent? I'd like to know more about this.
I don't know much more than what's on Wikipedia - "Koko's communication skills were hotly debated.[3][4][5] Koko used many signs adapted from American Sign Language, but the scientific consensus to date remains that she did not demonstrate the syntax or grammar required of true language."
But the point is that trying to answer this question brings up issues relevant to LLMs...
Koko did learn a number of ASL signs but my understanding is that her ability to combine/compose these signs was limited. Another impressive primate was Kanzi, for whom there are at least anecdotal reports of composing symbols to refer to novel concepts for which he didn’t have words (eg “slow lettuce” for “kale”): https://en.m.wikipedia.org/wiki/Kanzi
Maybe an even better example to your point though is clever Hans.
Yep, that animal cognition paper I cite leads off with a discussion of that famously brilliant horse! Thanks for the pointer on Kanzi, apparently he just passed away (sniff) but also managed to learn how to play Minecraft (wha?).
Thank you for writing this. I recently designed and conducted my first DIY AI experiment, and the process reminded me that the framework of the scientific method is to disprove one's own beliefs. All that pre-work of theorizing and researching is meant to lead up to the experiment, and the experiment should be designed to independently prove whether or not that theorizing holds up in the real world. It is so fundamentally absurd that AI evangelists follow the inverse of this, it's mind-boggling to even begin correcting their logic. Definitely printing this one out to come back to when I need a sanity check to pull myself out of the churn ...
Thanks! I'm curious, what was your DIY AI experiment?
I was digging into ChatGPT's GeoGuessr skills in response to Scott Alexander's post (https://www.astralcodexten.com/p/testing-ais-geoguessr-genius) that Sam Altman retweeted as a sign that superintelligence is pretty much already here. From Alexander's data, I noticed ChatGPT seemed to do better on nature photos than urban photos, and wondered if it was because the natural environment has more truly unique features than the built environments do. I ran a larger data set to test my theory and ultimately found out I was thinking about ChatGPT's skills right -- it may be counterintuitive to the untrained eye, but there is far more variation in the natural world than built environments, so ChatGPT had an easier time deducing a photo was taken in rural CO vs. rural WY than it did if a photo was taken in Detroit MI vs. Buffalo NY -- and it's an incredible deduction and synthesis program when it has the available data, but it isn't more intelligent than what it has to work with. Definitely a fun way to test its limits!
That's so interesting, I hope you write that up somewhere! I'm struck by the fact that when I watch certain movies, I usually can tell when they were filmed in a different city than what's purportedly on screen. Be really interesting to run an experiment side-by-side to see if my intuitions there are correct, and how humans generally stack up at this task compared to AI.
Thank you -- I'm working on my second draft right now :) It's a very cool experiment for exactly that reason: It's natural for humans (and I assume other animals) to want to orient ourselves, so we naturally pick out what's unique about our environment and infer what those features might mean about where we are. ChatGPT is remarkable at identifying those features, which is the impressive part to me, and then provided there's information somewhere in its knowledge base (the prompt I'm using doesn't allow it to search the Internet on the first pass), it can put forth a guess that's very valid, either because it's actually correct or because it's at least a well-reasoned guess. Like, even if it can't tell the difference between Detroit and Buffalo in some photos, it's at least guessing cities in the same region with similar architectural features, not throwing out random guesses like Phoenix Arizona that couldn't possibly be right.
And there is a benchmarking site the GeoGuessr community has put together, and right now, it seems like LLMs beat the average player, but professionals who spend a lot of time developing their own knowledge bases still have an edge: https://geobench.org
I just wrote a long note on the affect heuristic, a bias that I think is pretty prevalent in AI discourse. I’m not necessarily accusing anyone in particular of anything, I just think it’s good to keep in mind. I’m just gonna paste it here:
“The psychologist Paul Slovic has found that when people have positive feelings toward a technology, they tend to list a lot of reasons in favor of it and few against it. When they have negative feelings toward a technology, they list a lot of risks and very few positives. What’s interesting about this is that when graphed, these results show an incredibly strong inverse correlation, even though this doesn’t make sense for how technologies work in the real world. Many technologies are high risk AND high reward, and many things are low risk AND low reward.
To show the strength of the underlying bias, Slovic took the group of people who disliked a technology and showed them a strong list of benefits from that technology. He found that after doing so, people began to downplay the significance of the risks of using that technology, even though their knowledge of the risks stayed the same. If they were acting fully rationally, their assessments of the risk should have stayed the same. Our minds are not designed to hold incredibly nuanced understandings of things.
I feel like I see this happening in AI discourse sometimes. People who are pro AI might downplay environmental costs and the dangers of piracy. People who are anti AI might dislike it because of its effects on the art industry, but they also might refuse to ask GPT questions because of the “environmental impact,” despite the fact that conversations with chatbots only take up 1-3% of the total electricity cost of AI, and streaming Spotify and YouTube for a non-insignificant amount of time are both much more environmentally costly than talking to GPT. In both cases, people create inaccurate assessments of the risks and benefits of using AI.
The takeaway from this note is to understand your biases and be careful about how you evaluate things. Even if something might have strong negatives, it also might have strong positives, and unless you’re careful, you might be evaluating things based on emotion, rather than reason.
(A summary of Slovic’s work here is in Thinking, Fast and Slow, by Daniel Kahneman, which is where I found it. The name of this bias is the affect heuristic.)”
Thanks for the comment. My argument here is contra Slovic insofar as I'm saying the methods of science are designed to be adversarial to unproven claims.
Totally spot on. I think the affect heuristic is at play in every other post you read about AI - it's really tough to be in awe of certain aspects of the technology while at the same time recognizing the massive risks and significant ethical compromises that are going into its development. At times, you almost feel like the skeptics and pessimists are rooting for the new models to get worse or fail while boosters keep pointing to the future where all the current issues will be solved. We all know deep down the reality will likely be much more complicated.
I think this is totally fair. I think part of Harry Law's post was a reaction to the larger wave of professors in academia who are still trying to ban AI out of existence in their classrooms and claiming that it cannot be used effectively to help students learn (jury very much still out on that issue despite recent studies). But I agree with your central premise - even the most avid AI boosters need to acknowledge that current AI is nowhere near AGI but I think that's a red herring anyway. Whether or not AI reaches what anyone will eventually agree is actual AGI, what we have right now, hallucinations and all, is still extremely powerful. Will be interesting to see how far GPT-5 advances the ball when/if it drops this summer.
I don't know what "powerful" means in the face of apparently ineradicable 'hallucinations' (i.e., confabulation of statements, claims, information, etc.) with regard to *intelligence*. I can see all sorts of powerful user incentives -- look, it's free/cheap; it can write my papers for me; it can write legal briefs for me (oops); we'll look cutting edge if we tell everyone we're using it; think of the money we'll save if it really could replace our employees/some of our employees. All of these can be very powerful user incentives, but I don't see them as having anything to do with intelligence.
I see Benjamin's argument here as a variation on a basic staple of critical thinking: you test claims rigorously. Offering confabulations is not a sign of intelligence, nor is accepting them uncritically.
That's well said Paul, thanks for the comment.
Thanks for the comment. I'm glad *you* see AGI as a red herring, but as I'm typing this Mark Zuckerberg is announcing his new "superintelligence" unit, staffed largely by computer scientists poached from OpenAI, so achieving/surpassing human intelligence very much remains the stated goal of these Big Tech companies. And that's why I'm going to continue to delight in pointing out their many foibles!
Point out away - you misunderstand me. Regardless of the tech companies stated goals, arguing over what is or is not AGI is a waste of time while we already have powerful enough AI to wreak havoc throughout the standard education model.
I like this comparison - evaluating LLMs should be a lot like determining whether Koko the gorilla really could use language or not (she couldn't) .....
Thanks. Is that true? My hazy understanding was that Koko did develop a very rudimentary understanding of concepts that she could represent? I'd like to know more about this.
I don't know much more than what's on Wikipedia - "Koko's communication skills were hotly debated.[3][4][5] Koko used many signs adapted from American Sign Language, but the scientific consensus to date remains that she did not demonstrate the syntax or grammar required of true language."
But the point is that trying to answer this question brings up issues relevant to LLMs...
Koko did learn a number of ASL signs but my understanding is that her ability to combine/compose these signs was limited. Another impressive primate was Kanzi, for whom there are at least anecdotal reports of composing symbols to refer to novel concepts for which he didn’t have words (eg “slow lettuce” for “kale”): https://en.m.wikipedia.org/wiki/Kanzi
Maybe an even better example to your point though is clever Hans.
Yep, that animal cognition paper I cite leads off with a discussion of that famously brilliant horse! Thanks for the pointer on Kanzi, apparently he just passed away (sniff) but also managed to learn how to play Minecraft (wha?).