How intelligent is AI? The Story of Liam
Working hands-on with AI, it’s easy to feel a sense of cognitive dissonance. As one navigates its enormous strengths and sizable weaknesses, the public narrative seems weirder and weirder.
Pundits say, “Pretty soon we won’t need software engineers at all!” as I’m wondering, “Should I give Claude a fourth try at writing code that actually works? It’ll probably be quicker just to do it myself.” Journalists write about imminent superintelligence as I say, “Hey, ChatGPT, brighten up this image but don’t change anything else about it,” knowing full well it’s going to give me an entirely different image.
It’s easy to call me a Luddite, a reactionary, clinging to the old ways in the face of inevitable change. (Although: I question a stance that reacts to skepticism by discrediting the skeptics.) I can tell you I’m far from alone amongst AI practitioners. I can mention that the people claiming imminent superintelligence have been doing so for years while adjusting their timelines. I can remind you that I’m not a pure skeptic — as I said in “Practical AI for Product People,” today’s AI is “the biggest innovation in tech since the smartphone.”
And: if you’re not up to your elbows in AI on a daily basis, I can help you understand why and how it’s simultaneously powerful and limited.
In my last article I described LLMs (Large Language Models — the core of modern AI) using a metaphor of infinite interns:
It’s like having an infinite number of forgetful interns. (Props to Benedict Evans for the analogy.) They’re resourceful and tireless, but their skills are limited and they forget everything they do once they’ve done it.
It’s a versatile analogy, but a bit inaccurate: a real intern might be inexperienced and forgetful, but it would be silly to describe her as lacking human intelligence!
So here’s a new metaphor: more cumbersome, but also more accurate and relevant to understanding the dissonance between hype and reality.
It begins with Liam, who loves to sing. The melodies, the patterns…the mathematical order of the music is exhilarating for him.
One December, his choir teams up with the local symphony to perform An die Freude, the fourth movement of Beethoven’s Ninth Symphony. Liam is enraptured. The cadence and sound of the German language are thrilling to him. He doesn’t know what it means — he doesn’t want to know! He loves the purity of the pattern.
He seeks out other music in German: Brahms, Mendelssohn, Bach, Hasselhoff, Rammstein. Folk songs and lullabies. It’s not enough. He devours books: fiction, nonfiction, periodicals — it doesn’t really matter because the topic is irrelevant. He cares about the melody, the pronunciation, the system.
Liam works in customer service. One day, a ticket arrives in German. Naturally, he has no idea what they’re asking and he’s about to send back a canned response when he realizes…he knows what comes next. He’s seen so much German, spanning so many topics, learned the patterns so well that the question in the ticket suggests its answer. He takes a chance: he types and sends it. And the customer is happy! Well, he thinks the customer is happy…there are some smiley emojis in their response.
As it turns out, 20% of his company’s customers are German and Liam is suddenly very busy. He gets a raise and a promotion.
Liam grabs a beer with his friend Julia. Next to them at the bar is a group of German tourists, arguing animatedly about something in their language. Liam has that now-familiar feeling…he knows what comes next in their conversation. He can’t help himself: he jumps in. They’re thrilled to include him. Liam finds he’s doing most of the talking — he’s a bit self-conscious but the responses leap into his brain and the tourists don’t seem to mind.
After nearly an hour, his new friends pay and leave. Julia has been sipping her beer and listening: she’s been taking German classes for the past year. She looks at Liam with surprise and says, “How do you know so much about marine biology? You sound like an expert!”
Liam looks confused for a moment. “Oh! Is that what we were talking about? I had no idea.”
Real people don’t have Liam’s memory or infinite capacity for pattern-matching…but computers do, and modern LLMs operate much like Liam does here.
So, how close are LLMs to human-like intelligence? Well, how fluent is Liam in German?
