Why LLMs Seem Unreasonably Effective

LLMs work too well.

GPT is simply trained to replicate human language, the story goes, so it can’t be more than a stochastic parrot. ChatGPT (and the like) take a base LLM (like GPT) and use RLHF and other techniques to bend its answers to be actually helpful; instead of responding to ‘Translate this to French’ with ‘No, do it yourself’, it actually does the task given to it. But on this view, all we have done is bend stochastic parrots to helpful behavior.

That story does not resolve the question of why this can work. I can tell Claude to ‘make the button at the top left on Account green’ and it will first realize that it has no idea what I’m talking about, then read my project’s file structure, the README and whatever it needs to understand what I’m even talking about. It then goes ahead and makes the button green, and if it messes up the html in the process, it notices via a linter error, goes back and fixes it. Even with larger errors for bigger edits it can backtrack, change its plan, revert prior changes and try again. Claude can do meta-reasoning: it can model a lack of understanding (and thus go out and read files to fill the gap in context) and iterate on a plan based on trial-and-error. Of course it also often still breaks or fails at this, but more often than not it succeeds. LLMs are agentic.

Our engineering abilities have outpaced our scientific understanding. We do not have a solid scientific description of why this works so well. It should not. Scaling laws describe the curve (loss drops predictably with more data, compute, parameters), but a curve fit is not an explanation. It tells you that it works and by how much, never why.

Of course one can reason about how the LLM will first pick up on which chains of letters are a valid word and which are not, then how to compose these words into legal sentences (thus learning grammar). Moving up, it understands that ‘he’ in one sentence refers to a man in a prior sentence. Higher still, it gains the ability to model theory-of-mind: different characters in a story have different knowledge of a described situation, and thus will act on this limited knowledge. And higher and higher, moving from orthography, to grammar, to higher and higher meta-grammars modeling such.

This is roughly what grokking and mechanistic interpretability work is starting to find inside these models: actual circuits implementing these layers, one grammar sitting on top of the next.

But still, it remains unreasonable that all of this should fall out of next-token prediction: the ability to act agentic, to think, to solve math problems so far unsolved by humans; among them a counterexample disproving the 87-year-old Jacobian conjecture¹ and, separately, an infinite family of counterexamples disproving Erdős’s 1946 planar unit-distance conjecture, found by an OpenAI model².

I think this is because we are looking in the wrong location. The answer lies in a blind spot of computer scientists and engineers working on this: language is amazing. Let me explain.

Genes were conceptualized before they were physically found. For example in his work What Is Life?, Erwin Schrödinger encouraged the scientific community to look for ‘aperiodic crystals’ to be found within cells³. He believed those to be able to hold information in a durable fashion, being plausible to be written and read from by biological systems, and store information in a quantized fashion, required to explain the phenomenon described by Mendel about genetic inheritance.

In a similar vein, the term meme, while nowadays referring to funny internet pictures, was coined by Richard Dawkins in his work The Selfish Gene⁴: thought is comprised of a smallest unit, a mental gene, or meme. Any idea or thought is then comprised of these memes. Ideas can reproduce asexually (one person tells an idea to another), or sexually (a person holds two ideas in their mind and is able to bring them together into a bigger coherent idea). There are also mutation events (a person slightly misunderstands an idea, misremembers part of it, actively iterates on it). There is survival of the fittest (a stupid idea won’t be told further, while good ideas flourish). The actual value function varies: in trivial life forms it’s just don’t die and have sex; in higher life forms it’s impress a potential mate sufficiently so that you have sex; in memes it’s impress a mind sufficiently as not to be forgotten and spread, maybe even to be thought about and refined.

Humanity’s ascent from mere animals to God’s favorites is not a story of evolution, but of genetic-memetic co-evolution: a better brain allows better memetic evolution, better ideas provide more nutrients, thus allowing a larger brain to be sustained and filled by better memes. A child raised without human input, without access to human language, does not just fail to learn to speak; the few documented cases of such deprivation are stories of profound and lasting psychological damage.

"There are these two young fish swimming along, and they happen to meet an older fish swimming the other way, who nods at them and says, 'Morning, boys. How's the water?' And the two young fish swim on for a bit, and then eventually one of them looks over at the other and goes, 'What the hell is water?'"
— David Foster Wallace, This Is Water⁵

We don’t give human language enough credit. It is omnipresent; we take it for granted. We spend most of our working hours operating on language, and all the movies we watch are mostly conversations between humans with some actions sprinkled in to warrant more conversations. When we train LLMs off of the text corpus that humanity has produced, we allow AI to leapfrog millennia of cultural/memetic accumulation. Language is an amazing tool to efficiently transfer complex memetic structures between minds and operate on them.

And that accumulation is not just facts. The corpus does not only contain what we know; it contains how we think. Noticing that you don’t understand something and going to find out, breaking a problem into steps, checking your work, backtracking when a plan fails: these are memes too, procedures of thought that humanity refined over millennia and wrote down a million times over. When Claude reads my file structure before touching that button, it is not doing something beyond its training data. It is doing something its training data is saturated with. A stochastic parrot repeating words would be unremarkable. But a parrot repeating the accumulated thought-patterns of an entire civilization? That is what we call thinking.