The Thinking Machine

Forget “artificial intelligence” for a moment. Consider a humbler object: a box with an input, an output, and a few million adjustable knobs. Feed it a photo; it outputs a number between 0 and 1 meaning “how cat-like is this?”. With the knobs set randomly, its answers are garbage. The entire field of machine learning is the study of one question: how do you turn the knobs so the answers stop being garbage?

You could try knobs at random — hopeless with millions of them. You could reason about what each knob should do — also hopeless; nobody can articulate what knob #4,201,337 contributes to cat-ness. The trick that works is embarrassingly pragmatic. Define a single number that measures how wrong the box currently is, averaged over many examples. Call it the loss. Now the vague dream of “learning” becomes a concrete task from calculus: find knob settings that make the loss small.

So what's inside the box? Here the brain lends AI its most productive loan. A neuron, in this context, is almost insultingly simple: it takes several input numbers, multiplies each by a knob (its weight), adds them up, and passes the total through a threshold — staying quiet below it, firing above it. One neuron is a weighted vote: “given what I'm hearing, do I speak up?”

The power comes from composition. Arrange neurons in layers, each layer listening to the previous one, and something lovely happens: early layers learn to detect simple things (an edge, a color gradient), middle layers combine those into parts (an eye, a whisker), late layers combine parts into concepts (a cat). Nobody designs this hierarchy. It emerges, reliably, from turning knobs to reduce loss — one of the first genuinely surprising empirical facts of the field.

One question remains, and it's the crucial one. The fog tells the hiker which way is downhill — but with millions of knobs, how do you compute the downhill direction for every knob at once, without trying them one by one?

Everything since 1986 — including the trillion-knob systems you talk to today — is this exact recipe: a stack of weighted votes, a loss, blame flowing backwards, knobs nudged downhill, repeated until the money runs out. What changed is not the idea. What changed is the scale — and, as we're about to see, one architectural invention that made scale worthwhile.

Language has a property that tortured early neural networks: the meaning of a word depends on other words that may be arbitrarily far away. In “the keys to the old cabinet are missing”, the verb agrees with keys, six words back. Pre-2017 networks read text like a person hearing it through a keyhole, one word at a time, carrying a single summary of everything so far. By word forty, the summary was mush. And because each word had to wait for the previous one, the process couldn't be parallelized — poison, in a field whose entire strategy was to throw more computation at the problem.

The transformer's proposal: stop reading in order. Let every word look directly at every other word, all at once — and let the network learn which words are worth looking at. That mechanism is called attention, and the 2017 paper's title was literal: attention is all you need. The recurrence, the keyhole, the mush — all deleted.

Why did this change everything? Three reasons, in ascending order of importance. First, it works better: no more keyhole, no more mush — long-range relationships are one direct hop away. Second, it's parallel: all words are processed at once, which is exactly the workload GPUs were built for. Training that would take months of sequential reading became weeks of parallel matrix arithmetic.

Third — and this was the genuine surprise — the transformer scales. Most architectures improve with size and then plateau. Transformers kept improving, smoothly and predictably, across seven orders of magnitude of compute. Researchers plotted loss against model size, dataset size and compute, and got clean straight lines on log-log paper — the famous scaling laws. That predictability transformed AI from a science experiment into an industrial program: for the first time, you could budget for intelligence.

And what task do you train this architecture on? The dumbest one imaginable: predict the next word. Take all the text you can find, hide the ending of each sentence, and turn knobs until the model guesses well. The task sounds trivial. It is not. To predict the next word of a detective novel's final page, you must track the plot; to predict the result of “2+2=”, you must do arithmetic; to continue a Python file, you must model a programmer's intent. Prediction is a lever that quietly forces the machine to model the world that produced the text.

One piece of vocabulary before we go on, because everything from here is counted in it. Models don't actually work in words but in tokens — chunks of text averaging about three-quarters of a word, so that a long or unusual word arrives in several pieces. Where this essay says word, the machine says token; the distinction rarely matters for intuition, but every number you will see — context lengths, prices, training set sizes — is measured in tokens.

One honest caveat. Attention's cost grows with the square of the text length — every word attending to every word — which is why “context windows” (how much text a model can consider at once) became a battleground: from 2,000 tokens in GPT-3 to a million or more today, via a great deal of clever engineering. Keep this picture — a next-word predictor with learned attention, scaled beyond reason — because everything in the next section is about what such a thing does and doesn't become when you train it.

Start with the central tension of all learning, human or machine. A student who memorizes past exam papers word-for-word will ace any repeated question and fail any new one. A student who extracted the principles generalizes. Machine learning lives entirely inside this tension: a model with enough capacity can memorize its training data perfectly — and be useless. The whole craft is getting a model to compress experience into principles instead.

Here is what makes large language models philosophically interesting rather than just big: they generalize when theory said they should memorize. A trillion-knob model trained on text has, on paper, the capacity of our degree-15 polynomial a billion times over. Classical statistics predicts catastrophic overfitting. Instead, these models transfer to problems they never saw — and researchers have even watched small networks flip suddenly, deep into training, from pure memorization to a clean general algorithm (a phenomenon literally named grokking). Why the descent through a trillion-dimensional fog reliably finds solutions that generalize is still not fully understood. It is one of the field's honest open problems: we can build it, and steer it, without entirely knowing why it works.

In practice, a modern assistant is made in stages — each one a different answer to the question “what should the loss measure?”

Now the questions everyone actually asks — can it imagine? does it have an unconscious? does it understand? These deserve better than the two lazy answers on offer: “it's just autocomplete” (which explains nothing about the behaviour you can observe yourself) and “it thinks like us” (which assumes exactly what needs proving). The right tool is a ledger: for each claim, what the evidence — mostly from interpretability, the young science of opening these networks up and tracing what happens inside — actually supports.

The meta-lesson of the ledger: the interesting frontier is not “is it conscious, yes or no” but the growing catalogue of mechanically verified cognitive structure — concepts, plans, unfaithful self-reports — in systems built from nothing but next-word prediction. That catalogue should update you in both directions at once: these systems are more than a lookup table, and the words we borrow from human minds — imagine, know, want — fit them only loosely, like clothes tailored for someone else.

A useful way to hold them together: the LLM is an engine, and a mixture of experts is how that engine got enormous without becoming proportionally expensive to run; the context window is its working memory; inference is the engine running, and a chain of thought is it running longer on purpose. Fine-tuning and RAG are two ways of giving it your knowledge — surgery versus an open book. Around the engine, people build agents inside harnesses, connected to the world through MCP, working mostly in the CLI, carrying skills, sometimes driven by vibe coding, and increasingly pointed at long-running goals. One paragraph — that's the entire stack. The cards fill in the detail.

One axis organizes the whole landscape: open versus closed weights. A model's weights are the learned knob-settings — the crown jewels. Publish them and anyone can run, inspect, fine-tune, and build on your model forever; you gain ubiquity and surrender control. Keep them behind an API and you retain control, safety gates and margins — while betting that capability, not availability, is what wins. The American frontier mostly chose closed (with Meta the great exception); the Chinese ecosystem overwhelmingly chose open, partly as strategy under sanction. That divergence — who chose openness, and why — may matter more to how AI spreads through the world than any benchmark.

It's tempting to think of AI as ethereal — models in a “cloud”, weightless and everywhere. The truth is closer to heavy industry. Every response you get from a frontier model is the terminal event of a chain that runs through one Dutch company's light machines, one Taiwanese company's factories, three memory makers, largely one GPU designer, a handful of gigawatt-scale buildings, and a power grid straining to keep up. Each link is a place where physics, capital, and state power meet. Walk the chain:

Once you see the chain, the geopolitics reads itself. Since 2022, the United States has used export controls as its main lever to slow China's AI progress: no EUV machines, no leading-edge chips, no advanced HBM — a policy possible only because the chokepoints above happen to sit in allied territory. The results are genuinely mixed, and instructive. Controls created real ceilings: Chinese labs train on restricted or domestic silicon and it costs them. But constraint bred ingenuity — DeepSeek's efficiency breakthroughs were forged directly under sanction pressure — and it accelerated the very thing the policy feared: a determined, state-backed program to rebuild the entire chain domestically, from Huawei's accelerators to CXMT's memory. Meanwhile the policy itself oscillates — bans, then licensed H200 sales, then new thresholds — because every restriction also costs American firms their largest export market. There is no stable equilibrium here yet; supply-chain planning that once looked five years out now has roughly a twelve-month policy horizon.

And looming over the chessboard, one square matters more than the rest: Taiwan. The island produces the overwhelming majority of leading-edge chips; a blockade or war would not merely disrupt AI — it would halt the digital economy generally, which is precisely why some strategists call TSMC a “silicon shield” and others a single point of failure for civilization's compute. The tens of billions being poured into Arizona, Dresden and Kumamoto fabs are best read as the world buying insurance — slowly, expensively, and years behind demand.