How Transformers Turn Words Into Meaning

Arshavir Blackwell, PhD

Welcome back to Inside the Black Box. I'm Arshavir Blackwell, and today we're going to look closely at how a model like GPT turns a handful of tokens into something that carries structure, context, and meaning. If you've been following along, you know we've spent the last few episodes wandering through high-dimensional geometry, sparse autoencoders, and the different ways models carve up abstract space. Today, we're rewinding all the way to the beginning: how the model represents tokens before it does anything intelligent with them.

Arshavir Blackwell, PhD

Here's the key idea: tokens are converted to vectors—mathematical objects living in hundreds or thousands of dimensions. In spaces that large, many directions behave almost independently, which gives the model room to spread out different properties.

Arshavir Blackwell, PhD

A token embedding is the model's starting hypothesis about a word: broad early features like "name-ish," "verb-ish," "physical object-ish," "abstract-ish." These aren't final categories; they're more like rough coordinates the model begins with.

Arshavir Blackwell, PhD

But even with all those dimensions, the space eventually runs out of clean places to put everything. The model ends up representing multiple features in the same directions—a phenomenon called superposition. Instead of having one direction for one concept, you get several concepts sharing a direction, and they disentangle only when the model needs them. It's one of the reasons deep layers are so hard to interpret: everything is overlapping, compressed, and context-dependent.

Arshavir Blackwell, PhD

Of course, content alone isn't enough. Order matters. Transformers need a way to encode where each token appears in the sequence. Early models used hand-crafted sine–cosine waves for that. Newer ones use learned positional embeddings or RoPE—rotary position embeddings—which rotate vectors in a controlled way so the model can track relative positions.

Arshavir Blackwell, PhD

Once you have that, the model sends everything into attention—the place where structure begins to show up. Each token is projected into three forms: query, key, and value. It sounds abstract, but the roles are pretty intuitive: the query encodes what the token is looking for, the key encodes what it offers, and the value is the information it delivers if another token attends to it.

Arshavir Blackwell, PhD

Take the sentence "Alice gave a book to Bob." One attention head might specialize in recipients, so the query for "gave" could align strongly with Bob's key. Another head might track subjects and attend primarily to "Alice." Another might look for objects and pick out "book." And if the sentence continued with the phrase "and then she…," a later head might identify the link between "she" and "Alice."

Arshavir Blackwell, PhD

Attention heads often specialize. Some tend to follow syntax. Some track reference. Some key in on position or sequence patterns. We know this not just from theory but from empirical work—transformer circuits research from Anthropic, EleutherAI, and academic labs has mapped out heads that appear to lock onto these roles, though in practice the picture is often messier than clean functional labels suggest.

Arshavir Blackwell, PhD

After each round of attention, the model blends the new information with the old through the residual stream—a running additive ledger that carries the original token representation forward at every layer. The model never fully overwrites anything; it just keeps adding structure on top.

Arshavir Blackwell, PhD

By the time you stack multiple layers—attention, feed-forward transformations, residuals—the representation becomes richer and more entangled. Early layers tend to capture grammar and basic relationships. Middle layers start fusing roles, events, and signals of meaning. Deeper layers carry highly compressed mixtures of everything—the token itself, its relationship to others, the broader context, and world knowledge.

Arshavir Blackwell, PhD

To make that concrete, consider the phrase "…and then she…" at the end of a sentence about Alice giving Bob a book. By the final layer, the vector for "she" holds many threads at once: the identity of "she," the expectation for what typically follows, the sentiment of the scene, and the grammar of the earlier clause.

Arshavir Blackwell, PhD

The model then runs this vector through the unembedding matrix—a learned projection that compares the final representation against the entire vocabulary. In some architectures this matrix is tied to the input embeddings; in others it's trained separately. Either way, whichever vocabulary item the vector aligns with most strongly—maybe "smiled" or "thanked"—gets the highest logit. That's how it chooses the next token.

Arshavir Blackwell, PhD

What is compelling is how simple the underlying rule is—predict the next token—yet how much structure emerges from it. Layer by layer, the model builds something that looks a lot like comprehension. It's a geometric process that ends up behaving surprisingly cognitively.

Arshavir Blackwell, PhD

We'll dig deeper into how these internal vectors can be interpreted—and sometimes manipulated—in future episodes. But for today, that's our tour of how transformers move from raw tokens to structured meaning.This has been Inside the Black Box; I'm Arshavir Blackwell. Thanks for listening, and I'll see you next time.