What Are Word Embeddings, Anyway?

At the end of the tokens article , I left you hanging on one question. I said each token ID gets swapped for a row of the embedding matrix, a dense vector of floats where meaning starts to live. And then I asked: where did that matrix come from? Nobody sat down and typed 12,288 numbers describing "strawberry."

This article is that answer. But before we get to who fills in the numbers, we need to agree on what the numbers are even trying to do.

Start with something you do effortlessly. If I say "I am going to the bank" and "I am sitting by the bank of a river," you don't need me to explain anything.

Same word, two meanings, and you picked the right one both times without noticing you did it. Language for us isn't just words, it's context, emotion, all the surrounding machinery. You know I'm annoyed from the way I say "I'm fine."

A machine has none of that. A machine has bits. ON or OFF, 1 or 0. Every character on your screen, every emoji, every number, all of it is bits underneath. So the question becomes: how do you take a pile of bits and bake in the notion that two things are similar? That "rage" and "anger" belong together and "rage" and "delighted" don't?

Let's play a game

Here's a hypothetical experiment. Say we assign the number 5 to the word "Happy." Now I ask you to assign a number to "Sad," and I give you two options: 7 or -5.

Think about it for a second.

Most people pick -5, and the reasoning is lovely. The magnitude (5) says "this is an emotion, same category as Happy," and the sign says "but flipped." One number, and we've already smuggled in two pieces of meaning: category and polarity.

So what would 3, 4, 6 represent? Probably things in the neighbourhood of happiness. Content. Pleased. Delighted maybe at 6. We're placing similar words close together on a number line. That's the entire core idea of embeddings, and you just did it with primary school arithmetic.

But now push on it. One dimension gives you exactly two levers: magnitude and sign. What if I want to capture that "king" and "queen" share royalty but differ in gender? Where do I put "peasant"? One axis runs out of room almost immediately.

Fine, give me more dimensions

Say we allow two numbers per word instead of one. Now maybe the x axis captures gender and the y axis captures royalty. "King" might be [1, 3]. Then [-1, 3] would be something equally royal but opposite gender. A queen. And a peasant sits somewhere near [1, 0], all the royalty drained out.

Three dimensions, more room. A thousand dimensions, more room still. Every extra axis is another nuance we could represent: tense, formality, whether the thing is edible, whether it's abstract.

And this is all a "vector" is. [5] is a 1-dimensional vector, a point on a number line. [1, 3] is a 2-dimensional vector, a point on a plane. [0.2, -0.7, ..., 0.4] with 768 entries is a point in a 768-dimensional space, which you cannot picture and neither can I. Past three dimensions (four if you're feeling clever and count time), human visual intuition taps out. The maths doesn't care. A point is a point.

A word embedding is exactly this: a word's assigned point in an N-dimensional space, arranged so that similar words end up near each other.

One honesty checkpoint before we move on, because this is an educational blog and the gender-axis, royalty-axis picture is a simplification that will mislead you if you take it literally. In real trained embeddings, the individual dimensions mean nothing you can name. Dimension 412 is not "royalty." Dimension 87 is not "gender." It is not this clear, the dimension might represents something abstract, or a group of dimension might end up conveying something. The gender/royalty picture is just an example, an intuition to help us understand this black box. So, keep the intuition, discard the labels.

How do we know two points are "close"?

In one dimension it was easy, 5 and 6 are obviously near, 5 and -5 obviously aren't. In 768 dimensions we need something more mechanical.

The natural first instinct, and the one I'll be honest I reached for too, is the dot product. Multiply the vectors entry by entry, add it all up, big number means similar. And the dot product is the right raw ingredient. But let's just think for a moment, is it all that? a raw dot product does not range from -1 to 1. It's unbounded. [10, 0] dotted with [10, 0] is 100 (10 * 10 + 0 * 0). A word with a "louder" vector would look more similar to everything just by being long.

What actually ranges from -1 to 1 is cosine similarity, which is the dot product after you divide out both vectors' lengths:

1cos(A, B) = (A · B) / (|A| × |B|)

Geometrically it's the cosine of the angle between the two vectors. Direction, with the loudness removed. 1 means pointing the same way, -1 means opposite ways, 0 means perpendicular, nothing in common.

Small worked example:

1cat    = [2, 3]
2kitten = [3, 4]
3car    = [3, -1]
4
5cat · kitten = (2)(3) + (3)(4) = 18
6|cat| = √13 ≈ 3.61,  |kitten| = √25 = 5
7cos(cat, kitten) = 18 / (3.61 × 5) ≈ 0.998   → nearly identical direction
8
9cat · car = (2)(3) + (3)(-1) = 3
10|car| = √10 ≈ 3.16
11cos(cat, car) = 3 / (3.61 × 3.16) ≈ 0.26     → not much in common

So 0.998 says "these two live in the same corner of the space," 0.26 says "acquaintances at best," and a negative value would say "pointing away from each other." That's the whole similarity machine. Two lines of maths.

Okay. But who fills in the numbers?

Here's the gap. We've said "king" is a point, "queen" is a nearby point, and cosine similarity measures nearness. But who decided "Happy" is [-0.2, 0.3, 0.7, -0.1, ...]? Did someone sit and hand-assign vectors like we did with Happy (5) and Sad (-5), for a hundred thousand tokens, across 12,288 dimensions?

No. That would be utterly painful to watch, and worse, it wouldn't work, because no human knows what the "right" 12,288 numbers for strawberry are.

You don't need to know what "milk" means. You just need to notice which words it shows up next to. "Drink," "cold," "glass," "cow," "spilled." And here's the kicker: "water" shows up next to almost the same crowd. Same company, so by Firth's rule, similar word. Nobody defined milk. Its neighbours defined it.

So here's the scheme, and it's the same flavour of trick as BPE from the last article: take something dumb and countable, do it a few billion times, and structure falls out.

Step one. Give every word in the vocabulary a random vector. Genuine garbage. "King" starts as noise, "cabbage" starts as noise.

Step two. Slide a window across a mountain of real text and play a prediction game. In the word2vec version of this game (the 2013 Mikolov papers, linked below), you take a center word and ask the model to guess its neighbours. Take a toy corpus:

1I drink cold milk
2I drink cold water
3I drive a red car

Window lands on "milk." The game says: given the vector for "milk," predict that "drink" and "cold" are nearby. Right now the vectors are random garbage, so the prediction is garbage.

Step three. Measure how wrong the guess was, and nudge. Push the "milk" vector a tiny step in whatever direction would have made "drink" and "cold" more predictable, and push the vectors it wrongly favoured a tiny step away. This nudging is gradient descent, and I'm compressing a lot here (the actual objective, and tricks like negative sampling that make it cheap enough to run, are covered properly in the references). But the direction of the story is exact: wrong guess, small correction, next window.

Step four. Repeat a few billion times.

Now watch what happens to "water." The window keeps landing on it inside "drink cold ___," the very same slot "milk" keeps sitting in. So "water" keeps getting nudged toward predicting "drink" and "cold" too. Two words, receiving thousands of near-identical nudges, from near-identical contexts, migrate to the same corner of the space. Nobody told the model that milk and water are both drinks. They just kept the same company, and the nudges did the rest. Meanwhile "car" is off being nudged by "drive" and "red," drifting to its own corner.

That's it. That's who fills in the numbers. Not a person, a process. The vectors are the residue of winning a prediction game.

The trick, and the fine print

Once you have this trained space, you can do arithmetic that feels frankly illegal:

1vector(king) − vector(man) + vector(woman) ≈ vector(queen)

Take king, subtract the "man-ness" direction, add the "woman-ness" direction, and the nearest word to where you land is queen. It also works for things like Paris − France + Italy ≈ Rome. The differences between vectors have themselves become meaningful directions, which nobody asked for. It simply emerged.

And here's the twist I owe you from the start of this article. Remember Happy at 5 and Sad at -5, opposite signs for opposite meanings? Real embeddings refuse to cooperate. In a trained space, "happy" and "sad" sit close together, high cosine similarity. Why? Look at the mechanism we just built. "I am ___ today." "She seemed so ___." "That movie made me ___." Happy and sad keep the same company almost perfectly, so the nudges drag them together. The space learned "these words play the same role in a sentence," which is a real kind of similarity, just not the one your gut ordered.Antonyms are the roommates of embedding space. Once you know the numbers come from context prediction, this stops being a bug and becomes an obvious consequence.

(Word2vec isn't the only recipe, by the way. GloVe, out of Stanford in 2014, gets to a very similar space by directly crunching global co-occurrence counts instead of playing the sliding-window game. Different route, same destination: company determines position.)

So is this what's inside GPT?

Almost, with one important update. Modern LLMs don't bolt on a pre-trained word2vec. The embedding matrix from the tokens article, that vocab_size × d_model grid, starts as random noise like everything else and gets trained jointly with the rest of the model, nudged by the same next-token prediction game the whole network is playing. Different game rules, exact same principle: prediction pressure sculpts the geometry. Tokens used in similar company converge, and the matrix rows become the meanings.

If you want to actually see one of these spaces instead of trusting my hand-waving, open the TensorFlow Embedding Projector, load word2vec, and search for a word. Watching "piano" light up next to "violin" and "cello" does more for the intuition than three more paragraphs from me would.

The crack in the whole thing

Now let me break what we just built, because the break is the doorway to the next article.

These are called static embeddings, and this is their ceiling.

What you'd want instead is obvious once you say it out loud: the vector for "bank" should shift depending on the sentence it's currently in. Near "river," slide it toward wet things. Near "deposit," slide it toward money. A vector that gets rewritten on the fly by its neighbours.

That mechanism exists. It's called attention, it's the reason the T in GPT stands for Transformer, and it's what turns these frozen vector entries into something that reads context the way you do. That's the next article. Subscribe to the newsletter so it lands in your inbox.