What Are Tokens in an LLM, Anyway?

Ask ChatGPT how many r's there are in the word "strawberry."

For the longest time all the previous LLM models got this wrong, and people passed those screenshots around as proof that these models are dumb. I don't think that's what it proves. I think it proves something much more specific, and much more interesting.

The thing is that the model never saw the word "strawberry." It saw three numbers. Wait What? Yup, and in fact I've made a video on it as well, you can check that out.

So let me give you a high level overview of what we're covering here. We'll look at what tokens actually are, how they get made, why anyone thought this was a good idea in the first place, and what happens to a token after it's made. By the end you'll know why a machine that can write working Python code can't count letters.

Sounds interesting? Let's get into it.

So what actually happens when you hit enter?

The first thing that happens has nothing to do with intelligence. Your text goes through a tokenizer, which chops it into pieces and swaps each piece for a number.

So the word, "strawberry" comes out as three tokens. Not four, not one. Three. Well, to be honest, it depends on the nature of tokenizer.

And here's the bit that got me the first time I checked. The split isn't where you'd put it. You and I would break that word as straw + berry, because that's how the word is built, but the tokenizer doesn't care how the word is built. It breaks it as:

1str | aw | berry

Hang on to that one. It explains the letter counting thing at the end.

If you'd rather see it yourself than take my word for it, it's about four lines of python code or you can visit TikTokenizer :

1import tiktoken
2
3enc = tiktoken.get_encoding("cl100k_base")  # GPT-4's tokenizer
4ids = enc.encode("strawberry")
5print(ids)
6print([enc.decode([i]) for i in ids])

tiktokenizer

Try it with your own name. Try it with a Hindi sentence. Try it with an emoji. The results start telling you things, and we'll come back to why.

Why chop up the words at all?

You might ask, why even chop them? and that's a fair thing to ask. Why not skip this whole step?

Well, look at the two obvious ways to do it and we'll see why those don't work.

1. Give every word its own token.

It is Clean, intuitive, matches how we think about language. Now count the words. English alone runs into the millions once you include names, typos, plurals, slang, and every product name anyone has ever invented. Your lookup table gets enormous, and most of those millions of entries show up three times in training, which is nowhere near enough to learn anything from.

That's the boring problem of memory. Here's the real one. What happens when the model meets a word that wasn't on the list? Not a rare word, just a new one. A word someone coined last Tuesday. Like "Plumberry"? A typo. Your surname. There's no number for it, so the model gets nothing at all. A hole in the input, and no polite way to fail.

2. Give every character its own token.

Now the vocabulary is tiny, maybe a couple of hundred symbols, and nothing is ever missing. Because every word will always be amalgam of those characters. Sounds like a fix.

Inside the model, every token has to look at every other token in your input. That's what attention does, I am going to be writing a detailed article on that, stay tuned for that. If a token has to look at every other token, it means six tokens means roughly 36 of those little comparisons (encoder architecture). Twelve tokens means 144. The work doesn't double when the input doubles, it goes up four times.

Now take a sentence like "The cat sat on the mat." As words, that's 6 tokens. As characters, it's 22. So the sentence got about four times longer, which means the model does something like sixteen times the work. Same sentence. Nothing gained.

And there's a second cost. The model now has to figure out that c,a, t put together means something, and it has to do that before it can start learning anything about actual cats. You've handed it a spelling homework assignment on top of everything else.

So one extreme gives you a vocabulary you can't afford and gaps you can't fill. The other gives you sentences you can't afford and a model doing spelling drills.

And that is where Subword tokenization comes, it sits in the middle.

If you've spent any time with German language, you already have the intuition for this. A glove is a Handschuh, a hand shoe. A speed limit is a Geschwindigkeitsbegrenzung, which is just Geschwindigkeit (speed) and Begrenzung (limitation) welded together. German doesn't invent a fresh word for every concept, it keeps a box of pieces and assembles what it needs. Subword tokenization does the same thing, except the pieces get picked by counting (an algorithm we are gonna cover next) rather than by meaning.

So how does it decide where to cut?

It starts with a training corpus. In practice that corpus is a large chunk of the internet, but let's keep it small enough to do by hand. Say the entire language, as far as our tokenizer is concerned, is one book. The Lord of the Rings. Of course we won't be working on the whole book, but let's just say the book only contained 6 words.

Say we pull out these words along with how often they appear:

1("gold", 10), ("golden", 5), ("hold", 12), ("holds", 4), ("bold", 5), ("gollum", 8)

Step one, smash everything down to characters. That's our starting vocabulary:

1Vocabulary: b, d, e, g, h, l, m, n, o, s, u
2
3("g" "o" "l" "d", 10)
4("g" "o" "l" "d" "e" "n", 5)
5("h" "o" "l" "d", 12)
6("h" "o" "l" "d" "s", 4)
7("b" "o" "l" "d", 5)
8("g" "o" "l" "l" "u" "m", 8)

Now we look at every neighbouring pair and count how often each one turns up.

ol is in every single word: 10 + 5 + 12 + 4 + 5 + 8 = 44. Nothing else is close (ld gets 36, ho gets 16). So we merge it and add it to the vocabulary.

1Vocabulary: ..., ol
2
3("g" "ol" "d", 10)   ("g" "ol" "d" "e" "n", 5)   ("h" "ol" "d", 12)
4("h" "ol" "d" "s", 4)   ("b" "ol" "d", 5)   ("g" "ol" "l" "u" "m", 8)

Count again. Now ol next to d turns up 36 times, more than anything else. So we merge that too.

1Vocabulary: ..., ol, old
2
3("g" "old", 10)   ("g" "old" "e" "n", 5)   ("h" "old", 12)
4("h" "old" "s", 4)   ("b" "old", 5)   ("g" "ol" "l" "u" "m", 8)

Now, stop and look at what just happened there. The tokenizer found"old". Nobody told it that "old" is a word in English. It doesn't know what old means. It merged those two characters because they kept showing up next to each other and merging them was the cheapest win available.

Keep going. h plus old is 16, g plusold is 15. So hold becomes a token, then gold becomes a token.

1Vocabulary: ..., ol, old, hold, gold
2
3("gold", 10)   ("gold" "e" "n", 5)   ("hold", 12)
4("hold" "s", 4)   ("b" "old", 5)   ("g" "ol" "l" "u" "m", 8)

Two things worth noticing in that final state.

"golden" never became a single token. It'sgold + e + n, because it wasn't frequent enough to earn a slot of its own. It gets assembled from parts, exactly like Handschuh.

And Gollum is still lying there in pieces: g + ol + l + u + m. He only shows up 8 times, so he never won a merge. Which tells you a lot about the whole scheme. Frequent text is cheap, rare text is expensive, and the tokenizer picked its favourites long before you turned up with a prompt.

This is a very high level walkthrough, by the way. If you want the full thing, the Hugging Face tokenizer course covers it properly and I'll link it below in the references.

But why is it called byte pair encoding though?

You'd guess it's because a character takes up a byte, and we are pairing them so byte-pair, Right? Wrong!

The name comes from the original algorithm, which had nothing to do with language. Philip Gage published it in 1994 as a way to compress files. You scan for the most common pair of neighbouring bytes and replace every one of them with a single byte that doesn't appear in the file. Repeat. That's it. Sennrich and his co-authors borrowed the idea for machine translation in 2015, and it's been sitting inside language models ever since.

Modern tokenizers went a step further and work at the byte level for real. The starting vocabulary isn't "the letters of the alphabet," it's all 256 possible byte values. Neat trick, because now there's no such thing as text the tokenizer can't handle. Emoji, Devanagari, Chinese, a corrupted file, your cat walking across the keyboard. It's all bytes, so it's all expressible. That missing-word problem from earlier doesn't get reduced, it just stops existing.

But there's a cost though!

The merges are learned from the training corpus, and that corpus is mostly English. So English compresses beautifully, a few characters per token. Hindi, Tamil, Bengali, Telugu compress badly, because the tokenizer never learned enough merges for those byte sequences. Run the same sentence in English and in Hindi and you can watch the Hindi version cost three or four times as many tokens.

Longer prompt, bigger bill, less room in the context window, worse answers. For the same sentence. And that gap got baked in while someone was training a tokenizer, long before anyone typed anything into it.

So when do we stop merging?

Good question, because if we kept going forever we'd end up back where we started, one token per word, with all our old problems waiting for us.

So we stop at a target vocabulary size. It's a number someone picks before training:

• GPT-2: 50,257 tokens
• GPT-4 (cl100k_base): around 100,000
• GPT-4o (o200k_base): around 200,000

And then the vocabulary is frozen. Those merge rules are fixed for the entire life of the model. Every prompt we send gets chopped up by rules that the tokenizer learned from the corpus of the text.

Sit with that for a second. What does that mean? Well, it means that the model's whole view of language runs through a lookup table that was locked in before it learned anything. Just to be clear the tokenizer and the model learning are separate things.

Okay, we have tokens. Now what?

Well each token is assigned a unique number based on how long our vocabulary is, for e.g, in our earlier example of the Lord of the rings. We have 11 words, so our vocabulary size is 11. We will assign the tokens number accordingly. For e.g:

1 gold -> 0
2 e -> 1
3 n -> 2
4 s -> 3
5 b -> 5
6 old -> 5
7 g -> 6
8 ol -> 7
9 l -> 8
10 u -> 9
11 m-> 10
12

Now, do the numbers go straight into the model? No. A token ID is just a name. For e.g, 19772 isn't bigger or more important than 302, it's just further down the list, and the model needs something better than that to work with.

So first each ID becomes a one hot vector. Which sounds fancy but only means: a vector as long as the vocabulary, filled with zeros, with a single 1 sitting at that token's position.

1token 302 → [0, 0, 0, ..., 1, ..., 0, 0]
2                          ↑
3                     position 302

Then that vector gets multiplied by the embedding matrix, which has a shape of vocab_size × d_model. Thatd_model is how many dimensions the model uses to hold the meaning of a word. Something like 768 in the smaller models, 12,288 for GPT-3 sized ones.

What comes out is a dense vector of floats, and this is where meaning starts to live. In that space "rage" and "anger" end up sitting near each other, while "rage" and "delighted" end up far apart. Nobody defined that. It falls out of where those words showed up in text.

Note: Nobody actually does that multiplication. Multiplying a one hot vector by a matrix just picks out one row of the matrix, so every real implementation skips the maths and grabs the row directly.

Back to the strawberry

So here's the model, trying to count the r's.

As you saw from above explanation, the model never got "strawberry." It got str, aw, berry, and then three dense vectors holding something like "the fruit, the colour, the flavour, the thing that turns up next to cream and jam." The letters are gone. They were thrown away at step one, before the model even entered the picture.

Now, asking it to count the r's is like asking someone to count the letters in a word they've only ever heard out loud, in a language they can't write. They might reason their way to the right answer. But they can't just look.

The model isn't being stupid here. It's paying for a compression decision.

Which leaves one thing open. I said the embedding matrix holds the meaning of each token, and that similar words land near each other in that space. But where did that matrix come from? Nobody sat down and typed 12,288 numbers describing "strawberry."

That's the next piece, and it's where this gets properly interesting. Subscribe to the newsletter so that you get the next article right in your inbox.

What Are Word Embeddings, Anyway?

Meaning is not typed in by anyone. It falls out of a prediction game played over billions of sentences, and the leftover geometry is what we call an embedding.