Ask ChatGPT how many r's there are in the word "strawberry."
For the longest time it got this wrong, and people passed those screenshots around as proof that these models are dumb. I don't think that's what it proves. I think it proves something much more specific, and much more interesting.
The model never saw the word "strawberry." It saw three numbers.
So let me give you a high level overview of what we're covering here. We'll look at what tokens actually are, how they get made, why anyone thought this was a good idea in the first place, and what happens to a token after it's made. By the end you'll know why a machine that can write working Rust code can lose a fight with a piece of fruit.
Sounds interesting? Let's get into it.
So what actually happens when you hit enter?
The first thing that happens has nothing to do with intelligence. Your text goes through a tokenizer, which chops it into pieces and swaps each piece for a number.
"strawberry" comes out as three tokens. Not four, not one. Three.
And here's the bit that got me the first time I checked. The split isn't where you'd put it. You and I would break that word as straw + berry, because that's how the word is built. The tokenizer doesn't care how the word is built. It breaks it as:
1str | aw | berryHang on to that one. It explains the letter counting thing at the end.
If you'd rather see it yourself than take my word for it, it's about four lines:
1import tiktoken
2
3enc = tiktoken.get_encoding("cl100k_base") # GPT-4's tokenizer
4ids = enc.encode("strawberry")
5print(ids)
6print([enc.decode([i]) for i in ids])
tiktokenizer
Try it with your own name. Try it with a Hindi sentence. Try it with an emoji. The results start telling you things, and we'll come back to why.
Why chop up the words at all?
Fair thing to ask. Why not skip this whole step?
Well, look at the two obvious ways to do it and watch both of them fall apart.
Give every word its own token.
Clean, intuitive, matches how we think about language. Now count the words. English alone runs into the millions once you include names, typos, plurals, slang, and every product name anyone has ever invented. Your lookup table gets enormous, and most of those millions of entries show up three times in training, which is nowhere near enough to learn anything from.
That's the boring problem. Here's the real one. What happens when the model meets a word that wasn't on the list? Not a rare word, just a new one. A word someone coined last Tuesday. A typo. Your surname. There's no number for it, so the model gets nothing at all. A hole in the input, and no polite way to fail.
Give every character its own token.
Now the vocabulary is tiny, maybe a couple of hundred symbols, and nothing is ever missing. Sounds like a fix.
It isn't, and the reason is worth slowing down on.
Inside the model, every token has to look at every other token in your input. That's what attention does. Six tokens means roughly 36 of those little comparisons. Twelve tokens means 144. The work doesn't double when the input doubles, it goes up four times.
Now take a sentence like "The cat sat on the mat." As words, that's 6 tokens. As characters, it's 22. So the sentence got about four times longer, which means the model does something like sixteen times the work. Same sentence. Nothing gained.
And there's a second cost. The model now has to figure out that c,a, t put together means something, and it has to do that before it can start learning anything about actual cats. You've handed it a spelling homework assignment on top of everything else.
So one extreme gives you a vocabulary you can't afford and gaps you can't fill. The other gives you sentences you can't afford and a model doing spelling drills.
Subword tokenization sits in the middle. Common words stay whole, rare words get built out of reusable pieces, and nothing is ever unrepresentable.
If you've spent any time with German you already have the intuition for this. A glove is a Handschuh, a hand shoe. A speed limit is a Geschwindigkeitsbegrenzung, which is just Geschwindigkeit and Begrenzung welded together. German doesn't invent a fresh word for every concept, it keeps a box of pieces and assembles what it needs. Subword tokenization does the same thing, except the pieces get picked by counting rather than by meaning.
So how does it decide where to cut?
The algorithm behind GPT's tokenizer is called BPE, byte pair encoding. It's close to the simplest thing that could possibly work, which is usually a good sign.
It starts with a training corpus. In practice that corpus is a large chunk of the internet, but let's keep it small enough to do by hand. Say the entire language, as far as our tokenizer is concerned, is one book. The Lord of the Rings.
Say we pull out these words along with how often they appear:
1("gold", 10), ("golden", 5), ("hold", 12), ("holds", 4), ("bold", 5), ("gollum", 8)Step one, smash everything down to characters. That's our starting vocabulary:
1Vocabulary: b, d, e, g, h, l, m, n, o, s, u
2
3("g" "o" "l" "d", 10)
4("g" "o" "l" "d" "e" "n", 5)
5("h" "o" "l" "d", 12)
6("h" "o" "l" "d" "s", 4)
7("b" "o" "l" "d", 5)
8("g" "o" "l" "l" "u" "m", 8)Now we look at every neighbouring pair and count how often each one turns up.
ol is in every single word: 10 + 5 + 12 + 4 + 5 + 8 = 44. Nothing else is close (ld gets 36, ho gets 16). So we merge it and add it to the vocabulary.
1Vocabulary: ..., ol
2
3("g" "ol" "d", 10) ("g" "ol" "d" "e" "n", 5) ("h" "ol" "d", 12)
4("h" "ol" "d" "s", 4) ("b" "ol" "d", 5) ("g" "ol" "l" "u" "m", 8)Count again. Now ol next to d turns up 36 times, more than anything else. So we merge that too.
1Vocabulary: ..., ol, old
2
3("g" "old", 10) ("g" "old" "e" "n", 5) ("h" "old", 12)
4("h" "old" "s", 4) ("b" "old", 5) ("g" "ol" "l" "u" "m", 8)Stop and look at what just happened there. The tokenizer found "old." Nobody told it that "old" is a word in English. It doesn't know what old means. It merged those two characters because they kept showing up next to each other and merging them was the cheapest win available.
Keep going. h plus old is 16, g plus old is 15. So hold becomes a token, then gold becomes a token.
1Vocabulary: ..., ol, old, hold, gold
2
3("gold", 10) ("gold" "e" "n", 5) ("hold", 12)
4("hold" "s", 4) ("b" "old", 5) ("g" "ol" "l" "u" "m", 8)Two things worth noticing in that final state.
"golden" never became a single token. It's gold + e + n, because it wasn't frequent enough to earn a slot of its own. It gets assembled from parts, exactly like Handschuh.
And Gollum is still lying there in pieces: g + ol + l + u + m. He only shows up 8 times, so he never won a merge. Which tells you a lot about the whole scheme. Frequent text is cheap, rare text is expensive, and the tokenizer picked its favourites long before you turned up with a prompt.
This is a very high level walkthrough, by the way. If you want the full thing, the Hugging Face tokenizer course covers it properly and I'll link it below.
But why is it called byte pair encoding though?
You'd guess it's because a character takes up a byte. That's close enough to sound right and wrong enough to be worth correcting.
The name comes from the original algorithm, which had nothing to do with language. Philip Gage published it in 1994 as a way to compress files. You scan for the most common pair of neighbouring bytes and replace every one of them with a single byte that doesn't appear in the file. Repeat. That's it. Sennrich and his co-authors borrowed the idea for machine translation in 2015, and it's been sitting inside language models ever since.
Modern tokenizers went a step further and work at the byte level for real. The starting vocabulary isn't "the letters of the alphabet," it's all 256 possible byte values. Neat trick, because now there's no such thing as text the tokenizer can't handle. Emoji, Devanagari, Chinese, a corrupted file, your cat walking across the keyboard. It's all bytes, so it's all expressible. That missing-word problem from earlier doesn't get reduced, it stops existing.
There's a cost though, and it doesn't land evenly.
The merges are learned from the training corpus, and that corpus is mostly English. So English compresses beautifully, a few characters per token. Hindi, Tamil, Bengali, Telugu compress badly, because the tokenizer never learned enough merges for those byte sequences. Run the same sentence in English and in Hindi and you can watch the Hindi version cost three or four times as many tokens.
Longer prompt, bigger bill, less room in the context window, worse answers. For the same sentence. And that gap got baked in while someone was training a tokenizer, long before anyone typed anything into it.
So when do we stop merging?
Good question, because if we kept going forever we'd end up back where we started, one token per word, with all our old problems waiting for us.
So we stop at a target vocabulary size. It's a number someone picks before training:
- GPT-2: 50,257 tokens
- GPT-4 (cl100k_base): around 100,000
- GPT-4o (o200k_base): around 200,000
And then the vocabulary is frozen. Those merge rules are fixed for the entire life of the model. Every prompt anyone ever sends gets chopped up by rules that were finalised before training even began, based on whatever happened to be common in a pile of text scraped at one particular moment.
Sit with that for a second. The model's whole view of language runs through a lookup table that was locked in before it learned anything.
Okay, we have tokens. Now what?
Do the numbers go straight into the model? No. A token ID is just a name. 19772 isn't bigger or more important than 302, it's just further down the list, and the model needs something better than that to work with.
So first each ID becomes a one hot vector. Which sounds fancy but only means: a vector as long as the vocabulary, filled with zeros, with a single 1 sitting at that token's position.
1token 302 → [0, 0, 0, ..., 1, ..., 0, 0]
2 ↑
3 position 302Then that vector gets multiplied by the embedding matrix, which has a shape of vocab_size × d_model. Thatd_model is how many dimensions the model uses to hold the meaning of a word. Something like 768 in the smaller models, 12,288 for GPT-3 sized ones.
What comes out is a dense vector of floats, and this is where meaning starts to live. In that space "rage" and "anger" end up sitting near each other, while "rage" and "delighted" end up far apart. Nobody defined that. It falls out of where those words showed up in text.
One footnote worth having, because it's the sort of thing that separates people who've read about this from people who've built it. Nobody actually does that multiplication. Multiplying a one hot vector by a matrix just picks out one row of the matrix, so every real implementation skips the maths and grabs the row directly. The one hot story is a good way to explain what's happening, not a description of what runs. Open the code and you'll find an index lookup.
Back to the strawberry
So here's the model, trying to count the r's.
It never got "strawberry." It got str, aw, berry, and then three dense vectors holding something like "the fruit, the colour, the flavour, the thing that turns up next to cream and jam." The letters are gone. They were thrown away at step one, before the model even entered the picture.
Asking it to count the r's is like asking someone to count the letters in a word they've only ever heard out loud, in a language they can't write. They might reason their way to the right answer. They can't just look.
The model isn't being stupid here. It's paying for a compression decision that was made long before it was born.
Which leaves one thing open. I said the embedding matrix holds the meaning of each token, and that similar words land near each other in that space. But where did that matrix come from? Nobody sat down and typed 12,288 numbers describing "strawberry."
That's the next piece, and it's where this gets properly interesting. Subscribe to the newsletter so that you get the next article right in your inbox.