No. 65October 2026

Distributed systems, language models and the machinery underneath

Tokenization

1 story tagged “Tokenization”.

1 of 1 stories

2026

  1. 63

    What Are Tokens in an LLM, Anyway?

    The algorithm behind GPT's tokenizer is called BPE, byte pair encoding.

    Language models12 min ·