What Are Tokens? A Practical Guide to LLM Tokenization
A beginner-friendly explainer on LLM tokens: how tokenizers split text into tokens, map them to numbers, and decode outputs back to text. Using a tiny corpus ('the cat sat on the mat'), the author walks through vocabulary construction from characters to character groups, and cites concrete examples: vocabulary size from 1k to 200k reduces 'understanding' from 5 tokens to 2, while the made-up word 'Frabjous' burns 7 tokens for 15 characters. It also clarifies billing: input tokens include conversation history, system prompt, and tool definitions; output tokens are billed separately, so shorter generations save money. Useful for app developers who want to understand and optimize LLM token costs.
Tokens are the fundamental building blocks that help Large Language Models (LLMs) process text. Understanding them is essential, especially since you're billed based on token usage.
Tokens are simply numbers that represent how the LLM "thinks" about the text you provide. The process of converting text into tokens is called encoding.
The tokenization process works in two parts:
- The tokenizer splits text into tokens it recognizes
- These tokens are converted into numbers
Decoding is the reverse process:
- Numbers are converted back into text tokens
- The tokens are joined together to form the output
The complete LLM process looks like this:
- Tokenizer encodes your input text into tokens
- LLM processes your tokens
- LLM produces output tokens
- Output tokens are decoded back into readable text
Token 是帮助大型语言模型（LLM）处理文本的基础构建单元。理解 token 至关重要，尤其是因为你按 token 用量付费。
Token 本质上就是数字，代表 LLM 对你提供的文本的“思考”方式。将文本转换成 token 的过程叫作编码（encoding）。
分词过程分两步：
- 分词器先将文本拆成它能识别的 token
- 再把这些 token 转换成数字
解码则是相反的过程：
- 把数字重新转换回文本 token
- 将 token 拼接起来，形成输出
完整的 LLM 流程如下：
- 分词器将你的输入文本编码成 token
- LLM 处理这些 token
- LLM 生成输出 token
- 将输出 token 解码回可读文本
To clarify, input tokens include:
- Your conversation history with the LLM
- System prompts
- Tool definitions
Output tokens are what the LLM sends back as a response.
You're billed for both input and output tokens, typically at different rates. One way to save money is to design your prompts to generate fewer output tokens.
需要澄清的是，输入 token 包括：
- 你和 LLM 的对话历史
- 系统提示词（system prompts）
- 工具定义
输出 token 则是 LLM 回传给你的响应内容。
输入和输出 token 都要计费，通常费率不同。想省钱的一个办法，就是设计 prompt 时让它生成更少的输出 token。
The tokenization process starts with a large corpus of text - similar to what's used to train the LLM itself. Let's imagine a tiny corpus consisting of just one sentence: "the cat sat on the mat."
First, all individual characters are extracted:
T H E space C A T space S A T space O N space T H E space M A T
Each of these characters becomes its own token in the vocabulary.
Next, common groupings of characters are identified:
- "TH" appears in "the" (twice)
- "HE" appears in "the" (twice)
- "AT" appears in "cat", "sat", and "mat"
Each of these groupings also gets assigned its own token.
Then, groups of groups are identified - like "TH" + "HE" creating "THE" (the word "the"), which gets its own token.
分词过程起始于一个大文本语料库——与训练 LLM 本身所用的语料类似。我们想象一个极小的语料库，只有一句话："the cat sat on the mat."
首先，提取所有单个字符：
T H E 空格 C A T 空格 S A T 空格 O N 空格 T H E 空格 M A T
这些字符各自成为词表中的一个 token。
接下来，识别常见的字符组合：
- "TH" 出现在 "the" 中（两次）
- "HE" 出现在 "the" 中（两次）
- "AT" 出现在 "cat"、"sat" 和 "mat" 中
每个组合也都会被分配一个自己的 token。
然后，再识别“组合的组合”——比如 "TH" + "HE" 组成 "THE"（即单词 "the"），它也有自己的 token。
The goal is to create a large vocabulary of tokens because larger vocabularies can split words into fewer tokens, making processing more efficient.
For example, a vocabulary size of 1,000 tokens might split "understanding" into 5 tokens. A vocabulary size of 50,000 tokens might split it into 3 tokens, and a vocabulary size of 200,000 tokens might split it into 2 tokens.
Having a larger vocabulary means you can split words into fewer tokens, making processing more efficient.
目标是建立一个包含大量 token 的词表，因为词表越大，把词拆出来的 token 就越少，处理效率也越高。
例如，词表大小为 1,000 时，"understanding" 可能被切成 5 个 token；词表大小为 50,000 时可能切成 3 个；词表大小为 200,000 时可能切成 2 个。
词表越大，意味着你可以把单词切成更少的 token，处理效率更高。
The tokenizer struggles with uncommon words. For example, "O Frabjous Day" from Lewis Carroll's poem gets split into many tokens because "Frabjous" is a made-up word that doesn't appear frequently in the training corpus.
We can see that it turns it into 7 tokens - more than we'd expect from only 15 characters.
分词器面对生僻词时很吃力。比如刘易斯·卡罗尔诗中的 "O Frabjous Day" 就被切成了很多 token，因为 "Frabjous" 是个生造词，在训练语料中不常出现。
可以看到它被切成了 7 个 token——对只有 15 个字符的内容来说，比预期要多。
I hope that helps demystify tokens a bit. I found the tiktokenizer playground really useful for understanding this stuff.
Let me know if you have any questions - and what else would you like me to cover next?
Matt
希望这能帮你更好地理解 token。我发现，tiktokenizer playground 对搞懂这些东西真的很有帮助。
如果有什么问题，欢迎告诉我——你还想让我接着讲什么？
Matt