06:00
What Are Tokens? A Practical Guide to LLM Tokenization
一文看懂 LLM Token：从拆分、编码到计费
A beginner-friendly explainer on LLM tokens: how tokenizers split text into tokens, map them to numbers, and decode outputs back to text. Using a tiny corpus ('the cat sat on the mat'), the author walks through vocabulary construction from characters to character groups, and cites concrete examples: vocabulary size from 1k to 200k reduces 'understanding' from 5 tokens to 2, while the made-up word 'Frabjous' burns 7 tokens for 15 characters. It also clarifies billing: input tokens include conversation history, system prompt, and tool definitions; output tokens are billed separately, so shorter generations save money. Useful for app developers who want to understand and optimize LLM token costs.