What Is an LLM? Parameters, Sampling, and Training Costs
A beginner-friendly overview of large language models, framing a model as a compressed archive of 16-bit float parameters. It walks through inference, tokenization, common sampling strategies (greedy, top-k, top-p, temperature), and the two-phase training process, citing rough costs: 10TB of data, 6,000 GPUs for 12 days, around $2M, yielding a ~140GB parameter file. The author intentionally keeps things shallow and points to Karpathy's intro and Anthropic interpretability work. Experienced LLM engineers will find little new here; the piece is aimed at newcomers building their first mental model.
In this article, we'll cover the basics of large language models. We'll talk about what they are, how they work, and touch on the process of creating them.
Most of the resources out there go really deep into how LLMs work - we're not going to do that. Instead, I'll give you a brief overview of the most common concepts so you can go ahead and get building.
在这篇文章中，我们会介绍大型语言模型的基础知识：它们是什么、如何工作，并简要提及创建它们的过程。市面上的大多数资料都会深入讲解 LLM 的工作原理——我们不会这么做。相反，我会给你一份最常见概念的简要概览，让你可以直接开始动手构建。
A large language model is essentially a massive compressed file - think of it like a 1TB zip file. This file contains a bunch of numbers, encoded as 16-bit floats. These numbers are the parameters of the model.
0.1239784871238176123 // Parameter 1
0.1515689756890123123 // Parameter 2
These parameters represent the 'brain' of the model. They are the result of the model's pre-training: a process that takes a huge amount of text data and 'compresses' it into these numbers. They represent the model's understanding of the world, and give it the ability to remember facts and make decisions.
大型语言模型本质上是一个巨大的压缩文件——可以把它想成一个 1TB 的 zip 文件。这个文件里装满了一组数字，以 16 位浮点数编码。这些数字就是模型的参数。
0.1239784871238176123 // Parameter 1
0.1515689756890123123 // Parameter 2
这些参数代表了模型的“大脑”。它们是模型预训练的结果：预训练将海量文本数据“压缩”到这些数字中。这些参数体现了模型对世界的理解，让它具备记住事实和做出决策的能力。
The number of these parameters represents the size of the model's brain. In general, models with larger brains perform better, but run slower. A model with 70B parameters will run ~10x slower than a model with 7B parameters.
Already, we're looking at a size vs speed tradeoff - a common theme when choosing large language models.
参数的多少代表模型“大脑”的大小。一般来说，大脑越大的模型表现越好，但运行越慢。一个 70B 参数的模型运行速度大约比 7B 参数的模型慢 10 倍。这其实就是规模与速度之间的取舍——这是选择大型语言模型时经常遇到的主题。
In order to get the model to do anything useful, you need to perform inference on the model.
Inference is the process of sending text to the model and getting a response back. This is done using an inference function - a piece of software that takes the parameters of the model and runs an algorithm on them to find the next word. This is far cheaper than pre-training the model, and can be done on your laptop.
For a deep-dive into how inference works, check out this incredible interactive walkthrough.
要让模型做任何有用的事情，你需要对模型进行推理。推理就是把文本发送给模型并得到回应的过程。这通过一个推理函数完成——一个将模型参数作为输入、在其上运行算法以找出下一个词的软件。推理远比预训练模型便宜，可以在你的笔记本电脑上完成。深入了解推理的工作原理，请看这个精彩的交互式演练。
Sampling Strategy
To find the next word, the model looks at all the possible tokens it could choose, and picks one.
To do so, it uses a sampling strategy picked by the developer. This strategy determines how the model chooses the next word. The most common strategies are:
Greedy Sampling: The model always picks the most likely word.
Top-K Sampling: The model picks from the top K most likely words.
Top-P Sampling: The model picks from the words that make up P% of the probability mass.
Temperature Sampling: The model introduces randomness into the selection process, allowing for more diverse outputs.
It's beyond the scope of this article to go into the details of these strategies. Usually, as an AI Engineer, you don't have the ability to change the sampling strategy of the model you're using. However, you can tweak variables, like the temperature, to get different results.
采样策略
为了找出下一个词，模型会查看所有可能选择的 token，然后挑选一个。为此，它使用开发者选定的采样策略。这个策略决定了模型如何选择下一个词。最常见的策略有：
贪心采样：模型总是选择概率最高的词。
Top-K 采样：模型从概率最高的 K 个词中挑选。
Top-P 采样：模型从占总概率质量 P% 的词中挑选。
温度采样：模型在选择过程中引入随机性，让输出更多样化。
深入这些策略的细节不在本文范围内。通常，作为 AI 工程师，你无法更改所用模型的采样策略。不过，你可以调整温度等变量来获得不同的结果。
When you send text to the model, it first needs to be tokenized. This is the process of breaking the text up into individual words, and then converting those words into numbers. These numbers are the input tokens, which are passed to the inference engine.
Each model has its own tokenizer. Tiktokenizer is a great playground for exploring different tokenizers.
当你向模型发送文本时，它首先需要被分词。分词将文本拆分成一个个单词，再把这些单词转换成数字。这些数字就是输入 token，会被传给推理引擎。每个模型都有自己的分词器。Tiktokenizer 是一个很棒的游乐场，可以用来探索各种分词器。
In order to acquire the parameters, you need to train the model. Training large language models is an extremely involved process that requires a lot of time, expertise, and money. Learning how to do it is outside the scope of this article.
A rough guide is to take a chunk of the internet, let's say 10TB of data. You use 6,000 GPU's for 12 days, at the cost of around $2M. And you end up with a ~140GB file with all the parameters of the model.
要获得参数，你需要训练模型。训练大型语言模型是一个非常复杂的过程，需要花费大量时间、专业知识和金钱。学会如何训练不在本文范围内。
粗略来看：取互联网上一大块数据，假设 10TB。你用 6,000 块 GPU 跑 12 天，成本大约 200 万美元。最后你会得到一个约 140GB 的文件，里面装着模型的所有参数。
The training process has two main phases:
Pre-training: This gives the model its knowledge by compressing vast amounts of internet data into parameters
Post-training: This shapes the model's personality and behavior through careful instruction and example.
You end up with a huge file of parameters - a kind of 'compressed' version of all of the data the model was trained on, with its personality shaped by post-training. Without post-training, the model would just be an inert blob of knowledge - it wouldn't know how to behave like a helpful assistant.
训练过程有两个主要阶段：
预训练：将海量互联网数据压缩成参数，赋予模型知识。
后训练：通过精心设计的指令和示例，塑造模型的个性和行为。
你最终会得到一个庞大的参数文件——它像是模型训练所用数据的“压缩版”，个性则被后训练塑造。如果没有后训练，模型只是一团惰性的知识块——它不知道该如何表现得像一个乐于助人的助手。
Resources
Intro To Large Language Models by Andrej Karpathy
How I Use LLM's by Andrej Karpathy
资源
Intro To Large Language Models，作者 Andrej Karpathy
How I Use LLM's，作者 Andrej Karpathy
It's possible (though very difficult) to dive into the parameters of a model to work out which ones correspond to which real-world concepts. For instance, Anthropic found that models can:
Share concepts across languages, suggesting a kind of universal "language of thought"
Plan ahead when writing (like planning rhymes in poetry)
Use multiple parallel pathways for tasks (like mental math)
Sometimes fabricate plausible-sounding reasoning rather than showing their true thought process
By understanding how models think, we can better anticipate their behavior and potentially remove some of the "magic" that makes them hard to trust. This could lead to more controlled, deterministic AI systems in the future - but the research is still in its early stages.
深入模型的参数，去弄清哪些参数对应哪些现实概念，是有可能的（尽管非常困难）。例如，Anthropic 发现模型可以：
跨语言共享概念，暗示可能存在一种普适的“思想语言”
写作时提前规划（比如在诗歌中安排押韵）
使用多条并行通路处理任务（比如心算）
有时会编造听起来合理解释，而不是展示其真实思维过程
通过理解模型的思考方式，我们可以更好地预判它们的行为，并有可能消除一些让人难以信任的“魔力”。这可能会让未来的 AI 系统更加可控、更确定——但这项研究仍处于早期阶段。
Resources
Tracing the thoughts of a large language model by Anthropic
Golden Gate Claude by Anthropic
资源
Tracing the thoughts of a large language model，作者 Anthropic
Golden Gate Claude，作者 Anthropic
Large language models compress vast amounts of knowledge into numerical parameters. While they're built on complex math, you don't need to understand their inner workings to use them effectively. Getting a basic lay of the land is enough to build smart intuitions about how they work.
In the next article, we'll explore how to choose the right LLM for your specific needs.
大型语言模型将海量知识压缩成数值参数。虽然它们建立在复杂的数学之上，但你不必理解其内部运作就能有效地使用它们。对基本概念有个大致了解，就足以让你建立起关于它们如何工作的良好直觉。在下一篇文章中，我们将探讨如何为你的具体需求选择合适的 LLM。