How trillion-parameter AI models work and how to serve them

章节 1：Kimi K3架构创新与专家稀疏性

📝 本节摘要：

Kimi K3以超大规模参数为背景，针对序列长度、模型宽度和深度分别采用残差连接、Stable LatentMoE与混合注意力。讨论还解释了专家路由为何会逐渐失衡：早期略占优势的专家可能持续吸引更多 token，造成负载集中和 GPU 慢负载；训练时需平衡专家负载并规范化隐藏激活。

[原文] [Philip]: But you've essentially unrolled your recurrent relationship into a matrix multiplication, and you're able to get your answer in one shot using the tensor cores. Hi, I'm Philip from Baseten. With me today are Ali and Bryce from our model performance team. Kimi K3: it's the biggest open model in the world, with 2.8 trillion parameters. But what else is special about it? Well, a lot of things. Once you get to a size of 2.8 trillion, you can't just do the same things and expect it to work.

[译文] [Philip]: 不过，你基本上是把循环关系展开成了矩阵乘法，然后用张量核心一次就能得到答案。大家好，我是 Baseten 的 Philip。今天和我一起的是模型性能团队的 Ali 和 Bryce。Kimi K3 是世界上最大的开放模型，拥有 2.8 万亿个参数。它还有什么特别之处？特别之处很多。模型规模达到 2.8 万亿参数后，就不能指望照搬原来的做法还能奏效。

[原文] [Ali]: So they address three main problems: the depth of the model, the width of the model, and the sequence length. The sequence length, increasing up to a million, is addressed with residual connections, which they introduced in Kimi before. The width of the model—each MoE being extremely huge, with 896 experts and a sparsity level of around 56—is addressed with Stable LatentMoE. And the actual depth of the model is addressed with hybrid attention. So you've got KDA linear attention in three-quarters of the layers, and MLA attention in one-quarter.

[译文] [Ali]: 他们主要解决三个问题：模型深度、模型宽度和序列长度。序列长度扩展到一百万，则通过残差连接来处理；他们之前已经在 Kimi 中引入过这种连接。模型宽度方面，每个 MoE 都极其庞大，有 896 个专家，稀疏度约为 56，因此他们用 Stable LatentMoE 来应对。模型的实际深度则由混合注意力处理：四分之三的层采用 KDA 线性注意力，四分之一采用 MLA 注意力。

[原文] [Philip]: So it's a three-to-one ratio. That's where you mix some global attention and local linear attention. It's a great way to break it down. We've got three problems—the width of the model, the depth of the model, and the sequence length—and a new architectural innovation for each one. One thing I've been curious about is the expert sparsity problem. We have 896 experts, 16 active experts, plus two shared ones. First off, what makes expert sparsity unstable? And secondly, how does Stable LatentMoE address it?

[译文] [Philip]: 所以比例是三比一，也就是把全局注意力和局部线性注意力结合起来。这样拆解很清楚：模型宽度、模型深度和序列长度这三个问题，各自都有一种新的架构创新来应对。我一直很好奇专家稀疏性问题：我们有 896 个专家，其中 16 个是活跃专家，另外还有两个共享专家。首先，是什么导致专家稀疏性不稳定？其次，Stable LatentMoE 如何解决这个问题？

[原文] [Ali]: - Well, I guess if you just think about it intuitively, let's say you have four experts, right? So you've got expert 0, one, two, and three. Each of them are going to be instantiated at random. So each of them just has a random matrix with them. You get a request, a token X1, you get a router, and this router is going to do some heuristics and route it to the expert, which is most closely matched to it. By random choice, one of them is going to be the one that gets selected.

[译文] [Ali]: 直观地想，假设你有四个专家：专家 0、1、2 和 3。它们都是随机初始化的，所以各自都有一个随机矩阵。现在来了一个请求，也就是 token X1，路由器会根据一些启发式规则，把它分配给最匹配的专家。随机情况下，其中一个专家会被选中。

[原文] [Ali]: The token's going to get routed to that expert, you're going to get the answer, it's going to get trained, it's going to get backpropped through. This expert's going to become slightly better than all of the rest. Future tokens two, three, four, because this is a better expert, are going to also get routed to him. And eventually, if you don't do anything to fix this balancing, you're going to end up with one expert getting almost all of the load and none of the other experts getting any of the load.

[译文] [Ali]: 这个 token 被路由到该专家，模型得到答案后再进行训练和反向传播。这个专家会因此比其他专家稍微强一点。之后的 token 2、3、4 也会被路由给它，因为它表现更好。最终，如果不采取措施纠正这种失衡，一个专家就会承担几乎全部负载，其他专家则分不到任何负载。

[原文] [Ali]: And as a result of that, if you shard by the experts on GPUs, one of your GPUs is going to straggle behind because it's constantly doing all of the work. And some methods have tried to fix this, balancing, DeepSeek-V2, but at this level at 896 experts, it just got so large they had to introduce Stable LatentMoE. They had to fix it without using the step update that was in DeepSeek. And the other problem is the experts are so large in width that the activations or the inner latents are going to get so large, the activations, they need to be normalized, otherwise they could exceed or overflow in the precision or they could cause exploding gradients.

[译文] [Ali]: 因此，如果把专家分片部署到多张 GPU 上，一张 GPU 就会因为一直承担所有工作而落后。DeepSeek-V2 等方法尝试解决这种负载失衡，但在 896 个专家的规模下，问题变得太大，所以他们引入了 Stable LatentMoE。他们必须在不使用 DeepSeek 那种步进更新的情况下解决这个问题。另一个问题是专家的宽度太大，激活值，也就是内部潜变量，会变得非常庞大。激活值需要归一化，否则可能超出或溢出精度范围，也可能导致梯度爆炸。

[原文] [Ali]: So it's basically normalization of the hidden activations and load balancing between the experts. Gotcha. So right at the beginning of training, one expert gets slightly better than all of its peers. Unless we correct for that, it's just going to continue to get every single opportunity and throw the entire model out of balance. Once the model is trained, we do have these 896 experts, each of which is highly capable. How do we think about serving a model with that level of width? Yeah, you certainly can have similar problems to what you see during training.

[译文] [Ali]: 所以，本质上就是对隐藏激活值进行归一化，并在专家之间平衡负载。明白了。训练一开始，一个专家就比其他专家稍微强一点；如果不纠正，它就会不断得到所有机会，导致整个模型失衡。模型训练完成后，我们确实拥有了 896 个能力都很强的专家。那么，该如何部署宽度达到这种程度的模型呢？是的，部署时确实也可能遇到训练阶段类似的问题。