Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten

章节 1：长上下文请求的推理流程

📝 本节摘要：

本章先介绍用模型自动分析性能追踪并编写 GPU 内核的实践，随后主持人感谢听众支持并介绍嘉宾。围绕 20 万 token 的长请求，讨论缓存感知路由、预填充与解码资源分工，以及通过历史缓存降低处理成本。嘉宾也以轻松对话聊到 Waterloo 实习生称号的交接。

[原文] [Philip Kiely]: JLM52 is very very good at writing GPU kernels. It was very funny internally we had a JLM52 endpoint that we're using to like that we plugged in in our cloud code harness. So, every engineer on the team used like our JLM52. And it will do a forward pass on the JLM52 instance of the you know the node and then it will get the profile trace and it will analyze it and it will find the kernels that are the bottlenecks in that shelling and then it will write the new kernels and then it will do another profiling trace and that's done it uploads to our thing and then we can pull that image down and repeat the cycle. Some of the GPU kernels that we run on JLM52 within our inference engine is written by JLM52.

[译文] [Philip Kiely]: JLM 5.2 非常擅长编写 GPU 内核。内部有件事特别有意思：我们有一个 JLM 5.2 端点，接进了云端代码测试框架，团队里的每位工程师都在用。它会在节点的 JLM 5.2 实例上执行一次前向传播，获取性能追踪并进行分析，找出其中的瓶颈内核，然后编写新内核，再做一次性能追踪。完成后，它会把结果上传到我们的系统里，我们就能拉取镜像，重复这个循环。我们的推理引擎中有些 GPU 内核就是 JLM 5.2 写的。

[原文] [swyx]: Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you.

[译文] [swyx]: 进入今天的节目之前，我想先跟听众说几句话。谢谢大家。如果不是你们愿意点进来收听我们的节目，我们就没法继续为大家带来你们显然很喜欢的 AI 工程、科学和娱乐内容。我们几乎每天都会收到赞助商的邀约，但幸好有足够多的听众订阅，让我们不用广告也能维持节目，我们也希望继续这样做。不过，我只想请大家帮一个小忙。

[原文] [swyx]: The single most powerful, completely free thing you can do is click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team, which works so hard to bring In Space to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now let's get into it. Okay, we're here in the studio with Philip, an old friend from Inference Engineering, the book, as well as Baseten, and all the work you've done. You and I have worked together before, as has Ali. Welcome.

[译文] [swyx]: 你能做的最有力、完全免费的支持，就是点一下订阅按钮。这是我唯一会向大家提出的请求。对我和团队来说，这意义重大，我们每周都在努力为大家带来《In Space》。如果你愿意订阅，我保证我们会一直努力，把节目做得更好。现在进入正题。我们今天在演播室请到了老朋友 Philip，他参与了《Inference Engineering》这本书，也来自 Baseten，做了很多出色的工作。我和你以前合作过，Ali 也是。欢迎你们。

[原文] [Philip Kiely]: Pleasure to meet you.

[译文] [Philip Kiely]: 很高兴认识你。

[原文] [swyx]: Waterloo intern.

[译文] [swyx]: 滑铁卢实习生。

[原文] [swyx]: Waterloo intern Ali.

[译文] [swyx]: 滑铁卢实习生 Ali。

[原文] [swyx]: When did you get “Waterloo intern” as a—

[译文] [swyx]: 你什么时候把“滑铁卢实习生”当成——

[原文] [swyx]: As a handle?

[译文] [swyx]: 当作账号名的？

[原文] [Ali Taha]: Handle?

[译文] [Ali Taha]: 账号名？

[原文] [Ali Taha]: I think the rebranding happened around mid-March. When I saw it was open, I was like, “I have to take it before it's up for grabs.”

[译文] [Ali Taha]: 我记得改名是在三月中旬左右。我看到这个名字空出来了，就想：“得赶紧拿下，免得被别人抢走。”

[原文] [swyx]: The problem is that Ali is really good at his job and isn't going to be an intern much longer. So we have to figure out who's going to get the handle.

[译文] [swyx]: 问题是 Ali 工作做得太出色，很快就不再是实习生了。所以我们得想想这个账号名以后交给谁。

[原文] [Philip Kiely]: Pass the torch over to the—

[译文] [Philip Kiely]: 把接力棒传给——

[原文] [swyx]: Oh, okay. It can be like you just pass it to another Waterloo grad.

[译文] [swyx]: 哦，好啊。可以直接传给另一位滑铁卢毕业生。

[原文] [Ali Taha]: Yeah.

[译文] [Ali Taha]: 对。

[原文] [swyx]: Intern.

[译文] [swyx]: 实习生。

[原文] [Philip Kiely]: Intern.

[译文] [Philip Kiely]: 实习生。

[原文] [swyx]: You got to get an intern from Waterloo.

[译文] [swyx]: 你们得找个滑铁卢来的实习生。

[原文] [Ali Taha]: But they have to Oh, it could But it could come from Base 10. So, it's like wherever Base 10 gets from Waterloo

[译文] [Ali Taha]: 不过他们得——哦，也可以从 Baseten 这边来。也就是说，Baseten 招来哪个滑铁卢的人——

[原文] [swyx]: Right. Has the title of Waterloo—

[译文] [swyx]: 没错，头衔就是“滑铁卢——”

[原文] [swyx]: You either get it or you're out. You should also do a big graduation ceremony where you change the handle.

[译文] [swyx]: 要么拿到这个称号，要么就出局。你们还应该办个盛大的毕业典礼，当场把账号名换掉。

[原文] [swyx]: I mean, you guys are good at ceremonies, clearly. You know, we had a nice launch of the book, very successful. But before we get into all that, I want to start off with a fun question for you. Okay, you're an expert inference engineer. What happens when I send a long query, say 200,000 tokens, into Baseten's inference system? What's the process from query through GPU, model routing, balancing, all that? What are all the things we don't think about?

[译文] [swyx]: 你们显然很擅长办仪式。我们那本书的发布活动就办得不错，非常成功。不过在聊这些之前，我想先问个有趣的问题。你是推理工程专家。假如我把一个很长的请求，比如 20 万 token，发到 Baseten 的推理系统里，会发生什么？从请求进入系统，到 GPU 调度、模型路由和负载均衡，整个流程是怎样的？有哪些事情是我们平时不会想到的？

[原文] [Philip Kiely]: With a long query specifically, the first thing I'm going to ask is: have you sent me this query before, or at least part of it? And I really hope you have, because it's going to be a lot easier for me and a lot cheaper for you. The first thing we're going to look at is some kind of cache-aware routing. We probably have a number of instances, a number of replicas serving whatever model you're hitting. We want to send this request to one with, number one, available prefill workers, and number two, ideally, some cached input already there so that we can skip prefill on at least part of these 200,000 tokens. If you're sending 200,000 tokens, it's probably coding or a multi-turn agent or something where you'd expect to have that cached.

[译文] [Philip Kiely]: 对于长请求，我首先会问：你之前发过这个请求吗？或者至少发过其中一部分吗？我真希望你发过，这样我们处理起来会轻松很多，你的成本也会低得多。首先，我们会进行某种缓存感知路由。系统里大概有多个实例、多个副本在运行你要调用的模型。我们会把请求发给一个既有可用预填充工作进程、最好又已经缓存了部分输入的实例，这样至少能跳过这 20 万 token 中一部分内容的预填充。如果你发来 20 万 token，很可能是在编程，或是在使用多轮智能体之类的场景；这类场景通常会有可复用的缓存。