Dylan Patel: NVIDIA's New Moat & Why China is "Semiconductor Pilled”

章节 1：Nvidia 收购 Groq 与推理芯片的战略博弈

📝 本节摘要：

本章从 Nvidia 收购 Groq 这一热点事件切入。Dylan 指出，Nvidia 过去通过“通用 GPU”赢得了第一波 AI 浪潮，但在推理工作负载（Inference）日益庞大的今天，市场对“专用芯片”的需求正在激增（如专注于解码速度的 Groq 或专注于 KV Cache 的芯片）。

>

核心观点在于：Jensen Huang 极度信奉“只有偏执狂才能生存”。面对可能在成本和性能上击败 Nvidia 的单点解决方案（Point Solutions），Nvidia 选择通过收购（如 Groq）来获取稀缺的芯片工程人才，从而快速构建针对不同细分市场的专用产品线，以维持其高昂的利润率和霸主地位。

[原文] [Intro Clip]: this is the biggest change in human history maybe ever what's about to happen with AI this is the biggest revolution bigger than industrial revolution jensen is very paranoid about losing if he just kept making his mainline chip people crush him on cost and performance acquiring Grock is how you get those resources to make more solutions for different parts of the market to stay king at the end of the day this is an economic war if the US and the West win in AI China will not rise to be the global hedgeimony but without AI China definitely will rise they're just going to outrun America hi I'm Matt Turk welcome back to the Matt podcast today I'm joined by the one person Wall Street and Silicon Valley turn to when they need to cut through the hardware hype Dylan Patel of Semi analysis we dove into many of the most important topics of today nvidia's massive move to acquire Grock the truth about the capex bubble whether the US power grid can actually handle the AI boom and the geopolitical

[译文] [前言片段]: AI 即将带来也许是人类历史上最大的变革，这是一场比工业革命更宏大的革命。Jensen（黄仁勋）对于失败非常偏执（paranoid）。如果他只坚持制造主线芯片，人们会在成本和性能上击垮他。收购 Groq 是获取资源的方式，以便为市场的不同部分制造更多解决方案，从而保持王者地位。归根结底，这是一场经济战。如果美国和西方在 AI 领域获胜，中国将不会崛起成为全球霸主；但如果没有 AI，中国绝对会崛起，他们会直接超越美国。嗨，我是 Matt Turk，欢迎回到 Matt 播客。今天加入我的是华尔街和硅谷在需要看穿硬件炒作时都会求助的那个人——SemiAnalysis 的 Dylan Patel。我们深入探讨了当今许多最重要的话题：Nvidia 收购 Groq 的大动作、资本支出（Capex）泡沫的真相、美国电网是否真能承受 AI 繁荣，以及地缘政治。

[原文] [Intro Clip]: chess match playing out between the US and China but I have to warn you this conversation went off the rails in the best possible way and we ended up going into all sorts of fun tangents like the strange phenomenon of Chinese romance dramas set inside semiconductor factories and what's really like when three AI famous roommates live together in SF please enjoy this fantastic conversation with Dylan hey Dylan welcome

[译文] [前言片段]: 美中之间正在上演的棋局。但我必须警告你，这次谈话以最好的方式“跑题”了，我们最终聊到了各种有趣的题外话，比如以半导体工厂为背景的中国浪漫剧这一奇怪现象，以及三位 AI 界的知名室友在旧金山合租的真实生活。请享受这段与 Dylan 的精彩对话。嘿，Dylan，欢迎。

[原文] [Dylan Patel]: hello how are you

[译文] [Dylan Patel]: 哈喽，你好吗？

[原文] [Matt Turck]: i'm great i'd love to start with Grock and Nvidia since it's still fresh so not so long ago Nvidia was saying that uh one GPU could do it all and now they're doing this acquisition non-exclusive deal with Grock what does that mean from your perspective

[译文] [Matt Turck]: 我很好。我想从 Groq 和 Nvidia 开始聊，因为这事儿还很新鲜。不久前，Nvidia 还在说一个 GPU 就能搞定一切，而现在他们正在进行这项收购——与 Groq 的非独家交易。从你的角度来看，这意味着什么？

[原文] [Dylan Patel]: it's very clear we're not sure where AI models are headed in terms of you know over the next few years what happens to the architecture but you know the thing that I think everyone is sort of like agreed on is models are pretty auto reggressive right next token generation is like the thing but beyond that right attention mechanisms changed the how how it works everything changes right could could change

[译文] [Dylan Patel]: 很明显，我们并不确定 AI 模型在架构方面未来几年会走向何方。但我认为大家某种程度上达成共识的是，模型是非常自回归（auto-reggressive）的，对吧？“下一个 Token 生成”（next token generation）是核心。但除此之外，注意力机制（attention mechanisms）改变了它的工作方式，一切都在变化，对吧，一切都可能改变。

[原文] [Dylan Patel]: and so what's interesting is the reason Nvidia one is because they just took like the widest surface area bet and then people kept developing models on that and that kind of shape worked but now the workload is so large that there is room for specialization that will give you 10x increases in certain domains right

[译文] [Dylan Patel]: 有趣的是，Nvidia 之前之所以赢了，是因为他们押注了“最广泛的覆盖面”（widest surface area bet），然后人们不断在该基础上开发模型，这种形态奏效了。但现在工作负载如此之大，以至于有了“专用化”（specialization）的空间，这能在某些特定领域给你带来 10 倍的提升，对吧？

[原文] [Dylan Patel]: in a general purpose workload grock doesn't work right you know it can't train it can't you know it can't inference really really large models um cost efficiently right you can't serve many many many users but what it can do is it can go bl screamingly fast right same with the cerebrous open AI deal but that's like one workload right uh very decode focused right gener doing auto reggressive tokens in a in a single stream super fast

[译文] [Dylan Patel]: 在通用工作负载下，Groq 是行不通的，对吧。你知道它不能训练，也不能真正具有成本效益地推理（inference）非常非常大的模型，你无法服务海量的用户。但它能做的是快到尖叫（screamingly fast），对吧？Cerebras 与 OpenAI 的交易也是同理。但这就像是针对单一工作负载，非常专注于解码（decode focused），在单流中极速地进行自回归 Token 生成。

[原文] [Dylan Patel]: another direction AI models could head right we don't know are models going to think in one token stream or is it actually they're constantly context switching right and they're going from they have this humongous humongous context and they're generating in multiple parallel streams right and so Google and openi have both released mechanisms of this with their pro models where the model actually doesn't just have one single chain of thought for reasoning it has multiple right

[译文] [Dylan Patel]: AI 模型可能朝向的另一个方向——我们不知道模型是会以单一 Token 流进行思考，还是实际上它们在不断地进行上下文切换（context switching）？它们可能拥有巨大无比的上下文，并且在多个并行流中生成。Google 和 OpenAI 都在其 Pro 模型中发布了这种机制，模型实际上不只有一条单一的推理思维链（chain of thought），而是有多条，对吧？

[原文] [Dylan Patel]: and then I don't exactly like you know and and and how they choose which one and what the final answer to you delivers is is an area of research um but there there is room for that kind of chip right something that works on very parallel a lot lot of streams of chain of thought and maybe the latency requirements are not as crazy right maybe you don't want to go blindingly fast right maybe you're okay with it being you know because I can spin up 100 parallel you know streams of thought or agents or whatever you want to call them maybe I I care a lot about cost there and because it's 100 in parallel instead of one going super super fast it's not as deep right the tree search or the depth of the inference is not as deep but it is much wider

[译文] [Dylan Patel]: 至于它们如何选择哪一条路径以及最终给你提供什么答案，这是一个研究领域。但这种芯片是有生存空间的，即那种能处理高度并行、大量思维链流的芯片。也许（这种场景下）延迟要求没那么疯狂，对吧？也许你不需要快到瞎眼。也许因为我可以启动 100 个并行的思维流或智能体（Agents）——随你怎么称呼——我可能更在乎成本。而且因为是 100 个并行而不是一个极速运行，它的深度没那么深，树搜索或推理的深度没那么深，但它要宽得多。

[原文] [Dylan Patel]: you know there's other parts of inference hey process do creating the KV cache so Nvidia has a chip for that right that's the CPX so they they've made the CPX they bought Grock for decode and then they still have their general purpose GPU so they've they're kind of trying to cover their bases because unlike the first wave of AI chip companies where they sort of just made chips and then tried to figure out where it would work right they had a thesis Grock and Cerebrus both as well as Samanova right which was put a lot of memory on the chip and not necessarily in the case of Cerebrus and Grock no memory off chip and in the case of Samanova less memory offchip or slower memory offchip with higher capacity you know they they sort of all made similar bets in that direction and it didn't work for a while until it kind of did right um because there's a workload that now necessitates it

[译文] [Dylan Patel]: 推理还有其他部分，比如创建 KV Cache（键值缓存）。Nvidia 为此专门有一款芯片，那就是 CPX。所以他们制造了 CPX，他们买下 Groq 用于解码（Decode），然后他们仍然拥有通用 GPU。他们是在试图全面布局（cover their bases）。因为这不像第一波 AI 芯片公司，那时候大家只是造出芯片然后试图搞清楚它能用在哪。Groq、Cerebras 以及 SambaNova 当时都有一个论点，就是在芯片上放大量内存。Cerebras 和 Groq 是片外无内存（no memory off chip），SambaNova 是片外内存较少或较慢但容量更大。他们都在那个方向上做了类似的押注，这在很长一段时间内都没奏效，直到现在某种程度上奏效了，因为现在的某种工作负载需要它。

[原文] [Dylan Patel]: nvidia recognizes they're they're the leader they're at the tent pole hey in one respect they can just run faster than everyone but it's kind of hard to be 2x better than Google or or OpenAI or whoever else's internal chip right to justify their you know 75% plus margins right and then they have to be 2x to 4x better to justify 4x better to justify their margins because that's what they're charging above COGS you know the question is what what architecture will deliver that

[译文] [Dylan Patel]: Nvidia 意识到他们是领导者，他们处于顶梁柱的位置。从一方面看，他们可以跑得比谁都快，但想要比 Google 或 OpenAI 或其他任何人的内部自研芯片好上 2 倍是很难的，尤其是为了证明他们 75% 以上的利润率（margins）是合理的，对吧？他们甚至需要好上 2 到 4 倍，甚至 4 倍，才能证明其利润率的合理性，因为那是他们在销货成本（COGS）之上收取的溢价。问题在于，什么样的架构能实现这一点？

[原文] [Dylan Patel]: well yes keep the programmability of their GPUs is great for training and for a lot of workloads but you know guess what i think I think a lot of people will just be downloading an open source model downloading an inference framework and pressing go right a little bit more complicated than that but that's that's going to be the consumption method for a lot of enterprises a lot of uh startups a lot of tech companies is they're just going to do that or they're going to rent the G GPUs or or rent the chips and then download an open source framework and model and go right

[译文] [Dylan Patel]: 是的，保留 GPU 的可编程性对训练和许多工作负载来说都很棒。但你知道吗，我认为很多人只会下载一个开源模型，下载一个推理框架，然后点击“运行”。当然比这稍微复杂一点，但这将是许多企业、初创公司和科技公司的消费方式。他们就是会这样做，或者租用 GPU 或芯片，然后下载开源框架和模型并开始运行。

[原文] [Dylan Patel]: and Nvidia recognizes this and hey there is room for products that aren't general purpose right the general purpose GPU will still probably be the main line for training and for a lot of inference and for costefficient inference but maybe blindingly fast or workloads that have a ton of prefill i.e creating the the KV cache maybe that those workloads could be different chips right and the CPX chip they announced right they say it's for the context processing creating the KV cache it's also really useful for video models because video models don't care about memory bandwidth and so you know why pay for the expensive memory that the general purpose chip has or why do what Grock is doing which is tying hundreds or thousands of chips together and not having memory but keeping the entire model on chip

[译文] [Dylan Patel]: Nvidia 意识到了这一点，即“非通用”产品是有生存空间的。通用 GPU 可能仍将是训练、许多推理任务以及高成本效益推理的主线。但也许对于极速任务，或者有大量预填充（Prefill，即创建 KV Cache）的工作负载，这些可能需要不同的芯片。他们发布的 CPX 芯片，他们说是用于上下文处理（Context Processing）、创建 KV Cache 的。它对视频模型也非常有用，因为视频模型不太在乎内存带宽。所以，为什么要为通用芯片上昂贵的内存买单？或者为什么要像 Groq 那样把成百上千个芯片连在一起，没有片外内存，而是把整个模型放在芯片上？

[原文] [Dylan Patel]: the trade-off for that of course is you need thousands of chips and you have less compute per chip and so like Nvidia's trying to capture the whole surface area because again you don't know where models are headed and it's hard to say where the research is headed

[译文] [Dylan Patel]: 当然，那样做（Groq 模式）的权衡是你需要数千个芯片，而且每个芯片的算力较低。所以 Nvidia 正试图捕捉整个“覆盖面”（surface area），因为同样地，你不知道模型会走向何方，也很难说研究会走向何方。

[原文] [Matt Turck]: and do you think it's a good thing for the market yet another one of those deals that's structured as a as a license but really an acquisition i certainly think it's not good from an anti-competitive sense right i don't think people should just be able to buy companies without like any antitrust like process at all now in the case of like a large company buying a startup I'm completely fine with it the flip side is like hey we know the deal is happening right uh this happened for a company I was an adviser for Nvidia acquired in fabrica just maybe a few months before they did Grock and similar style of deal right if someone wanted to strike it down that's the biggest limbo right we've seen this happen in venture and you probably know more stories of this but like a company trying to get acquired they get stuck in limbo for like a year and then it falls apart stories yeah it falls apart the deal did because some regulatory BS and now the company was and the founders were focused on getting the deal done instead of like making the product better for a year and now they're like behind or you know they they they weren't focused on growth as much right you know you only have so much time as a founder so in that sense I like the license deals right

[译文] [Matt Turck]: 你认为这对市场是件好事吗？这又是一笔名义上是“授权”但实际上是“收购”的交易。

[Dylan Patel]: 我当然认为从反竞争的角度来看这并不好，对吧？我不认为人们应该能够完全不经过任何反垄断流程就买下公司。如果是大公司收购初创公司，我完全没意见。但这事的另一面是，嘿，我们知道交易正在发生。这种事发生在我担任顾问的一家公司身上——Nvidia 收购了 Infabrica，就在他们搞定 Groq 之前的几个月，也是类似的交易风格。如果有人想否决它，那就会陷入最大的僵局（limbo）。我们在风投圈见过这种情况，你可能知道更多这类故事：一家公司试图被收购，结果陷入了一年的僵局，然后交易黄了。是的，交易黄了，因为一些监管的破事（regulatory BS）。而现在这家公司和创始人在这一年里专注于搞定交易而不是优化产品，现在他们落后了，或者他们没那么专注于增长了。作为创始人你的时间是有限的。所以从这个意义上说，我喜欢这种“授权”式的交易。

[原文] [Matt Turck]: so now is uh Nvidia also dominating the the inference market is there any world where Nvidia is no longer the king or they seem to be getting stronger

[译文] [Matt Turck]: 那么现在 Nvidia 是不是也主宰了推理市场？有没有这种可能，即 Nvidia 不再是王者，还是说他们看起来越来越强了？

[原文] [Dylan Patel]: I think the thing about Nvidia is they take the Andy Grove mentality like more serious than anyone else right like okay fine Google like implemented OKRs because Intel did it but that's like you know management stuff right only the paranoid survive right this is like core to the Bay Area um core to Nvidia um Jensen is very paranoid about losing right these specializations if he just kept making his mainline chip would mean people could you know point point solutions for specific parts of the market would crush him on cost and performance then he can't justify his margin that's a threat to Nvidia's business model as a whole

[译文] [Dylan Patel]: 我认为 Nvidia 的特点是，他们比任何人都更认真地贯彻安迪·格鲁夫（Andy Grove）的心态。好吧，Google 实施 OKR 是因为 Intel 做了，但那是管理层面的东西。“只有偏执狂才能生存”（Only the paranoid survive），这就像是湾区的核心，也是 Nvidia 的核心。Jensen 对于失败非常偏执（paranoid）。如果他只坚持制造主线芯片，这些“专用化”趋势意味着人们可以用针对市场特定部分的“单点解决方案”（point solutions），在成本和性能上击垮他。那样他就无法证明他的高利润率是合理的，这对 Nvidia 的整个商业模式都是威胁。

[原文] [Dylan Patel]: especially if the best model only changes every 3 months or the model you want to roll out okay well then you're going to have three months to figure out how to make a model work on one chip architecture for that point solution and you know it's fine software software advantage of Nvidia is not that important then Jensen's super paranoid about losing and frankly it's really hard to hire enough talented chip people when you look across the market there is only a few companies who have successfully created a chip architecture software to run the models accurately run the run the models accurately right

[译文] [Dylan Patel]: 特别是如果最好的模型每 3 个月才变一次，或者你要推出的模型确定了，那你就有三个月的时间去研究如何让模型在那个针对特定点解决方案的芯片架构上运行。这其实没问题，那时 Nvidia 的软件优势就没那么重要了。Jensen 超级偏执于失败。坦率地说，很难招到足够多的天才芯片人才。环顾市场，只有寥寥几家公司成功创造了芯片架构和软件来准确地运行模型，是准确地运行模型，对吧？

[原文] [Dylan Patel]: like cuz you can look at random APIs of say an Alibaba Quen model and different people are doing all sorts of tricks like quantizing it but also many other tricks which then end up like making the model quality lower you know building a rack scale solution networking thousands of chips together and then deploying an API and Grock did the whole thing with frankly not that many people

[译文] [Dylan Patel]: 因为你可以看看比如阿里巴巴 Qwen 模型的各种随机 API，不同的人在搞各种花样，比如量化（quantizing），但也还有很多其他的花样，结果导致模型质量下降。你知道，构建一个机架级（rack scale）解决方案，把成千上万个芯片联网在一起，然后部署一个 API，Groq 用坦白说并不多的人手就做到了这一切。

[原文] [Dylan Patel]: so now it's like okay well I'm Nvidia I want to make four different chip architectures and actually four different point solutions maybe the general purpose and then one here one here one here and in addition my general purpose thing is actually not just like a GPU chip it's like GPU chips CPU chips networking chips NV switch nicks like you know there's many many chips and each of those chips has many chiplets you don't have enough engineering resources right and so like acquiring gro is like how you get those resources to make more solutions for different parts of the market

[译文] [Dylan Patel]: 所以现在情况是：好吧，我是 Nvidia，我想制造四种不同的芯片架构，实际上是四种不同的单点解决方案。也许是通用型，然后这儿一个，那儿一个，这儿再一个。而且我的通用型产品不仅仅是一个 GPU 芯片，它包括 GPU 芯片、CPU 芯片、网络芯片、NV Switch、网卡等等。你知道有非常多的芯片，每个芯片又有许多小芯片（chiplets）。你根本没有足够的工程资源，对吧？所以，收购 Groq 就像是你获取这些资源（人才）的方式，以便为市场的不同部分制造更多的解决方案。

紧接上一章关于 Nvidia 战略转型的讨论，本章 Dylan 首先简要点评了 Nvidia 面临的竞争威胁（初创公司与巨头自研芯片），随即深入探讨了核心话题：CUDA 护城河是否依然坚固？

Dylan 提出了一个反直觉的观点：传统的“CUDA 护城河”（即编写底层内核代码的能力）正在消失，因为大多数人不再手写 CUDA。取而代之的是，Nvidia 正通过解决推理成本（Inference Cost）和KV Cache 管理等新一代复杂工程问题，构建起一层更隐蔽、更难以逾越的“新软件护城河”。