Dario Amodei: Anthropic CEO on Claude, AGI & the F

章节 01：扩展定律（Scaling Laws）的历史与假设

📝 本节摘要：

本章涵盖了访谈的正式开场与核心理论的起源。Lex Fridman 首先介绍了 Anthropic 的三位重量级嘉宾：CEO Dario Amodei、研究员 Amanda Askell 和 Chris Olah。随后，对话切入正题，Dario 回顾了他在 2014 年至 2017 年间的职业经历。他描述了自己在百度（Baidu）与吴恩达（Andrew Ng）共事时，通过观察语音识别系统的训练，首次萌生了“简单扩大规模（Scaling）就能提升性能”的直觉。这种直觉在 2017 年看到 GPT-1 的结果时得到了验证，使他确信只要持续增加数据和算力，AI 就能处理极其广泛的认知任务。

[原文] [Lex Fridman]: The following is a conversation with Dario Amodei, CEO of Anthropic, the company that created Claude that is currently and often at the top of most LLM benchmark leaderboards.

[译文] [Lex Fridman]: 接下来的内容是与 Anthropic 公司 CEO Dario Amodei 的对话，该公司创造了 Claude，目前该模型经常在大多数大语言模型（LLM）基准测试排行榜上名列前茅。

[原文] [Lex Fridman]: On top of that, Dario and the Anthropic team have been outspoken advocates for taking the topic of AI safety very seriously, and they have continued to publish a lot of fascinating AI research on this and other topics.

[译文] [Lex Fridman]: 除此之外，Dario 和 Anthropic 团队一直是直言不讳的倡导者，主张非常严肃地对待 AI 安全这一议题，并且他们持续在该领域及其他课题上发表大量引人入胜的 AI 研究成果。

[原文] [Lex Fridman]: I'm also joined afterwards by two other brilliant people from Anthropic.

[译文] [Lex Fridman]: 稍后，还有另外两位来自 Anthropic 的杰出人士也将加入我的谈话。

[原文] [Lex Fridman]: First Amanda Askell, who is a researcher working on alignment and fine tuning of Claude, including the design of Claude's character and personality.

[译文] [Lex Fridman]: 首先是 Amanda Askell，她是一名研究员，致力于 Claude 的对齐（alignment）和微调（fine tuning）工作，包括设计 Claude 的角色和性格。

[原文] [Lex Fridman]: A few folks told me she has probably talked with Claude more than any human at Anthropic.

[译文] [Lex Fridman]: 有几个人告诉我，她可能比 Anthropic 的任何人都更多地与 Claude 交谈过。

[原文] [Lex Fridman]: So she was definitely a fascinating person to talk to about prompt engineering and practical advice on how to get the best out of Claude.

[译文] [Lex Fridman]: 所以，如果要探讨提示词工程（prompt engineering）以及如何充分利用 Claude 的实用建议，她绝对是一个令人着迷的对话对象。

[原文] [Lex Fridman]: After that, Chris Olah stopped by for a chat.

[译文] [Lex Fridman]: 在那之后，Chris Olah 也过来聊了聊。

[原文] [Lex Fridman]: He's one of the pioneers of the field of mechanistic interpretability, which is an exciting set of efforts that aims to reverse engineer neural networks to figure out what's going on inside, inferring behaviors from neural activation patterns inside the network.

[译文] [Lex Fridman]: 他是机械可解释性（mechanistic interpretability）领域的先驱之一，这是一系列令人兴奋的尝试，旨在对神经网络进行逆向工程以弄清其内部运作机制，通过网络内部的神经激活模式来推断其行为。

[原文] [Lex Fridman]: This is a very promising approach for keeping future super intelligent AI systems safe.

[译文] [Lex Fridman]: 这是确保未来超级智能 AI 系统安全的一种非常有前途的方法。

[原文] [Lex Fridman]: For example, by detecting from the activations when the model is trying to deceive the human it is talking to.

[译文] [Lex Fridman]: 例如，通过激活模式检测模型何时试图欺骗与之交谈的人类。

[原文] [Lex Fridman]: This is the "Lex Fridman Podcast." To support it, please check out our sponsors in the description.

[译文] [Lex Fridman]: 这里是《Lex Fridman 播客》。如果要支持本节目，请查看描述中的赞助商信息。

[原文] [Lex Fridman]: And now, dear friends, here's Dario Amodei.

[译文] [Lex Fridman]: 现在，亲爱的朋友们，有请 Dario Amodei。

[原文] [Lex Fridman]: Let's start with the big idea of scaling laws and the Scaling Hypothesis. What is it? What is its history? And where do we stand today?

[译文] [Lex Fridman]: 让我们从扩展定律（scaling laws）和扩展假说（Scaling Hypothesis）这个大概念开始吧。它是什么？它的历史是怎样的？我们今天处于什么位置？

[原文] [Dario Amodei]: So I can only describe it as, you know, as it relates to kind of my own experience.

[译文] [Dario Amodei]: 我只能根据我自己的经历来描述它。

[原文] [Dario Amodei]: But I've been in the AI field for about 10 years and it was something I noticed very early on.

[译文] [Dario Amodei]: 我进入 AI 领域大约 10 年了，这是我很早就注意到的事情。

[原文] [Dario Amodei]: So I first joined the AI world when I was working at Baidu with Andrew Ng in late 2014, which is almost exactly 10 years ago now.

[译文] [Dario Amodei]: 我最初进入 AI 世界是在 2014 年底，当时我在百度（Baidu）与吴恩达（Andrew Ng）共事，距离现在几乎正好 10 年。

[原文] [Dario Amodei]: And the first thing we worked on was speech recognition systems.

[译文] [Dario Amodei]: 我们当时做的第一件事就是语音识别系统。

[原文] [Dario Amodei]: And in those days I think deep learning was a new thing, it had made lots of progress, but everyone was always saying, we don't have the algorithms we need to succeed.

[译文] [Dario Amodei]: 在那个年代，深度学习还是个新鲜事物，虽然已经取得了很多进展，但每个人都还在说，我们还没有获得成功所需的算法。

[原文] [Dario Amodei]: You know, we're not, we're only matching a tiny, tiny fraction.

[译文] [Dario Amodei]: 你知道，我们还没有，我们只匹配了极其微小的一部分。

[原文] [Dario Amodei]: There's so much we need to kind of discover algorithmically.

[译文] [Dario Amodei]: 在算法层面上，我们还有太多东西需要去探索。

[原文] [Dario Amodei]: We haven't found the picture of how to match the human brain.

[译文] [Dario Amodei]: 我们还没找到能够匹敌人类大脑的图景。

[原文] [Dario Amodei]: And when, you know, in some ways I was fortunate, I was kind of, you know, you can have almost beginner's luck, right?

[译文] [Dario Amodei]: 而当……你知道，在某些方面我很幸运，某种程度上，这几乎算是初学者的运气，对吧？

[原文] [Dario Amodei]: I was like a newcomer to the field and, you know, I looked at the neural net that we were using for speech, the recurrent neural networks, and I said, I don't know, what if you make them bigger and give them more layers?

[译文] [Dario Amodei]: 我当时就像这个领域的新人，我看着我们用于语音识别的神经网络——也就是循环神经网络（recurrent neural networks）——我说，我不知道，如果不把它们做得更大，给它们加更多层会怎样？

[原文] [Dario Amodei]: And what if you scale up the data along with this, right?

[译文] [Dario Amodei]: 如果与此同时你也扩大数据规模会怎样？

[原文] [Dario Amodei]: I just saw these as like independent dials that you could turn.

[译文] [Dario Amodei]: 我只是把这些看作是可以转动的独立旋钮。

[原文] [Dario Amodei]: And I noticed that the model started to do better and better as you gave them more data, as you made the models larger, as you trained them for longer.

[译文] [Dario Amodei]: 我注意到，当你给它们更多数据，当你把模型做得更大，当你训练它们的时间更长时，模型开始表现得越来越好。

[原文] [Dario Amodei]: And I didn't measure things precisely in those days, but along with colleagues, we very much got the informal sense that the more data and the more compute and the more training you put into these models, the better they perform.

[译文] [Dario Amodei]: 那时候我没有精确测量这些指标，但我和同事们强烈地产生了一种非正式的感觉：你投入这些模型的数据越多、算力（compute）越多、训练越多，它们的表现就越好。

[原文] [Dario Amodei]: And so initially my thinking was, hey, maybe that is just true for speech recognition systems, right?

[译文] [Dario Amodei]: 所以起初我的想法是，嘿，也许这只适用于语音识别系统，对吧？

[原文] [Dario Amodei]: Maybe that's just one particular quirk, one particular area.

[译文] [Dario Amodei]: 也许那只是一个特定的怪现象，一个特定的领域。

[原文] [Dario Amodei]: I think it wasn't until 2017 when I first saw the results from GPT-1 that it clicked for me that language is probably the area in which we can do this.

[译文] [Dario Amodei]: 我想直到 2017 年，当我第一次看到 GPT-1 的结果时，我才突然明白（clicked for me），语言可能就是我们可以实现这一点的领域。

[原文] [Dario Amodei]: We can get trillions of words of language data, we can train on them.

[译文] [Dario Amodei]: 我们可以获得数万亿字的语言数据，我们可以在这些数据上进行训练。

[原文] [Dario Amodei]: And the models we were training those days were tiny.

[译文] [Dario Amodei]: 我们那个年代训练的模型非常小。

[原文] [Dario Amodei]: You could train them on one to eight GPUs whereas, you know, now we train jobs on tens of thousands, soon going to hundreds of thousands of GPUs.

[译文] [Dario Amodei]: 那时候你可以在 1 到 8 个 GPU 上训练它们，而你知道，现在我们的训练任务是在数万个 GPU 上进行的，很快就要达到数十万个 GPU。

[原文] [Dario Amodei]: And so when I saw those two things together and, you know, there were a few people like Ilya Sutskever, who you've interviewed, who had somewhat similar views, right?

[译文] [Dario Amodei]: 所以当我把这两件事结合起来看时……你知道，还有少数人，比如你采访过的 Ilya Sutskever，也持有类似的观点，对吧？

[原文] [Dario Amodei]: He might have been the first one, although I think a few people came to similar views around the same time, right?

[译文] [Dario Amodei]: 他可能是第一个，尽管我认为有几个人大约在同一时间得出了类似的观点，对吧？

[原文] [Dario Amodei]: There was, you know, Rich Sutton's Bitter Lesson, there was Gwern wrote about the Scaling Hypothesis.

[译文] [Dario Amodei]: 比如 Rich Sutton 的文章《苦涩的教训》（The Bitter Lesson），还有 Gwern 写的关于扩展假说（Scaling Hypothesis）的文章。

[原文] [Dario Amodei]: But I think somewhere between 2014 and 2017 was when it really clicked for me, when I really got conviction that, hey, we're gonna be able to do these incredibly wide cognitive tasks if we just scale up the models.

[译文] [Dario Amodei]: 但我认为大概在 2014 年到 2017 年之间，我真正顿悟了，我真正确信：嘿，只要我们要扩大模型规模，我们将能够完成这些极其广泛的认知任务。

[原文] [Dario Amodei]: And at every stage of scaling, there are always arguments.

[译文] [Dario Amodei]: 而在扩展的每一个阶段，总会存在争议。

[原文] [Dario Amodei]: And you know, when I first heard them, honestly, I thought probably I'm the one who's wrong, and, you know, all these experts in the field are right.

[译文] [Dario Amodei]: 你知道，当我第一次听到这些争论时，老实说，我想可能我才是错的，而领域内的所有这些专家是对的。

[原文] [Dario Amodei]: They know the situation better than I do, right?

[译文] [Dario Amodei]: 他们比我更了解情况，对吧？

[原文] [Dario Amodei]: There's, you know, the Chomsky argument about like, you can get syntactics, but you can't get semantics.

[译文] [Dario Amodei]: 比如有乔姆斯基（Chomsky）的论点，认为你可以获得句法（syntactics），但你无法获得语义（semantics）。