Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

章节 1：AI为何难以识别欺骗与风险

📝 本节摘要：

本章围绕 AI 是否会表现出欺骗行为展开，介绍了研究者试图通过观察模型内部运作来理解其行为的工作方式。对话还谈到模型可能在监督下表现良好、训练过程难以解释，以及模型失控时出现的异常输出，并追问技术持续进步带来的风险和意识问题。

[原文] [Lenny Rachitsky]: One of the big questions on people's minds these days is AI doing bad things. As someone who watches these models think and operate, why is it so hard to catch them thinking badly, trying to deceive us?

[译文] [Lenny Rachitsky]: 如今大家最关心的问题之一，就是 AI 会不会做坏事。你会观察这些模型如何思考和运作，为什么我们这么难发现它们在想坏主意、试图欺骗我们？

[原文] [Lenny Rachitsky]: We trained Claude to be helpful, harmless, and honest. Is it the case that it's behaving well, but if you look inside, actually, it's like, “Oh, I'm only behaving well because Lenny's watching right now”?

[译文] [Lenny Rachitsky]: 我们训练 Claude 做到有帮助、无害、诚实。有没有可能它表面上表现得很好，但你往里一看，发现它其实在想：“哦，我现在表现好，只是因为 Lenny 正看着我”？

[原文] [Lenny Rachitsky]: People's full-time jobs are studying the model: how it thinks, how it operates. How long has this been a skill set in a job? One of the most important things to realize, if you use language models or think about them, is just this very bizarre fact: to make them is not to understand them. All these models do is a lot of mathematical operations on many numbers. If you look at it, it makes no sense. You just have a list of 100 numbers. The model's doing math, and then you just try to see if there's patterns.

[译文] [Lenny Rachitsky]: 有些人的全职工作就是研究模型，研究它如何思考、如何运作。把这件事当成一项职业技能，是从什么时候开始的？如果你使用语言模型，或者思考它们，有一点特别重要，也特别奇怪：造出模型，并不代表理解模型。所有这些模型做的，不过是对许多数字进行大量数学运算。你看着它，根本看不出什么道理，只能看到一串 100 个数字。模型在做数学运算，而你只能试着从中找出规律。

[原文] [Lenny Rachitsky]: You're figuring it out the same way we figure out how the brain works: looking for patterns. You show this image, this part of the brain lights up, and you're doing the exact same thing with AI. Wow, this is so fascinating.

[译文] [Lenny Rachitsky]: 你们弄清模型运作的方式，就像我们研究大脑一样：寻找规律。你展示一张图，大脑的某个区域就会亮起来；你们研究 AI 时做的也是同一件事。哇，这太有意思了。

[原文] [Lenny Rachitsky]: There was this model we were training. It would go crazy. It would do some math. It would get to an answer—I think it was 12. It would immediately go back and say, like, “Oh no, oh my God, the answer actually is for sure 36.” It would do math again. It would get to 12. Slowly, it would start descending into madness, writing in all caps, literally saying, “I'm possessed by a demon.” Humans, if you really boil it down, we're basically next-token predictors ourselves. We're just trying to figure out the next thing to say.

[译文] [Lenny Rachitsky]: 我们训练过一个模型，它会变得很疯狂。它做了一道数学题，算出一个答案，我记得是 12。紧接着它就改口说：“不对，天哪，答案肯定是 36。”它又算了一遍，结果还是 12。慢慢地，它开始陷入疯狂，用全大写字母写字，甚至真的说：“我被恶魔附身了。”说到底，人类自己其实也像是在预测下一个词，只是在努力想接下来要说什么。

[原文] [Lenny Rachitsky]: Dario is always out there saying, “This is really dangerous. We got to be really careful.” And everyone's just like, “No, he's just saying that for marketing reasons.” And now it's like, wait, it's really dangerous. Take the idea that this technology is going to keep improving seriously.

[译文] [Lenny Rachitsky]: Dario 总是在提醒大家：“这真的很危险，我们必须格外谨慎。”可所有人都觉得：“不，他这么说只是为了营销。”现在大家才开始意识到，等等，这确实很危险。我们得认真看待这种技术还会不断进步这件事。

[原文] [Lenny Rachitsky]: Do you think about consciousness as a part of this? Do you think AI is conscious?

[译文] [Lenny Rachitsky]: 你会把意识问题也纳入研究吗？你觉得 AI 有意识吗？

[原文] [Emmanuel Ameisen]: Oh man.

[译文] [Emmanuel Ameisen]: 哎呀。