Deep Dive into LLMs like ChatGPT

章节 1：引言与预训练数据管道 (Introduction and Pre-training Data Pipeline)

📝 本节摘要：

在本章中，演讲者Andre Karpathy首先介绍了视频的主旨：向普通受众全面解析ChatGPT等大语言模型背后的工作原理。核心内容聚焦于模型构建的第一阶段——预训练（Pre-training）。该阶段始于从互联网下载海量数据（如Common Crawl），并经过一系列严格的管道处理：URL过滤、文本提取、语言筛选及隐私信息（PII）移除，最终形成如FineWeb这样的高质量数据集。随后，讲解了计算机如何通过分词（Tokenization）技术（具体为字节对编码BPE），将原始文本转化为神经网络能够处理的整数序列（Tokens）。

[原文] [Andrej Karpathy]: hi everyone so I've wanted to make this video for a while it is a comprehensive but General audience introduction to large language models like Chachi PT and what I'm hoping to achieve in this video is to give you kind of mental models for thinking through what it is that this tool is it is obviously magical and amazing in some respects it's uh really good at some things not very good at other things and there's also a lot of sharp edges to be aware of so what is behind this text box you can put anything in there and press enter but uh what should we be putting there and what are these words generated back how does this work and what what are you talking to exactly so I'm hoping to get at all those topics in this video we're going to go through the entire pipeline of how this stuff is built but I'm going to keep everything uh sort of accessible to a general audience so let's take a look at first how you build something like chpt and along the way I'm going to talk about um you know some of the sort of cognitive psychological implications of the tools

[译文] [Andrej Karpathy]: 大家好，我想做这个视频已经有一段时间了，这是一个面向普通受众的、关于像ChatGPT这样的大型语言模型（Large Language Models）的全面介绍。我希望通过这个视频实现的是，给你们提供一种思维模型，用来思考这个工具到底是什么。这显然在某些方面是神奇和令人惊叹的，它在某些事情上非常擅长，但在另一些事情上却不太擅长，而且还有很多需要注意的“锋利边缘”（即缺陷或风险）。那么这个文本框背后是什么呢？你可以输入任何内容并按回车，但我们应该在那里输入什么？这些生成的回复文字是什么？它是如何工作的？你到底在和什么对话？所以我希望在这个视频中探讨所有这些话题。我们将完整地过一遍这些东西是如何构建的整个管道（pipeline），但我会保持所有内容对普通观众来说都是通俗易懂的。所以让我们首先看看你是如何构建像ChatGPT这样的东西的，在此过程中，我会谈论一些关于这些工具的认知心理学层面的影响。

[原文] [Andrej Karpathy]: okay so let's build Chachi PT so there's going to be multiple stages arranged sequentially the first stage is called the pre-training stage and the first step of the pre-training stage is to download and process the internet now to get a sense of what this roughly looks like I recommend looking at this URL here so um this company called hugging face uh collected and created and curated this data set called Fine web and they go into a lot of detail on this block post on how how they constructed the fine web data set and all of the major llm providers like open AI anthropic and Google and so on will have some equivalent internally of something like the fine web data set

[译文] [Andrej Karpathy]: 好的，那我们来构建ChatGPT。这包含按顺序排列的多个阶段，第一个阶段被称为预训练阶段（Pre-training stage）。预训练阶段的第一步是下载并处理互联网数据。为了让你大致了解这看起来是什么样子，我建议看看这个网址。这家叫Hugging Face的公司收集、创建并整理了一个名为FineWeb的数据集。他们在这篇博文中详细介绍了他们是如何构建FineWeb数据集的。所有主要的大语言模型提供商，如OpenAI、Anthropic和Google等，内部都会有某种类似于FineWeb数据集的等价物。

[原文] [Andrej Karpathy]: so roughly what are we trying to achieve here we're trying to get ton of text from the internet from publicly available sources so we're trying to have a huge quantity of very high quality documents and we also want very large diversity of documents because we want to have a lot of knowledge inside these models so we want large diversity of high quality documents and we want many many of them and achieving this is uh quite complicated and as you can see here takes multiple stages to do well so let's take a look at what some of these stages look like in a bit for now I'd like to just like to note that for example the fine web data set which is fairly representative what you would see in a production grade application actually ends up being only about 44 terabyt of dis space um you can get a USB stick for like a terabyte very easily or I think this could fit on a single hard drive almost today so this is not a huge amount of data at the end of the day even though the internet is very very large we're working with text and we're also filtering it aggressively so we end up with about 44 terabytes in this example

[译文] [Andrej Karpathy]: 那么我们在这里大致想要实现什么呢？我们试图从互联网的公开来源获取大量的文本。所以我们试图拥有海量的高质量文档，我们也希望文档具有非常大的多样性，因为我们希望这些模型内部拥有大量的知识。所以我们需要具有高度多样性的高质量文档，而且我们需要非常多这样的文档。要实现这一点相当复杂，正如你在这里看到的，这需要多个阶段才能做好。稍后我们来看看其中一些阶段是什么样的。现在我只想指出，例如FineWeb数据集——这在生产级应用中具有相当的代表性——实际上最终只占用了大约44TB的磁盘空间。你可以很容易买到一个1TB的USB盘，或者我认为这几乎可以装进今天的单个硬盘里。所以归根结底，这并不是一个巨大的数据量，尽管互联网非常非常大，但我们处理的是文本，而且我们还在积极地过滤它，所以在这个例子中我们最终得到了大约44TB的数据。

[原文] [Andrej Karpathy]: so let's take a look at uh kind of what this data looks like and what some of these stages uh also are so the starting point for a lot of these efforts and something that contributes most of the data by the end of it is Data from common crawl so common craw is an organization that has been basically scouring the internet since 2007 so as of 2024 for example common CW has indexed 2.7 billion web pages uh and uh they have all these crawlers going around the internet and what you end up doing basically is you start with a few seed web pages and then you follow all the links and you just keep following links and you keep indexing all the information and you end up with a ton of data of the internet over time so this is usually the starting point for a lot of the uh for a lot of these efforts

[译文] [Andrej Karpathy]: 那么让我们来看看这些数据是什么样的，以及其中一些阶段是什么。许多这类工作的起点，也是最终贡献大部分数据来源的，是来自Common Crawl的数据。Common Crawl是一个组织，基本上从2007年开始就在搜寻互联网。截至2024年，Common Crawl已经索引了27亿个网页。他们有这些爬虫程序（crawlers）在互联网上到处跑。你基本上要做的就是从一些种子网页开始，然后跟随所有的链接，不断地跟随链接，不断地索引所有的信息，随着时间的推移，你就得到了大量的互联网数据。这通常是许多此类工作的起点。

[原文] [Andrej Karpathy]: now this common C data is quite raw and is filtered in many many different ways so here they Pro they document this is the same diagram they document a little bit the kind of processing that happens in these stages so the first thing here is something called URL filtering so what that is referring to is that there's these block lists of uh basically URLs that are or domains that uh you don't want to be getting data from so usually this includes things like U malware websites spam websites marketing websites uh racist websites adult sites and things like that so there's a ton of different types of websites that are just eliminated at this stage because we don't want them in our data set

[译文] [Andrej Karpathy]: 这个Common Crawl数据非常原始，并且会被以许多不同的方式进行过滤。这里他们记录了——这是同一张图表——他们记录了一些在这些阶段发生的处理过程。首先是所谓的URL过滤（URL filtering）。这指的是有一些黑名单，基本上是你不想从中获取数据的URL或域名。这通常包括恶意软件网站、垃圾邮件网站、营销网站、种族主义网站、成人网站之类的东西。所以在这一阶段有大量不同类型的网站被剔除，因为我们不希望它们出现在我们的数据集中。

[原文] [Andrej Karpathy]: um the second part is text extraction you have to remember that all these web pages this is the raw HTML of these web pages that are being saved by these crawlers so when I go to inspect here this is what the raw HTML actually looks like you'll notice that it's got all this markup uh like lists and stuff like that and there's CSS and all this kind of stuff so this is um computer code almost for these web pages but what we really want is we just want this text right we just want the text of this web page and we don't want the navigation and things like that so there's a lot of filtering and processing uh and heris that go into uh adequately filtering for just their uh good content of these web pages

[译文] [Andrej Karpathy]: 第二部分是文本提取（text extraction）。你要记得所有这些网页，爬虫保存的是这些网页的原始HTML。所以当我在这里点击检查时，这就是原始HTML实际的样子。你会注意到它有所有的标记，比如列表之类的，还有CSS和所有这类东西。所以这几乎是这些网页的计算机代码。但我们真正想要的只是文本，对吧？我们只想要这个网页的文本，我们不想要导航栏之类的东西。所以有大量的过滤和处理以及启发式方法（heuristics）被用于充分过滤，只保留这些网页中的优质内容。

[原文] [Andrej Karpathy]: the next stage here is language filtering so for example fine web filters uh using a language classifier they try to guess what language every single web page is in and then they only keep web pages that have more than 65% of English as an example and so you can get a sense that this is like a design decision that different companies can uh can uh take for themselves what fraction of all different types of languages are we going to include in our data set because for example if we filter out all of the Spanish as an example then you might imagine that our model later will not be very good at Spanish because it's just never seen that much data of that language and so different companies can focus on multilingual performance to uh to a different degree as an example so fine web is quite focused on English and so their language model if they end up training one later will be very good at English but not may be very good at other languages

[译文] [Andrej Karpathy]: 接下来的阶段是语言过滤（language filtering）。例如，FineWeb使用语言分类器进行过滤，他们试图猜测每个网页是什么语言，然后作为一个例子，他们只保留那些英语含量超过65%的网页。你可以感觉到这就像是一个设计决策，不同的公司可以自己决定在我们的数据集中包含多少比例的各种不同语言。因为例如，如果我们过滤掉所有的西班牙语，那么你可以想象我们的模型之后在西班牙语方面就不会很好，因为它从未见过那么多该语言的数据。所以不同的公司可以不同程度地关注多语言性能。例如FineWeb非常专注于英语，所以如果他们后来训练一个语言模型，它在英语方面会非常好，但在其他语言方面可能就不那么好了。

[原文] [Andrej Karpathy]: after language filtering there's a few other filtering steps and D duplication and things like that um finishing with for example the pii removal this is personally identifiable information so as an example addresses Social Security numbers and things like that you would try to detect them and you would try to filter out those kinds of web pages from the the data set as well so there's a lot of stages here and I won't go into full detail but it is a fairly extensive part of the pre-processing and you end up with for example the fine web data set so when you click in on it uh you can see some examples here of what this actually ends up looking like and anyone can download this on the huging phase web page and so here are some examples of the final text that ends up in the training set

[译文] [Andrej Karpathy]: 在语言过滤之后，还有一些其他的过滤步骤和去重（De-duplication）之类的操作，最后以例如PII移除（个人身份信息移除）结束。例如地址、社会安全号码之类的东西，你会试图检测它们，并试图从数据集中过滤掉包含这些内容的网页。这里有很多阶段，我不会完全详细地讲，但这是预处理中相当广泛的一部分。最终你会得到例如FineWeb数据集。当你点击进去时，你可以看到一些例子，看看它实际上最终是什么样子的，任何人都可以在Hugging Face网页上下载它。这里有一些最终进入训练集的文本示例。

[原文] [Andrej Karpathy]: so this is some article about tornadoes in 2012 um so there's some t tadoes in 2020 in 2012 and what happened uh this next one is something about did you know you have two little yellow 9vt battery sized adrenal glands in your body okay so this is some kind of a odd medical article so just think of these as basically uh web pages on the internet filtered just for the text in various ways and now we have a ton of text 40 terabytes off it and that now is the starting point for the next step of this stage now I wanted to give you an intuitive sense of where we are right now so I took the first 200 web pages here and remember we have tons of them and I just take all that text and I just put it all together concatenate it and so this is what we end up with we just get this just just raw text raw internet text and there's a ton of it even in these 200 web pages so I can continue zooming out here and we just have this like massive tapestry of Text data and this text data has all these p patterns and what we want to do now is we want to start training neural networks on this data so the neural networks can internalize and model how this text flows right so we just have this giant texture of text and now we want to get neural Nets that mimic it

[译文] [Andrej Karpathy]: 这是一篇关于2012年龙卷风的文章，讲的是2012年发生的一些龙卷风以及发生了什么。下一个是关于“你知道你体内有两个像9伏电池大小的小黄色肾上腺吗”，好吧，这是某种奇怪的医学文章。所以就把这些基本上看作是互联网上的网页，经过各种方式过滤只保留了文本。现在我们有大量的文本，40TB的文本，这就是这个阶段下一步的起点。我想让你们直观地感受一下我们现在的处境。所以我取了这里的前200个网页，记得我们有海量的网页，我把所有的文本放在一起，把它们串联（concatenate）起来。这就是我们最终得到的结果，我们得到了这些原始文本，原始的互联网文本，即使在这200个网页中也有大量内容。我可以继续在这里缩小视图，我们就像拥有了一幅巨大的文本数据挂毯。这些文本数据包含了所有这些模式，我们现在要做的是开始在这个数据上训练神经网络，以便神经网络能够内化并模拟这些文本是如何流动的，对吧？所以我们就有了这巨大的文本纹理，现在我们想让神经网络来模仿它。

[原文] [Andrej Karpathy]: okay now before we plug text into neural networks we have to decide how we're going to represent this text uh and how we're going to feed it in now the way our technology works for these neuron Lots is that they expect a one-dimensional sequence of symbols and they want a finite set of symbols that are possible and so we have to decide what are the symbols and then we have to represent our data as one-dimensional sequence of those symbols so right now what we have is a onedimensional sequence of text it starts here and it goes here and then it comes here Etc so this is a onedimensional sequence even though on my monitor of course it's laid out in a two-dimensional way but it goes from left to right and top to bottom right so it's a one-dimensional sequence of text

[译文] [Andrej Karpathy]: 好的，在我们将文本输入神经网络之前，我们必须决定如何表示这些文本，以及如何将其输入进去。我们的神经网络技术的工作方式是，它们期望一个一维的符号序列（one-dimensional sequence of symbols），并且它们需要一个有限的可能符号集。所以我们必须决定什么是符号，然后我们必须把我们的数据表示为这些符号的一维序列。目前我们拥有的是一维的文本序列，它从这里开始，到这里，然后到这里等等。这是一个一维序列，尽管在我的显示器上当然是以二维方式排列的，但它从左到右，从上到下，对吧？所以它是一个一维的文本序列。

[原文] [Andrej Karpathy]: now this being computers of course there's an underlying representation here so if I do what's called utf8 uh encode this text then I can get the raw bits that correspond to this text in the computer and that's what uh that looks like this so it turns out that for example this very first bar here is the first uh eight bits as an example so what is this thing right this is um representation that we are looking for uh in in a certain sense we have exactly two possible symbols zero and one and we have a very long sequence of it right now as it turns out um this sequence length is actually going to be very finite and precious resource uh in our neural network and we actually don't want extremely long sequences of just two symbols instead what we want is we want to trade off uh this um symbol size uh of this vocabulary as we call it and the resulting sequence length so we don't want just two symbols and extremely long sequences we're going to want more symbols and shorter sequences

[译文] [Andrej Karpathy]: 既然是计算机，当然这里有一个底层的表示。如果我对此文本进行所谓的UTF-8编码，我就能得到计算机中对应于此文本的原始比特（bits）。这就是它看起来的样子。实际上，例如这里的第一个条形就是前8个比特。这是什么东西呢？在某种意义上，这就是我们在寻找的表示，我们确切地有两个可能的符号：0和1，并且我们现在有一个非常长的序列。事实证明，这个序列长度在我们的神经网络中实际上是非常有限且宝贵的资源，我们实际上不想要只有两个符号的极长序列。相反，我们想要做的是权衡这个符号集大小（我们称之为词汇表vocabulary）和结果序列的长度。所以我们不想要只有两个符号和极长的序列，我们会想要更多的符号和更短的序列。

[原文] [Andrej Karpathy]: okay so one naive way of compressing or decreasing the length of our sequence here is to basically uh consider some group of consecutive bits for example eight bits and group them into a single what's called bite so because uh these bits are either on or off if we take a group of eight of them there turns out to be only 256 possible combinations of how these bits could be on or off and so therefore we can re repesent this sequence into a sequence of bytes instead so this sequence of bytes will be eight times shorter but now we have 256 possible symbols so every number here goes from 0 to 255 now I really encourage you to think of these not as numbers but as unique IDs or like unique symbols so maybe it's a bit more maybe it's better to actually think of these to replace every one of these with a unique Emoji you'd get something like this so um we basically have a sequence of emojis and there's 256 possible emojis you can think of it that way

[译文] [Andrej Karpathy]: 好的，一种压缩或减少我们序列长度的朴素方法基本上是考虑一组连续的比特，例如8个比特，并将它们组合成一个所谓的字节（byte）。因为这些比特要么开要么关，如果我们取一组8个，实际上只有256种可能的组合。因此，我们可以将这个序列重新表示为一个字节序列。这个字节序列将缩短8倍，但现在我们有256个可能的符号。所以这里的每个数字都在0到255之间。我真的鼓励你们不要把这些看作数字，而是看作唯一的ID或唯一的符号。也许这有点……也许更好的思考方式是把每一个都替换成一个独特的表情符号（Emoji），你会得到像这样的东西。所以我们基本上有一个表情符号序列，有256种可能的表情符号，你可以这样想。

[原文] [Andrej Karpathy]: now it turns out that in production for state-of-the-art language models uh you actually want to go even Beyond this you want to continue to shrink the length of the sequence uh because again it is a precious resource in return for more symbols in your vocabulary and the way this is done is done by running what's called The Bite pair encoding algorithm and the way this works is we're basically looking for consecutive bytes or symbols that are very common so for example turns out that the sequence 116 followed by 32 is quite common and occurs very frequently so what we're going to do is we're going to group uh this um pair into a new symbol so we're going to Mint a symbol with an ID 256 and we're going to rewrite every single uh pair 11632 with this new symbol and then can we can iterate this algorithm as many times as we wish and each time when we mint a new symbol we're decreasing the length and we're increasing the symbol size

[译文] [Andrej Karpathy]: 事实证明，在生产环境中最先进的语言模型中，你实际上想要超越这一步。你想要继续缩短序列的长度，因为再说一次，这是一个宝贵的资源，作为交换，你可以增加词汇表中的符号数量。实现这一点的方法是运行一种称为“字节对编码”（Byte Pair Encoding, BPE）的算法。其工作原理基本上是我们寻找非常常见的连续字节或符号。例如，事实证明序列116后面跟着32是非常常见的，出现频率很高。所以我们要做的就是将这一对组合成一个新的符号。我们要铸造一个ID为256的新符号，并将每一个116、32对重写为这个新符号。然后我们可以随心所欲地多次迭代这个算法，每次我们铸造一个新符号时，我们都在减少序列长度并增加符号集的大小。

[原文] [Andrej Karpathy]: and in practice it turns out that a pretty good setting of um the basically the vocabulary size turns out to be about 100,000 possible symbols so in particular GPT 4 uses 100, 277 symbols um and this process of converting from raw text into these symbols or as we call them tokens is the process called tokenization so let's now take a look at how gp4 performs tokenization conting from text to tokens and from tokens back to text and what this actually looks like so one website I like to use to explore these token representations is called tick tokenizer and so come here to the drop down and select CL 100 a base which is the gp4 base model tokenizer and here on the left you can put in text and it shows you the tokenization of that text

[译文] [Andrej Karpathy]: 在实践中，基本上词汇表大小的一个相当好的设定大约是100,000个可能的符号。具体来说，GPT-4使用了100,277个符号。这个从原始文本转换为这些符号——或者我们称之为“Token”（词元）——的过程就是所谓的分词（Tokenization）。现在让我们来看看GPT-4是如何执行分词的，即从文本转换到Token，以及从Token转换回文本，看看这实际上是什么样子的。我喜欢用来探索这些Token表示的一个网站叫Tiktokenizer。到这里的下拉菜单选择cl100k_base，这是GPT-4的基础模型分词器。在这里左边你可以输入文本，它会显示该文本的分词结果。

[原文] [Andrej Karpathy]: so for example heo space world so hello world turns out to be exactly two Tokens The Token hello which is the token with ID 15339 and the token space world that is the token 1 1917 so um hello space world now if I was to join these two for example I'm going to get again two tokens but it's the token H followed by the token L world without the H um if I put in two Spa two spaces here between hello and world it's again a different uh tokenization there's a new token 220 here okay so you can play with this and see what happens here also keep in mind this is not uh this is case sensitive so if this is a capital H it is something else or if it's uh hello world then actually this ends up being three tokens instead of just two tokens yeah so you can play with this and get an sort of like an intuitive sense of uh what these tokens work like

[译文] [Andrej Karpathy]: 举个例子，“hello space world”。结果“hello world”正好是两个Token：Token "hello"（ID为15339）和Token " world"（注意前面有空格，ID为1917）。所以是"hello" + " world"。如果我把这两个连起来（去掉空格），我还是会得到两个Token，但是是Token "H" 加上 Token "elloworld"（此处为示例解释，实际分词可能不同，意指分词方式改变）。如果我在hello和world之间输入两个空格，这又是一个不同的分词，这里会出现一个新的Token 220。好的，你可以玩一下这个，看看会发生什么。另外要记住这是区分大小写的，所以如果是大写H，那就是别的东西；如果是"hello world"（某种特定拼写或组合），实际上最终可能是三个Token而不是两个。是的，你可以玩一下这个，对这些Token是如何工作的获得一种直观的感觉。

[原文] [Andrej Karpathy]: we're actually going to loop around to tokenization a bit later in the video for now I just wanted to show you the website and I wanted to uh show you that this text basically at the end of the day so for example if I take one line here this is what GT4 will see it as so this text will be a sequence of length 62 this is the sequence here and this is how the chunks of text correspond to these symbols and again there's 100, 27777 possible symbols and we now have one-dimensional sequences of those symbols so um yeah we're going to come back to tokenization but that's uh for now where we are okay so what I've done now is I've taken this uh sequence of text that we have here in the data set and I have re-represented it using our tokenizer into a sequence of tokens and this is what that looks like now so for example when we go back to the Fine web data set they mentioned that not only is this 44 terab of dis space but this is about a 15 trillion token sequence of um in this data set and so here these are just some of the first uh one or two or three or a few thousand here I think uh tokens of this data set but there's 15 trillion here uh to keep in mind and again keep in mind one more time that all of these represent little text chunks they're all just like atoms of these sequences and the numbers here don't make any sense they're just uh they're just unique IDs

[译文] [Andrej Karpathy]: 我们实际上会在视频后面再回头讲分词，现在我只想给你们展示这个网站，并想向你们展示这段文本归根结底是什么样的。例如如果我取这一行，这就是GPT-4所看到的。这段文本将是一个长度为62的序列，这就是那个序列，这就展示了文本块是如何对应这些符号的。再说一次，有100,277个可能的符号，我们现在有了这些符号的一维序列。所以，是的，我们会回到分词这个话题，但这目前就是我们的进度。好的，所以我现在做的是，我取了我们在数据集中拥有的这个文本序列，我用我们的分词器将其重新表示为一个Token序列，这就是它现在的样子。例如，当我们回到FineWeb数据集时，他们提到这不仅是44TB的磁盘空间，而且在这个数据集中大约有15万亿（15 trillion）个Token序列。这里只是这个数据集前几千个Token的展示，但要记住这里有15万亿个。再一次记住，所有这些都代表小的文本块，它们就像是这些序列的原子，这里的数字本身没有任何意义，它们只是唯一的ID。