Things every AI Engineer should be using in agents..

章节 1：AI 工程师面临的成本与效率挑战

📝 本节摘要：

本节指出了当前 AI 工程师面临的核心痛点：如何在用户与大模型（LLM）交互的过程中降低对话成本。讲者提到了大模型（如 Claude Opus）昂贵的 Token 费用，以及“中间迷失（Lost in the middle）”现象导致的性能问题。核心观点是，尽管有各种限制手段，工程师仍需处理高达 50-70% 的 Token 浪费问题。

[原文] [Speaker]: the real question for AI engineers nowadays is how do you bring the cost for a single conversation from this to this

[译文] [Speaker]: 如今对于 AI 工程师来说，真正的问题是如何将单次对话的成本从这个数降到这个数。

[原文] [Speaker]: Given that your product is an agent that's sitting between the user and some kind of a open API or a cloud API that's hosting the LLM you're going to be paying for each prompt that the user makes

[译文] [Speaker]: 鉴于你的产品是一个介于用户和某种托管 LLM（大语言模型）的 Open API 或云 API 之间的智能体（Agent），你将不得不为用户的每一个提示词（Prompt）付费。

[原文] [Speaker]: And obviously you want to reduce that payment because depending on what kind of an LM you're going to be using Obviously stronger LM have bigger contact sizes and they also cost more

[译文] [Speaker]: 显然你想减少这笔支出，因为这取决于你使用的是哪种 LLM，很明显，更强大的 LLM 拥有更大的上下文窗口（Context Sizes），成本也更高。

[原文] [Speaker]: For example clot oppus is going to cost this much for 1K tokens and as tokens are our currency of LLM This is way too much

[译文] [Speaker]: 例如 Claude Opus（原文误识别为 clot oppus）每 1000 个 Token 要花这么多钱，而 Token 是我们在 LLM 中的货币，这实在是太多了。

[原文] [Speaker]: And not to forget we're also have to pay for the output

[译文] [Speaker]: 别忘了我们还得为输出内容付费。

[原文] [Speaker]: Not to mention the fact that bigger context sizes are not even always ideal We have a problem called lost in the middle where the LLM actually disregards the user prompt if it gets too big and especially the middle section

[译文] [Speaker]: 更不用说更大的上下文窗口甚至并不总是理想的，我们面临一个被称为“中间迷失（Lost in the middle）”的问题，即如果内容太长，LLM 实际上会忽略用户的提示词，特别是位于中间部分的内容。

[原文] [Speaker]: So despite all the products that we have in our hand with guard railing rate limiting and batching engineers are the ones who have to deal with this waste Meaning 50 to 70% of our tokens are actually wasted and this has to be optimized

[译文] [Speaker]: 因此，尽管我们手头有各种具备护栏（Guard railing）、速率限制和批处理功能的产品，工程师们仍是必须处理这种浪费的人，这意味着我们 50% 到 70% 的 Token 实际上被浪费了，而这必须得到优化。