模型切换工作坊：GPT-5.5、Kimi、DeepSeek 与 Mistral 如何在应用中比较

简要说明Brief Description

Pamela Fox 从计数、事实依据、视觉能力、工具使用和智能体五个方面比较了四种模型，随后演示了如何通过评估、ASSERT 和 DSPy 来测试并改进模型在特定应用中的行为。Pamela Fox compares four models on counting, groundedness, vision, tool use, and agents, then demonstrates evaluations, ASSERT, and DSPy as ways to test and improve model behavior in a specific application.

目录Table of Contents

• 研讨会设置
  Workshop setup
• 为什么你需要能够切换模型
  Why you need to be able to swap models
• 这四个模型
  The four models
• 基准数据与你的场景对比
  Benchmarks vs your scenario
• 测试 1：字母计数
  Test 1: counting letters
• 通过提示工程修复它
  Fixing it with prompt engineering
• 测试 2：RAG 与依据充分性
  Test 2: RAG and groundedness
• 测试 3：图像输入
  Test 3: image input
• 切换语言
  Switching languages
• 测试 4：工具调用
  Test 4: tool calling
• 测试 5：循环中的工具调用
  Test 5: tool calling in a loop
• 使用 LangChain v1 构建智能体
  Agents with LangChain v1
• 并行工具调用
  Parallel tool calls
• 使用字符串检查的评测
  Evals with string checks
• 问答：评估框架
  Q&A: eval frameworks
• 将 LLM 用作评判者
  LLM as a judge
• 断言：来自规范的评估
  ASSERT: evals from a spec
• 跨模型的 ASSERT 结果
  ASSERT results across models
• 使用 DSPy 优化提示词
  Prompt optimization with DSPy
• 用于 Mistral 的 DSPy 提示词
  A DSPy prompt for Mistral
• Anthropic 与引用
  Anthropic and citations
• 总结
  Wrap-up

工作坊设置Workshop setup

好，现在已经差不多 11:12 了，我想开始聊聊这个 workshop。我会以带领式 workshop 的形式进行，会带大家过一遍屏幕上展示的所有示例，你们也可以跟着一起尝试。仓库已经准备好了，环境变量也都设置好了。环境变量可以让你访问我在 Microsoft Foundry 项目中部署的一系列 Foundry 模型，它们的 token 容量有限。我们在这次课程中有可能会用超出 token 容量的量。All right. since it's already like 11:12, I want to start talking about the workshop. so, I'm going to do this as a guided workshop, so I'll go through all the examples up here on screen, but you can also try and follow along. So, the repo is there, the environment variables are there. so, the environment variables give you access to a bunch of Foundry models that I've deployed on Microsoft Foundry project. and they have a certain amount of token capacity. It is possible that we are going to go over the token capacity in you know, in this session.

我也是最近才知道这个工作坊被接受了，所以只能在很短的时间内尽可能多地收集资源。因此，在那个 .env 文件中，实际上有多个不同的环境变量可以选择注释掉或取消注释。I only got found out very recently that this workshop was accepted, so I tried to get as much resources as I could in the small period of time. so, in that .env file, there is actually multiple different possible environment variables that you can comment in or comment out.

如果我们真的开始遇到 429 错误，也就是超出速率限制的错误，那就……对。我希望容量足够，让你们所有人都能试用这些模型，咱们到时候再看。我同事 Cedric 也在这里。我讲话的时候，Cedric 会四处走动，所以他也许能帮忙处理问题。关于代码设置，如果你的机器上已经安装了 Python 和 UV，那么可以 git clone 这个仓库，然后运行 uv sync；有了 .env 文件后，就应该具备运行示例所需的一切。否则，你也可以创建一个 GitHub Codespace。If we do start getting 429s, you know, rate limit exceeded errors. so, yeah. So, I'm hoping that there's enough capacity for all of you to be able to try out those models, and we will see. I also have my colleague Cedric here. so, Cedric is going to be wandering around while I'm talking, so he might be able to help with issues as well. if you In terms of setting up the code, if you already have Python and UV on your machine, then you can do a git clone of the repo and run uv sync, and that should have everything you need to run the examples once you have that.env. otherwise, you can make a GitHub code space.

基本上，这就是一个在浏览器里运行、所有东西都已为你配置好的 VS Code。这里有人用 UV 吗？Cedric 想知道。好吧，它目前还不是第一名。That's basically a VS code in the browser that has everything set up for you. who here uses UV? Cedric wants to know. Okay. So, it's not number one yet.

不过，这里还有一个问题：谁使用 Python？好吧，那人数更多。好的。But, the other question here, who uses Python? All right, then more people. Okay.

Cedric：这里几乎所有人都用 Python。Cedric: Pretty much everyone uses Python here.

Pamela Fox：是的。哦，你算过重叠部分了吗？好吧。好的。Pamela Fox: Yeah. Oh, did you count the overlap? All right. Okay.

好，如果你还没给那个屏幕拍照，现在就拍一张吧。否则，Cedric 手机里也有一张，所以如果你需要那些链接，他可以到处帮你找。All right, so if you didn't take a picture of that screen yet, just take a picture of it now. Otherwise, Cedric also has a picture of it on his phone, so he can go around if you need those links.

为什么你需要能够切换模型Why you need to be able to swap models

所以，这次工作坊的目标是让大家感受一下不同顶尖模型之间的差异，看看它们有什么不同，以及可以在哪些方面加以改进。我准备了许多不同的示例，以及我们可以进行的评估和优化，帮助我们提升在不同模型上的使用体验。我认为，对我们所有人来说，作为 AI 工程师，做这件事都非常重要，因为你们也看到了，新模型一直在不断推出。而且，美国政府随时可能切断对某个模型的访问。So the goal of this workshop is to be able to get a feel for the differences between the different, you know, top-of-the-line models and see how they differ and, you know, what you can improve about them. So, I've set up a bunch of different examples and also evaluations and optimizations that we can do to improve our experience across models. and I think this is a really important thing for all of us to just be to be doing as AI engineers because, you know, as you've seen, like, there's so many new models that come out all the time. And, you know, the US government at any point could cut off access to a model.

所以，我们需要有选择，对吧？我们得能够让 AI 应用使用不同的模型，而且它仍然能正常运行，或者知道该如何改进它。So, we need to like, we need to have options, right? So, we need to be able to, you know, point our AI applications at a different model and have it still work or know how to improve it.

这四种模型The four models

所以这里的目标是多体验一些模型。这些是我在 Microsoft Foundry 中部署的模型。我有 GPT-5.5，这是我一直在用的标准模型。不过，我还有 Kimi K2.6，这是一个较新的模型，表现相当不错。我还部署了 DeepSeek V4 Flash。So that's the goal here is to get more experience with more models. So, these are the models I've deployed inside Microsoft Foundry. so I've got GPT-5.5, that's a standard one I use all the time. But, then I've got Kimi K2.6, that's a more recent one that's quite good. I've got DeepSeek V4 Flash.

然后我还有 Mistral Large 3。这就是我们将在本次工作坊中进行比较的四个模型。正如我刚才所说，要设置工作坊，可以使用这个大家都在用的仓库。如果你的电脑上没有安装 Python，只需点击“Code”按钮，这里会有一个很大的“Create codespace”按钮。点击它，就会创建一个代码空间。And then I have Mistral Large 3. So, these are the four models that we'll be comparing in this workshop. so as I was saying, you can in order to set up the workshop, here's the repo that you're all into. So, if you don't have Python on your machine, you can just click on the code button and there'll be a big button here that says create code space. So, you click on that, it'll make a code space.

那么，这里就是代码空间，我已经把 .env 放在这里了。接着你去拿我放在上面的另一个 URL 来获取环境变量。So, then here's the code space and then this has I've already put the dot env, but then you go and you grab that other URL that I put up there to get the env.

然后转到这个文件。你把它下载下来，打开它，然后根据里面的内容创建一个 .env 文件，好吗？That goes to this file here. You download it. You open that. And then you create a dot env with the contents, okay?

所以，这里指向的是一个部署了这四个模型的 Foundry 项目。正如你所看到的，它有令牌限制。所以，如果我们开始遇到速率限制，你可以试试这些其他模型，它们也部署在另外四个项目中。然后我还有一个专门用于 Anthropic 的独立项目，因为该项目的条款有所不同。So, this is pointing at a that foundry project that has those four models deployed. As you've seen, that does have you know, it has token limits. So, if we start getting rate limits, you can start you can try these other models that also have the four project deploys. And then I have a separate one specifically for Anthropic just because it has different terms for that project.

好。那么，是的，等你把这些设置好之后，uv sync 就会同步 Python 所需的所有包。好了，设置部分就到这里。现在，我会实际演示一遍，在这里把它做、做出来，这样你想的话就可以跟着操作，或者也可以只看、看我怎么做。All right. So, so yeah, and then once you've got that set up, then uv sync will sync all of the packages that you need for Python. All right, so that's the setup. Now, I'm going to actually go through it and do it do it up here so that you can you know, hopefully follow along if you want, or you can just watch watch what I do.

所以，就像我刚才说的，我们确实有这四个模型，你可以通过跑基准测试来比较它们。你会想，哦，你知道，这个模型在质量基准测试和安全性基准测试中的表现都不错。你还可以看看成本、吞吐量之类的。So, as I was saying, we do have the four models, and you can compare models just doing benchmarks. You're like, oh, you know, this one has good quality benchmarks and good safety benchmarks. You can also look at you know, the cost and throughput and stuff like that.

基准测试与您的场景对比Benchmarks vs your scenario

这样你就能大致了解这些模型的表现，但只有在你的具体场景中使用它，才能真正知道一个模型有多好，因为使用模型的方式有很多种。某个特定的基准测试并不能真正告诉你它在你的场景中会表现如何。所以，我们在这个仓库里准备了大量不同的场景，可以针对每种场景试用这些模型。And you can get this approximate notion for how good these models are, but you don't really know how good a model is until you use it for your particular scenario because there's so many different ways of using models. And a particular benchmark isn't really going to tell you how it's going to work for your scenario. So, that's what we have in this repo here is a ton of different scenarios, and we can try out these models for each of them.

测试 1：字母计数Test 1: counting letters

所以，我们要从非常非常基础的内容开始，好吗？就是进行一次单独的 LLM 调用，让 LLM 做某件事。你会看到，这些文件都是从 model 变量开始的，而且默认使用 GPT-5.5。这算是我们的基准。So, we're going to start really really basic, all right? So, just a single LLM call that asks the LLM to do something. So, what you see for each of these files is that they all start off with the model variable and they default to GPT-5.5. That's kind of our baseline.

显然，这是一个非常好的模型。但如果我们想尝试不同的模型，只需取消该行的注释，之后使用的就是这个模型。这里的文件通过 OpenAI Python SDK 连接到 Foundry URL，并传入环境变量中的密钥。我们在这里使用的提示是：以下句子中出现了多少次字母 E？It's obviously it's a very good model. But then if we want to try a different model, we can just uncomment that line and that will be the model that gets used. So, this file here is connecting using the OpenAI Python SDK to the Foundry URL and passing in the key from the environment. And the prompt we're doing here is how many times does the letter E appear in the following sentence?

那位老先生兴冲冲地走进了电梯。仔细数数。好吧。那么，我们把它发送到 OpenAI responses API，并说：“来，用这个模型和这个提示词，然后我们看看原始输出，对吧？”The elderly gentleman eagerly entered the elevator. Count carefully. All right. So, then we send that to the OpenAI responses API and say, "Here, use this model with this prompt and then let's see the raw the output, right?"

所以，我们要做的是让 LLM 数数这件经典的事。接下来看看它们表现如何。我会输入 `uv run examples`，然后加上这里这个文件的名称。这些内容都在 README 里。So, we're doing the classic thing of asking LLMs to count. And so, we'll see, you know, how well they're able to do. So, I'm going to say uv run examples with the name of this file name here. And this is all in, you know, in the read me.

所以，我们就一个一个地看。你可以直接从那里复制粘贴。好。那么，GPT-5.5 实际上很擅长数学，数学能力好得令人惊讶。So, we're just going through one by one. So, you can just copy-paste from there. All right. So, GPT-5.5 is actually very good at math, surprisingly good at math.

我最近的同事一直在它上面跑很多金融基准测试，他们会说：“天哪，它居然能在脑子里做这么多计算。”所以，它确实正确数出了这句话里有 13 个字母 E，对吧？那现在我们来试试。我们只要改一下，对吧？好，假设我们想试试 Kimi。那我们就切换到 Kimi，现在使用的模型就是 Kimi。My colleagues have been running a lot of finance benchmarks on it lately and they're like, "Oh my god, like it does so much math in its head." So, it does actually get this correct that there are 13 E's in this sentence, right? So, now let's try Now we just change it, right? So, okay, let's say we want to try Kimi. So, we go and we change to Kimi and now Kimi will be the model that's used.

然后我们看到：“好吧，Kimi 能像 GPT-5.5 一样数数吗？”它确实能，数到了 13。好吧，你现在可能会想：“哦，这是不是太简单了？”And we see, "Okay, can Kimi count as well as GPT-5.5?" And it does. It gets 13. All right. Now you might be thinking, "Oh, is this just too easy?

我们试试 DeepSeek V4 Flash。这次它得了六分。好吧。所以，它漏掉了七个字母 E，对吧？Let's try DeepSeek V4 Flash. And this time it gets six. Okay. So, it missed out on seven of the E's, right?

因此，我们可以立刻看出，DeepSeek V4 和前两个模型之间确实存在差异。最后，我们来试试 Mistral。这个得了八分。所以，我们看到这里有差异。好了。So, we can immediately see that there's definitely difference between you know, DeepSeek V4 and those first two models. And then finally, we'll try Mistral. And this one gets eight. so, what we see there's a difference here. All right.

通过提示工程修复它Fixing it with prompt engineering

所以，当你看到这个时，你会想：“嗯，我想能够使用所有这些模型。”然后你就开始做一些提示工程。你会想：“好吧，如果我说，你知道，把句子拆分成单词，按单词计数，求和，并展示你的推理过程，怎么样？”我们来试试，我们来试试，好吗？你知道，我们还可以修正一下拼写错误，虽然这也不重要。好吧。So, when you see this and you're like, "Well, I want to be able to use all these models." Then you start doing some prompt engineering. You're like, "Okay, what if I say, you know, break the sentence into words, count per word, sum, show your reasoning, right?" Let's try Let's try that, right? And you know, we can fix our typos, not that it matters. All right.

那么，我们用 Mistral 试试看，稍微做一点提示词工程，看看能不能得到更好的回答。好，它答对了，对吧？你看，把问题拆开，再做一点思维链推理，它就能得出正确答案了，对吧？So, let's try it with Mistral and see with a little bit of prompt engineering, can we get a better response here? All right. And it got it, right? So, you know, by breaking it up, doing a little chain of thought here, it was able to come up with the correct answer, right?

因此，我们马上就能看出，不同模型之间显然存在差异；如果我们更改提示词，就能得到不同的输出。好。So, immediately, we can see that there's obviously differences across models and if we change prompts, you know, we can get different outputs. All right.

测试 2：RAG 与基于事实依据的回答能力Test 2: RAG and groundedness

那么，现在让我们进一步深入，看看更多贴近现实的场景。我们来看看 RAG。RAG 仍然是一个重要的用途。可以说，几乎所有东西都是 RAG。智能体也属于 RAG。So, now let's get more interesting and go to more real-world scenarios. so, let's look at RAG. So, RAG is still one of the big uses. Arguably, really everything is RAG. Agents are RAG, too.

所以，我们有这个……所以，RAG就是让LLM根据一些上下文来回答问题，对吧？而RAG的关键在于，你希望它回答问题时始终以这些上下文为依据。你不希望它编造任何只来自模型权重的信息，对吧？你是想强制它紧扣上下文，对吧？So, we have this So, RAG is when you ask an LLM to answer a question based off some context, right? And the thing that's important for RAG is that you want it to answer the question and stay grounded in the context. You don't want it to make up anything that is just from its weights, right? You're trying to force it stick to the context, right?

我发现，不同模型之间真正的差异就在于：较小、能力较弱的模型更有可能去查自己的权重，并假装那是上下文的一部分。所以，我们这里有一个关于木蜂的数据集，然后向它提问：“蜜蜂和木蜂筑巢的方式有哪些关键区别？”And that's what I find really varies across the models is that the smaller models, the less powerful models are actually much more likely to go to their weights and pretend it was part of the context. so, what we have here is we have a data set about carpenter bees and we ask it a question, "What are the key differences between how honey bees and carpenter bees build their nests?"

但这个数据集里只有关于木蜂的信息。所以，它实际上应该只能回答有关木蜂的问题。因此，这种情况下，一个不错的回答应该是告诉我木蜂的相关信息，然后说：“抱歉，我无法告诉你关于蜜蜂的信息。我不知道。”所以我们要这样告诉它：“嘿，你的任务是根据资料来回答，并引用资料。这是你的这是你的问题，这是你的资料。”好吧，那我们就来试试看，看看会得到什么。好吧，它……是的，所以它先进行搜索时，做的只是非常基础的搜索。But this data set only has carpenter bee information. so, it should only actually be able to answer about carpenter bees. So a good response in this case would tell me the carpenter bee information and then it say, "Sorry, I can't tell you about honey bees. I don't know." so we tell it like, "Hey, your job is to answer based on the sources, cite the sources. here's your here's your question, here's your sources." All right, so let's go ahead and try that out and see what we get. All right, and it's Yeah, so when it's doing the search first, very basic search.

好吧，所以它表示已经进行了搜索。好，那么我们看到它说数据集没有提供有关蜜蜂筑巢的详细信息。这正是我们想看到的。这是来自 GPT-5.5。Okay, so it's sending did the search. All right, and what we see and it says the data set does not provide details on honey bee nest construction. So this is what we actually want to see. This is from GPT-5.5.

GPT-5.5 很清楚自己不知道什么。GPT-5.5 is very good at knowing what it doesn't know.

现在我们也试着切换到 Kimi，效果也不错。为了节省时间，我就不每次都展示所有模型了。我们来试试 DeepSeek 吧？因为我们已经看到这里确实存在区别。DeepSeek 再次表示，提供的来源中不包含相关信息。So now, let's try switching Kimi's also good as well. So I'm not going to show every model every time now just for sake of time. so let's try DeepSeek, right? Cuz we've already seen that, you know, there's a distinction here. Now DeepSeek, once again, says provided sources do not contain information.

好，DeepSeek 这次做得不错。现在我们来看看 Mistral。抱歉，是 Mistral。Cedric，Mistral 怎么念？All right, so DeepSeek did a good job here. And now let's go to Mistral. Mistral, sorry. Cedric, how do we say Mistral?

这是法国的，对吧？法国公司？It's French, right? French company?

塞德里克：Mistral。Cedric: Mistral.

帕梅拉·福克斯：Mistral。塞德里克会做我的法语翻译。每次我需要说的时候，你就直接替我说。现在，Mistral。Pamela Fox: Mistral. Cedric's going to be my French translator. Every time I need to say it, you're just going to say it. Now, Mistral.

Mhm? Yeah, okay. Mistral. so Mis- MisWith Mistral

我们确实看到，Mistral 实际上是根据自己的权重给出了回答，尽管这些信息并不在资料来源中。所以这是一个模型调用了自身权重中的知识、而我们并不希望它这么做的例子，对吧？现在我们可以试着强制它遵守要求。再一次，我们可以尝试进行提示工程。我们可以说，等等，实际上，它会这样说：如果某条信息的答案不在资料来源中……we do see that Mistral actually answered from its weights even though that information was not in the sources. So this is an example where a model went into its weights even though we didn't want it to do, right? Now we can try to force it. Once again, we can try to do prompt engineering. we can say wait, actually so it would say if the if the answer for a bit of info is not in sources.

说没有提供源代码。我想这就是我让它运行起来的方式。哦，我会试试那个，别编造答案。好吧，我们试试这个。Say no source provided. I think this is what I got it to work. oh, I will try that and do not make up an answer. All right, let's try this.

所以，我们正试着让它意识到，什么时候它给出的信息并不在模型权重里。有时候，如果你强迫它说：“嘿，你在答案里给出的每一条信息，都得证明你是从来源中获取的，对吧？”那么，我这里就是试着让它把答案里的每一项内容都和一个来源关联起来，也许这样就能让它意识到：“哦，等一下，那其实并不在我的答案里。”好吧，它还在继续输出。它现在会在这件事上想得特别认真。So, we're trying to force it to realize when it's giving us information that's not in the weights. And sometimes if you force it to say, "Hey, for every bit of information that you're giving back in your answer, you have to prove that you got that from the sources, right?" so, here I'm trying to have it associate each of the things in that in that answer with a source and maybe that will force it to realize, "Oh, wait a second, that wasn't actually, you know, in my in my answers here." All right, it's still going out. It's going to think it's thinking really hard about it here.

好。就这样。关于蜜蜂的细节，除了与木蜂的互动之外，没有提供其他来源。Okay. And there we go. Honeybee details are no source provided beyond interactions with carpenter bees.

所以，它确实知道哪些内容并非来源中提供的，但还是给出了相关信息。我们又更接近了一步。不过你也看到了，这是我在 RAG 中观察到的一个明显差异，而 RAG 是我最常接触的领域。因此，在评估要用于 RAG 的模型时，你需要考察它们是否愿意承认自己不知道什么，以及它们有多大可能会编造内容。so, it definitely knew when things were not source provided. It still gave the information. So, we got a little closer. But as you can see, this is definitely a difference that I see with rag and rag is what I work with the most. so, it's something you want to evaluate when you're looking at models to use with rag is are the models willing to admit what they don't know, how likely are they to make stuff up?

测试 3：图像输入Test 3: image input

好，现在我们来看看图像输入。很多这类模型都支持多模态，因此可以接收图像作为输入。你知道，图像输入能应用在许多非常有趣的场景中。这里我们在本地有一些文件，接下来就把它们发送给大型语言模型，并询问一些关于这些文件的问题，对吧？All right, so now let's move to image input. A lot of these models are multimodal, so they will accept images as input. And you know, there's lots of really cool situations you can do with image input. So, here we've got some files locally and we're just going to send them to the LLMs and ask them some questions about the files, right?

我给你看看这些文件。示例、图像输入。好。行。I'll show you the files. Examples, image input. Okay. All right.

所以，我们这里有的文件就是这张图片。这是一头原牛，不是独角兽。这张图片。画的是我在哥斯达黎加看到的一些鳄鱼，我当时非常害怕它们。So, the files we have here are this picture. This is an aurochs, not a unicorn. This picture. This is of some crocodiles that I saw in Costa Rica that I was very afraid of.

然后我们还有这张图片，里面列出了一些植物。基本上就是一张表格的截图，字体特别小。所以，我们让每个 LLM 分别解读每张图片，对吧？首先我们会问：“哦，这是一只独角兽吗？”And then also we have this picture which is a list of plants. Basically like a screenshot of a table with some really small fonts in it. So, we're asking each of the LLMs to interpret each of the images, right? So, first we say, "Oh, is this a unicorn?"

对吧？这应该很容易判断它不是独角兽。很好。然后我们说：“嘿，这些是短吻鳄还是鳄鱼？”这就有点棘手了。Right? That should be easy to say it's not a unicorn. Great. Then we say, "Hey, are these alligators or crocodiles?" That's a bit tricky.

它们其实是鳄鱼，因为它们的吻部就是这个样子。我花了很多时间阅读维基百科上关于鳄鱼的页面，你只要看看它们的吻部就知道了。幸运的是，GPT-5.5 也沉迷于刷维基百科，所以它知道看到那种吻部时，眼前的就是鳄鱼。至于最便宜的植物，它也正确地从那张文字相当小的表格截图中找到了最便宜的植物。They are actually crocodiles because of their snout. I spent a lot of time reading the Wikipedia page about crocodiles and you got to look at their snout and then you know. Luckily, GPT-5.5 has also been on a Wikipedia binge and so it knows that when it sees that snout, then that is crocodiles. And then the cheapest plant, it correctly found the cheapest plant in that like fairly small text screenshot of a table.

所以，GPT-5.5 能做到这一点确实相当令人印象深刻。那么现在我们来试试……对于这些模型，我们不能用 DeepSeek，因为 DeepSeek 只接受文本。它不是多模态模型。所以我们只会尝试 Kimi 和 Mistral。So, that's pretty impressive that GPT-5.5 could do that. So, now let's try it with Now, for these models, we can't use DeepSeek cuz DeepSeek only accepts text. It's not a multimodal model. So, we're only going to try Kimi and Mistral.

所以，你知道，它识别出了原牛。So, you know, it figured out the aurochs.

好，没问题。只是确认一下它能正常运行。好的。至于鳄鱼，它确实正确地说出了“鳄鱼”。Okay, good. Just making sure it works. All right. And then for crocodiles, it does say crocodiles correctly.

然后是最便宜的植物，它得到了最便宜的植物。所以，Kimi 也做得非常出色。最后，我们来看看 Mistral。就在这里。And then cheapest plant, it got the cheapest plant. So, Kimi is doing a great job as well. And then finally, let's go to Mistral. Here.

我们来看看。好吧。所以，你知道，它知道自己知道自己不是独角兽。它不知道自己是一头原牛。Let's see. All right. So, you know, it knows it knows it's not a unicorn. It didn't know it was an aurochs.

这个，它不太确定。它觉得那些是短吻鳄。所以，你知道，它在看那个……时更费劲，因为它觉得那些是短吻鳄。至于植物，它挑出了一株售价 2.30 美元的植物，比最便宜的植物贵得多。This one, it wasn't sure. It thinks they're alligators. So, it you know, it's having a harder time looking at the it thinks that they're alligators. And then for the plants, it pulled out a plant that's $2.30, which is way more expensive than the cheapest plant.

它本来应该找到一个售价58美分的商品。所以，Mistral没法回答这类图像问题。问题是，在多模态理解方面，你其实没法做太多提示工程，因为模型要么能理解图像，要么就不能。It was supposed to find one for 58 cents. So, the Mistral isn't able to answer these kind of images. Now, here's the thing, when it comes to multimodal understanding, you can't really do much prompt engineering cuz either like the like the model either understands an image or it doesn't.

如果你知道自己处于某个非常特定的领域，也许可以做一些提示工程，给模型一些提示，告诉它该如何解读某种特定类型的图表。但实际上，对于图像输入，关键在于弄清楚：对于你准备发送给 LLM 的这类图像，LLM 能不能看懂？它能不能适用于这个场景？因为也许 Mistral 确实很擅长处理某些类型的图像任务，但就是不擅长处理这种任务，对吧？Like you might be able to do some prompt engineering if you know you're in a very particular domain and there's some hints that you can give to it about how to read a particular kind of graph. but really with image input, it's about figuring out, you know, for the kind of images that you're going to send to the LLM, does the LLM understand those images? Is it going to, you know, is it going to work for that scenario, right? because it may be that, you know, maybe Mistral is really good at particular kinds of image tasks, but just not this, right?

所以，关键在于弄清楚你想完成什么任务，以及模型能不能把它做好？So, it's all figuring out what is the task you want to do and is the model going to do a good job at it?

切换语言Switching languages

有意思的是，多模态模型在很多情况下也会受到语言的影响，对吧？我这里用的是英语。记住，这些模型都非常偏重英语；但如果你换一种语言，比如用西班牙语问它们，那些动物是 caimanes 还是 cocodrilos，它实际上会给出错误答案，因为它脑子里对这些不同的词，以及它们如何对应到英语中的词，感到非常困惑。要同时理解短吻鳄和鳄鱼之间的区别，又记住 caiman 在西班牙语里指的是短吻鳄，而不是佛罗里达州那种凯门鳄，对它来说实在太难了。Now, what's interesting with the multimodal, too, is that a lot of these also depend on the language, right? Here I'm using English. Remember, all these models are super English-centric, but if you switch to a different language, like if I ask in Spanish if they're caimanes or cocodrilos, it actually thinks it says it gives the wrong answer because it's getting so confused in its head between these different like words and how they map across English. Like it just becomes too hard of a task both understand the difference between alligators and crocodiles and also remember that caiman means alligator in Spanish and is not the same thing as the caiman that we have in like Florida.

总之，很有意思的是，一旦我们切换到西班牙语，它就无法完成任务了。所以这也是另一点需要牢记的，尤其是我们在加州。我们在这里开发应用程序时，很多时候都得支持双语，因为你知道，这里有 50% 的人说西班牙语。Anyway, it's like it's fascinating to see that once we switch to Spanish, it fails at the task. so, that's another thing to keep in mind, especially we're in California, a lot of times when we develop applications here, they are bilingual cuz a lot of, you know, 50% Spanish-speaking here.

所以，嗯，随着你切换语言，效果可能会有所不同。So, you know, your mileage may vary as you change languages.

测试 4：工具调用Test 4: tool calling

好，现在我们来讲函数调用，也叫工具调用。实际上，工具调用可以说是所有智能体的基础。所有智能体都建立在工具调用之上。这是大语言模型在训练过程中专门训练的一种能力。Okay, so now let's go on to function calling, also known as tool calling. So, tool calling is in fact the basis of like all agents. All agents are based off of tool calling. And this is a capability that the LLMs are trained with specifically during the training process.

所以，它们经过训练，能够查看一个函数定义或多个函数定义，再查看用户输入，然后说：“好的，根据这个用户输入，我想用这些参数调用这个函数，然后，你知道的，就直接返回结果。”在这里这个示例中，我使用 JSON schema 定义了一个函数。这是 LLM 能理解的形式。我会说：“好的，这是我的函数。它的名称是 create calendar event。”So, they are trained to be able to look at a definition of a function or multiple function definitions and look at the user input and say, "Okay, based off that user input, I want to call this function with these arguments and then, you know, just return that." So, in this example here, I'm defining a function using JSON schema. This is what the LLMs understand. So, I say, "Okay, here's my function. It has the name create calendar event.

它有这样的描述，这些是它接受的参数。它接受一个标题，标题是字符串。这里有个描述，对吧？It has this description and these are the parameters it takes. It takes a title with this that's a string. Here's a description, right?"

而且这些内容都会发送给 LLM。我们会把完整的 JSON Schema 发给 LLM，然后说：“嘿，这是这个函数。你得决定要不要调用它，以及怎么调用，对吧？”所以，实际上，尽管这看起来像是一个对象，但这里做的其实是提示工程。这里的一切，实际上都是我在决定什么对 LLM 来说最合适，对吧？And all of this is all going to get sent to the LLM. So, we send this full JSON schema to the LLM and say, "Hey, here's the function. You got to decide whether you're going to call this and how you're going to call it, right?" So, this is in fact, even though this is like an object, this is prompt engineering here. Everything here is actually like me deciding what is going to be the best for the LLM, right?

我应该使用什么类型？这里应该添加什么描述？这些参数应该叫什么名字？因为 LLM 会根据所有这些信息来决定如何调用这个函数。然后，我们有一条用户消息。What type should I use? What description should I add here? What should be the names of these parameters? Cuz the LLM's going to decide based on all that information how it's going to call this function. so, then we have a user message.

所以，就像是，用户会说：“哦，你能在我的日历上加个安排吗？是平台团队的同步会。我、来自 Ange 的 Sarah、Marcus，还有新任产品经理 Priya。”So, it's like So, the user's like, "Oh, can you throw something on my calendar? It's a platform team sync. Me, Sarah from Ange, Marcus, the new PM Priya.

“明天下午1:30。通过 Microsoft Teams 线上进行，可以吗？”然后，我们就告诉 LLM：“嘿，你是一个日历助手。今天是6月29日，星期一。使用可用工具来处理用户的请求。”接着，我们把这些工具发送给它。Tomorrow at 1:30 p.m. It's virtual on Microsoft Teams, okay?" So, then our then we tell the LLM like, "Hey, you're a calendar assistant. Today is Monday, June 29th. Use the available tools to process the user's request." And we send it the tools.

所以现在，LLM 正在查看用户请求和可用工具。在这个例子中，它只有一个创建日历事件的工具。它会判断：“嘿，这里有没有我能用来回答用户的工具？”接下来我们就看看它决定调用哪些工具。好了，就是这样。这里讲的是非常底层的工具调用。等我们构建智能体时，会在此基础上做抽象，但我想先从最底层的工具调用讲起。And so, now the LLM is looking at the user request, looking at the tools. And in this case, it just has that create calendar event. And it's going to decide, "Hey, is there a tool here I can use to answer the user?" and then we're just going to go through and see what tool calls it decides to make. so there we go. So in and this is like really low-level tool calling here. once we build an agent, we're going to abstract on top of this, but I wanted to start with the most low-level tool calling we can do.

然后，LM 只会回复函数名称以及它希望你传入的参数。And the LM just responds back with here's the name of the function and here's the arguments that I want you to send.

语言模型本身不会调用这个函数，这得由我们来做。好吧，我们可以看到，我把预期结果放在了左边，这样你就能看到；我们也能看到实际结果。GPT-5.5 确实把我加成了参会者，这其实不是我想要的。不过除此之外，它做得相当不错。The LM itself doesn't call the function. That's up to us to do. All right, so what we can see is that I put my I put the expected on the left just so you can see and we can see the actual. So GPT-5.5 did add me as an attendee, which I didn't really want. but it did pretty good otherwise.

那么我们来试试 DeepSeek，看看它在这个任务上的表现。因为我们想看看，在调用工具时，它有没有调用正确的工具？这次确实调用对了。它调用的是创建日历事件工具。So let's try DeepSeek and see how DeepSeek does with this one. Cuz what we want to see like when we're doing tool calling, we want to see is it calling the right tool? So it is in this case. It is calling the create calendar event tool.

而且它传入的参数是我们预期的吗？我们预期会有一些特定参数。在这个例子中，DeepSeek 把你设成了参会者。至于地点，它写的是线上（Microsoft Teams），这不是我要求的，对吧？And is it passing in the arguments that we expect, right? We have certain arguments we expect. In this case, DeepSeek put you as an attendee. And then for the location, it said virtual {parentheses} Microsoft Teams, which is not what I asked for, right?

所以它对我的论证约束遵循得没那么严格。So it's not adhering to my argument constraints as much.

我们也试试 Mistral，看看效果怎么样。看到这个之后，我就会开始意识到：“好吧，这里存在一些歧义，对吧？”你知道，如果它调用时用了错误的参数，那就说明存在歧义。我需要回头修改描述，说清楚，比如，不要把我或你算进去，只列出具名的参会者。如果提供了姓名，就写名和姓，对吧？Let's try Mistral as well. And see how it does. So once I see this, I can start to realize like, "Okay, there's some ambiguity, right?" And you know, if it if it called it with the wrong arguments, that means there's ambiguity. I need to go back into my descriptions and say, you know, you know, don't you know, don't include me or you, just named attendees. you know, you could say first and last if provided, right?

这样你就会开始发现，你知道，哪些地方还不够具体，并开始澄清其中的歧义，然后评估所有不同的输入，看看效果如何。好，这就是基本的工具调用。接下来，我们要做的是逐步构建智能体。So you start to see, you know, what you haven't been specific enough about and start to clarify that ambiguity and then, you know, evaluate all the different inputs and see how it's doing. All right. So that was a basic tool calling. The next thing we want to do, we're building up to agents here.

测试 5：循环中的工具调用Test 5: tool calling in a loop

接下来要做的是在循环中调用工具，对吧？因此，当我们有一个智能体时，它会在循环中调用工具，直到达成目标。我们将演示如何使用一个名为 calculate 的函数。这个 calculate 接收一个数学表达式并返回结果，对吧？我们还提供了对这些数学表达式进行求值的基本支持。The next thing is to do tool calling in a loop, right? So when we have an agent, it's calling tools in a loop until it reaches a goal. so, we're going to demonstrate using this just a function calculate. So, this calculate takes a mathematical expression and returns back the output, right? And we have, you know, basic support for evaluating these mathematical expressions.

所以，我们基本上是给它一道文字题，比如：“哦，这家商店正在打折。如果我买三件衬衫，再付这么多税，总价是多少？”这道题可以用几种不同的方法来算。可以多次调用计算器工具，每次输入一个小表达式，比如“先算这个，再算那个，然后算这个，最后算这个。”或者，如果语言模型足够聪明，也可以把所有这些步骤合并成一个长表达式，只调用一次工具。So, we're giving it basically a word problem where it's like, "Oh, this store is having a sale. If I buy three shirts and pay, you know, this much tax, what's my total?" And this can be done several different ways. It could be done with multiple tool calls to that calculator call with like one little expression at a time, like, "Do this, and then do this, and then do this, and then this." Or, if the LM is very clever, it could batch all that into one long expression and do a single tool call.

所以，我们这里测试的是效率，对吧？也就是说，很多时候，一旦让工具调用进入循环，模型之间的差异更多体现在效率上。它们能不能更快达成目标？能不能用更少的工具调用完成同样的目标，对吧？So, here we're testing efficiency, right? Where, you know, there's more like a lot of times once you have tool calling in a loop, the differences between the models is their efficiency. Are they able to get to the goal faster? Can they accomplish the same goal with less tool calls, right?

所以，这就是我们要说的。唯一的系统提示是不要在脑子里做数学运算。好。So, that's what we're going to say. the only system prompt is do not do math in your head. Okay.

Pamela Fox：那么，我们来看看。这个真的是给 GPT-5.5 准备的，因为它对自己的数学能力太自豪了，总是试图在脑子里把计算做完。就像是在说：“拜托。”好吧。GPT-5.5 确实用一个表达式就完成了整个计算。所以，它只调用了一次工具就得出了最终结果。Pamela Fox: So, let's see. that one's really for GPT-5.5 cuz it's so proud of its math abilities that it's always trying to do it in its head. Like, "Come on." All right. So, GPT-5.5 indeed does the entire calculation in one expression. So, it only required one tool call to get to the final result.

所以，这很好。这很好。你知道，这很快，能帮我们节省时间。So, that's good. That's good. That's, you know, fast. It saves us time.

所以，现在我们来试试 DeepSeek。我一直跳过 Kimi，唯一的原因是它实际上是个非常好的模型，所以在这些任务中的很多方面，它的表现基本上和 GPT-5.5 不相上下。好，现在我们可以看到 DeepSeek 正在处理，它在做……它做了好多，多少个来着？它做了大概四个。So, now let's try with DeepSeek. The only reason I skip Kimi all the time is cuz it's actually a very good model, and so it does pretty much on par with GPT-5.5 for much of these. Okay. So, now we can see with DeepSeek, it is going through and it's doing and it's doing like so many how many it did like four.

它调用了四次工具，对吧？所以，它最终还是得出了正确答案，但一共调用了四次工具。而每次工具调用都意味着又发起了一次网络请求，对吧？It did four tool calls, right? So, it still got to the correct answer, but it took four tool calls. And every one of those tool calls is another network request, right?

所以你知道，当我们评估两个模型时，我们既想看最终答案，也想看得出它花了多长时间才得到最终答案，对吧？其实，前几天 Kimi 确实做了件特别有意思的事。看看这次它会不会再来一次。因为 Kimi 大多数时候表现都挺不错的，但有时候它在自我复核时会陷入奇怪的循环。好吧，这次它表现不错。是啊，前几天我用 Kimi 的时候，它第一次就得出了正确答案，然后却决定用另外四次调用来检查自己的结果。And so, you know, when we're evaluating two two models, we want to both look at the final answer, but also how long did it take to get to that final answer, right? actually, Kimi did do something really interesting the other day. Let's see if it does it again with this one. Cuz Kimi most of the time works quite well, but Kimi does sometimes get into weird loops when it's kind of like double-checking itself. okay, this time it did well. Yeah, the other day when I did Kimi, it like got the right answer on the first try and then decided to check its work with the like four other calls.

我当时就说：“不，没事。你没问题。”I was like, "No, you're good. You're good."

所以，我们这里要看的就是这个，对吧？它到底需要调用多少次才能到达终点。so that's what that's what we're looking at here, okay? Is how many calls does it actually need to get to the end.

使用 LangChain v1 的智能体Agents with LangChain v1

好，接下来我们可以讲智能体循环了。好吧，我现在讲得确实挺快的。好，不错。好了，我们已经打下基础了，对吧？我们已经看过工具调用，也看过循环中的工具调用。All right, so next we can go to agent loops. Okay, I'm actually going quite fast through this now, actually. okay, good. all right. So now we've got the basis, right? We've seen tool calling, tool calling in a loop.

现在我们就能真正构建一个像样的智能体了。通常我们会使用智能体框架来构建。这里我已经放入了三种不同的智能体框架，供你挑选自己喜欢的一个：LangChain、Pydantic AI 和 Agent Framework。So now we can actually build like a proper agent. And for this, we'd usually use an agent framework. I've gone ahead and put three different agent frameworks in here so that you can pick your favorite. So we have LangChain, Pydantic AI, and Agent Framework.

我肯定还漏掉了其他智能体框架，不过希望你能从中找到一个喜欢的。所以这些框架的理念都差不多。好吧，我们想演示哪个？大家投个票吧。LangChain。I'm sure there's other agent frameworks as well that I missed, but hopefully you can find one that you like there. so these are all, you know, the same idea. All right, which one do we want to show? Someone vote. LangChain.

好吧，LangChain 仍然很受欢迎，希望今天的会议现场有一些 LangChain 的朋友。All right, LangChain is still very popular and hopefully there's some LangChain folks here today at the conference.

好的，关于 LangChain，我们使用的是 LangChain V1。如果你过去一年没试过，这是他们最新、很不错的 TK 版本。我们通过 LangChain 连接 Foundry 模型。这里是一个旅行规划智能体，这是个经典示例。它有一些工具：搜索航班、搜索酒店，以及检查预算，用来确认某个航班和酒店是否在预算范围内；还有一个工具可以搜索某个城市的活动。我还添加了一些中间件，方便我查看正在调用哪些工具。All right, so with LangChain, we are using LangChain V1, which is a nice newest TK version of theirs if you haven't tried it in the last year. And so we connect to the Foundry models from LangChain. and what we have here is a travel planning agent, which is a classic example. So, it has tools. So, we have a tool for search flights, search hotels, and check budget to check if a certain flight and hotel fits within the budget, and then search activities within a city. then I've got some middleware, which is just for me to see what tools are happening.

然后是实际的代理调用。代理有一个系统提示词，告诉它要帮助用户查找航班和酒店。你知道，它还包含一些规则，比如同时搜索航班和酒店、优先选择直飞航班，诸如此类；始终检查预算，最后再根据剩余预算询问用户想参加哪些活动。And then we have the actual agent call. So, the agent has a system prompt that says you're going to help users find flights and hotels. you're you know, it has some rules here like search for both flights and hotels, prefer direct flights, yada yada, always check budget, and then ask for activities at the end based off the remaining budget.

然后是用户消息。我下个月要从旧金山飞往纽约市，住3晚。我的总预算是6,600，希望乘坐直飞航班，并安排一项活动。好的。Then we have the user message. I need to fly from SF to New York City next month for 3 nights. My total budget is 6 600, and I want direct flights and an activity. All right.

好了，我们已经有了所有这些内容，然后正在运行代理。这里还有一些额外的代码，只是用于记录工具调用，这样我们就能看到它们。那我们就来试试看吧。好。So, we have all that, and then we are running the agent. There's some additional code here that's just logging out the tool calls so that we can see it. So, let's go ahead and try that out. Okay.

所以，这会记录发生过的工具调用。我们先从 GPT-5.5 开始。这里有意思的是，你会看到第一个回合。让我把它放大一点。And so, this is going to log out the tool calls that happened. We're starting off with GPT-5.5 here. Now, what's interesting here, you'll see turn one. Let me make this bigger.

第一轮，它说是并行的。Turn one, it says is parallel.

并行工具调用Parallel tool calls

所以，LLM 有一项能力是这样的：很多 LLM 都支持并行工具调用。如果你给它们一组工具定义，它们就可以建议同时调用多个工具，也就是在同一轮中并行调用。这从性能角度来说非常有用，因为在你的服务器上，你实际上可以并行向两个不同的数据库发送请求，对吧？所以，只要这些操作彼此独立，这就是 LLM 能做到的一件很棒的事：它可以说：“嘿，这两件事可以并行处理。”So, one thing is you can do with LLMs is like many LLMs support parallel tool calling where if you give them a bunch of tool definitions, they can suggest a multiple tools that can be called at the same time in the same turn. And this is really good from a performance perspective because on your server, you can actually file like you can send off requests in parallel to like two different databases, right? so, as long as they're independent of each other, this is a great thing that LLMs can do is say, "Hey, these two these two things can be done in parallel.

“为什么不做这些呢？”所以，GPT-5.5 在第一轮时说：“嘿，搜索纽约市的酒店，并搜索从旧金山飞往纽约市的航班。”因为它判断这两者之间没有依赖关系。然后，等结果返回、拿到所有选项后，它根据那次调用返回的机票费用和酒店费用检查了预算。Why don't you do them?" So, GPT-5.5 on the first turn and "Hey, search for hotels in New York City and search for flights in from SF to New York City." cuz it's decided that there wasn't a dependency between them. And then, once that came back, once it had all the options, it did a check budget based off of the flight cost and the hotel cost that it got back from that call.

然后，它查看了剩余预算，并根据剩余预算调用了搜索活动。因此，这大概是我们针对这种场景能用到的最高效的智能体。接下来，我们看看换用其他模型时，结果是否相似。执行。And then, it saw what the remaining budget was and then it did a call for search activities with that remaining budget. So, this is probably the most efficient agent that we can have for this scenario. So, let's see if we get similar results when we change models here. Do.

当然，在这里你甚至可以切换并比较不同的模型和框架，对吧？因为每个框架的工具调用循环都略有不同。我们也把它称作 harness，对吧？所以很多时候，你会想比较不同的模型和 harness，并从中发现一些有趣的差异。And of course, here you can even change and compare across both models and frameworks, right? Because every framework has a slightly different tool calling loop. We also call it the harness, right? So, a lot of times that you're you want to compare both models and harnesses and you'll see there's interesting differences across them.

好。这次我们用的是 Kimi，而 Kimi 的并行处理效果也不太好。我不知道这是因为 Kimi 不支持并行调用工具，还是它只是在这种情况下决定不这么做。All right. So, this time we're with Kimi and Kimi is not parallelizing as well. I don't know if that's because Kimi doesn't support parallel tool calling or it's just deciding not to in this case.

所以，这次它做了同样的事。实际上，它调用得还更多。它搜索了航班和酒店，分两轮完成，所以效率已经低了一些，因为它没有并行处理。接着，它决定调用两次来检查预算，因为我觉得它是在多核对一个选项。So, this time it did the same Actually, it did even more calls. So, it did search flight and search hotels. It did over two turns, so that's already less efficient cuz it didn't parallelize. Then, it decided to do two calls to check budget cuz I think it was checking like one more option.

然后，它完成了最后一步。所以，这次多用了两次工具调用，也花了更长时间，因为它没有把第一组轮次并行处理。好，看看 DeepSeek 的表现如何。你现在应该能看出为什么这很难，因为大家都问我：“你怎么评估智能体？”智能体是最难评估的，因为其中涉及的事情太多了。And then, it did the final thing. So, this one took two extra tool calls and it took longer because it didn't parallelize that first set of turns. All right, let's see how we do with DeepSeek. And you can start to see why it's hard cuz everybody asked me like, "How do you evaluate agents?" Agents are the hardest to evaluate because there's so much going on.

在这个过程中会发生很多不同的事情。比如，它调用的工具对不对？调用得高效吗？有没有并行调用？There's so many different things that are happening along the way. It's like, are is it calling the right tools? Is it calling them efficiently? Is it calling things in parallel?

在这种情况下，你知道，这个模型并没有进行并行处理，但至少只调用了四次工具。所以，嗯，效率还算可以。然后说到 Mistral，Mistral 经常会忘掉最后一步。我们拭目以待吧。你知道，这就是智能体的另一个问题。in this case, you know, this one didn't do any parallelizing, but at least it only did four tool calls. So, you know, reasonably efficient. And then Mistral Mistral often will see Mistral often forgets the very last step. And so, we'll see. you know, that's the other problem with the agents.

有时，当智能体的上下文窗口快满时，可能会忘记一些较早的指令。它们可能不会那么仔细地留意最初的步骤是什么。有时你得在过程中稍微提示一下，让它们记起来：“哦，等等，你应该用这最后一步收尾，对吧？”好，它没有并行执行。所以，我们接下来会看到，GPT-5.5 非常擅长并行调用工具。Sometimes agents, when they get to the like once their context window fills up, they might forget some earlier instructions. They may not be as thorough with, you know, paying attention to what the original steps were. And sometimes you have to like kind of prompt them along the way to remember like, "Oh, wait, you're supposed to you know, finish up with this final step here, right?" Okay, so it did it didn't do parallel. So, what we're going to see is GPT-5.5 is very good at parallel tool calling.

其他那些没有、没有在做这件事。好吧，Mistral 决定只搜索航班和酒店，然后就不做其他任何事了。所以，它就这么半途而废了。这就是一个例子：我们得再引导它一下，比如说：“嘿，你确定所有事情都做完了吗？”你可以用类似 route loop 的东西。Jeff Huntley 今天也在这里。他做出了 route loop。the other ones not not aren't doing it. Okay, so Mistral decided to only search flights and search hotels and then not do anything else. so, it did it did it kind of gave up there. so, that's an example where, okay, we'd have to do some additional steering there to be like, "Hey, are you sure that you have everything, right?" you could use something like the route loop. Jeff Huntley is here today. He made the route loop.

而且你知道，你可以——你可以强制智能体表现得更积极一些，而不是只使用默认循环。好吧。那么刚才是一个智能体示例，我们在其他框架中也有智能体示例。不过你会发现，它们的代码真的非常相似。我倾向于认为，目前大多数智能体框架彼此之间更相似，而不是更不同。And you know, you can you can force you know, you can force agents to be more aggressive instead of just using the default loop. All right. Okay, so that was an agent example of saying we also have agent examples in the other frameworks. but what you see is the code is really similar across them. I tend to think most agent frameworks are, you know, pre- more similar than they are different at this point.

使用字符串检查进行评估Evals with string checks

不过，你知道，你可以去看看它们。好吧，现在我们可以谈谈评估了，对吧？到目前为止，我们做的都是手动运行并观察结果。But, you know, you're welcome to check them out. All right. So, now we can talk about evaluations, right? So, we've been doing everything just, you know, manually running them and observing the results.

了解你能看到的差异类型是很有帮助的。不过，当你真正要把某些东西用于生产环境时，你确实会希望进行批量自动化评估，能够可靠地运行并比较结果，对吧？那么，我们可以做的第一步，就是只用字符串比较来进行评估，对吧？我这里的例子是调用 create calendar event 工具，因为我们实际上会拿到参数，所以可以直接进行字符串比较，问一句：“嘿，它返回的是 virtual，对吧？”That's nice to get a feel for the kind of differences that you can see, but when you're actually going to be, you know, doing something for production, you do want to actually have bulk automated evaluations that you can reliably run and compare results, right? So, the first step that we can do is just doing evals for without, you know, with just using string comparison, right? so, the example I have here is with that create calendar event tool call, cuz what we can do is like we're actually getting back the arguments, we can actually just do a straight string comparison say, "Hey, did it give back virtual, right?

我们希望这里返回 virtual，它返回的是 virtual 吗？”所以，测试单次工具调用时，你只要检查一下：它有没有检查——它有没有调用正确的工具，以及有没有使用正确的参数？这就是我们要在这里做的事。我们还设置了一大堆测试用例，对吧？因为运行评估时，你需要很多很多测试用例。你也希望添加更多测试用例的过程足够简单，因为等你的应用上线后，你就能看到用户实际是怎么使用它的，然后你会想：“哦，好吧。We wanted virtual for this, did it give back virtual?" So, when you're testing a single tool call, you can just check to see, did it check did it call call the right tool and did it use the right arguments? So, that's what we're going to do in this one. and we set up a bunch of test cases, right? Cuz when you're running evals, you want to have lots and lots of test cases. You also want it to be easy to add more test cases, cuz once your app is in production, you're going to see how users really use it and you're like, "Oh, okay.

“谢谢。一切都坏了。”不，然后你会把这些新的用户输入案例添加到你的评估中，确保自己不会偏离，对吧？这就叫模型漂移，或者说逐渐偏离。所以，你不希望你的评估偏离用户输入。Thank you. Everything's broken." No, and then you're going to go and add those new cases of user input to your, you know, your evaluations to make sure that you don't drift, right? This is called like model drift or your drift away from it. So, you don't want your evals to drift away from your user inputs.

所以，一旦它们投入生产，你就需要一套好用的系统，把新的用户输入加入评估用例中。我这里的做法是准备了很多不同的用例。比如，这里有一条用户消息，还有一些检查项。这些检查项只是做字符串比较、列表比较之类的操作，我们会检查不同的内容，对吧？我们会检查日期计算。So, once they're in production, you want a good system for getting new user inputs into your eval cases. So, here what I do is I've got like lots of different you know, cases here. Like, here's a user message, here's the checks, and these checks are just doing really string comparison, some list comparison, and we check different things, right? We check the date math.

我总会拿日期计算来测试大语言模型，因为它们通常在这方面不太行。所以再给个建议：如果你要处理日期，就给它们——给它们一个做日期计算的工具，好吗？因为在脑子里算日期太难了，尤其是遇到闰年的时候。我当时正在评估……I always check LLMs on date math, cuz usually they're kind of bad at it. So, another tip, if you're doing dates, give them just give them a tool to do date math, okay? Cuz it's too hard to do date math in your head. especially on leap years. I was evaluating

Pamela Fox：还有关于闰年的 LLM，那挺难的。然后我们遇到了，嗯，含糊不清的时长。我做了一个西班牙语的，对吧？用不同的语言进行测试挺好的，不管你们打算支持哪些语言。Pamela Fox: And LLM on a leap year, that was hard. then we had okay, ambiguous durations. I did one in Spanish, right? It's nice to test across different languages, whatever languages you're going to support.

我猜这个写得、写得特别俚语化。嘿，能帮我订一下大会议室吗？好。行，那我们那儿有各种边界情况，对吧？this one I guess is written written real slangy. Yo, can you book the big conference room? Okay. All right, so we have various edge cases there, right?

然后我们就可以运行这些评测了。好，这种情况下，我会针对每个模型和每种用例都运行这些评测。很多时候，实际运行评测时，我只会针对单个模型运行，然后拿结果和之前某次运行的结果进行比较，比如之前用的是另一个模型。不过你知道，今天我们正在积极地比较不同模型，所以我会针对所有模型运行，这样我们就能看到每个模型是否都进行了工具调用、检查返回的参数，并判断结果是通过还是失败。评测有一点需要注意：你不希望它们只是作为 pytest 的一部分，比如放进你的测试套件里，因为它们运行很慢，也很耗费资源。And then we're let me to run these evals. Okay, and these evals in this case I'm going to run it across each of the models and the use cases. A lot of times in reality when I'm running evals, I'm just running it on a single model and then I'm comparing to like a previous run where maybe I'd done a different model. but you know, for today since we are actively comparing across models, I'm going to run it across all the models so we can see it's for each of them it's doing the tool call, checking the args that come back, and deciding if it's a pass or a fail. now here's the thing about evals is that you don't want to just have these like just be part of like your pytest, like your test suite because these are slow, these are expensive.

是的，有什么问题吗？Yeah, question?

问答：评估框架Q&A: eval frameworks

观众：我看你就是直接上手，你知道，拿字符串来比对。你知道有没有什么框架能更优雅地实现这一点吗？Audience: I see you just roll, you know, checking strings against do you know if any frameworks do this a bit more elegantly

Pamela Fox：你不喜欢我做字符串检查。是啊，说到评估所用的框架，首先，大多数可观测性服务商都内置了评估功能，对吧？比如你用 LangFuse、LangSmith、LangChain、Arize，它们都有各自内置的框架。至于 Foundry，我们有两种不同的框架，我其实在代码仓库里放了示例。一个是 Azure AI evaluation SDK，另一个是 OpenAI 包里的 OpenAI.evals。框架还有很多，比如 DeepEval；如果你用 JavaScript，也可以用 Promptfoo。Pamela Fox: You don't like my string checks. yeah, so framework-wise for evals a lot of so first of all, most observability providers have evals built in, right? So if you're doing LangFuse, LangSmith, LangChain, Arize, like they all have their own framework built in. For Foundry, we have we have two different frameworks for it and I actually I do have the examples in the repo. those ones there's Azure AI evaluation SDK, Open AI.evals from the OpenAI package. So there's a bunch of frameworks, there's also DeepEval and Prompt Prompt foo in JavaScript if you're a JavaScript person.

那是哪一个？And which one?

受众：Vali。Audience: Vali.

帕梅拉·福克斯：瓦利？Pamela Fox: Vali?

受众：类似这样的东西。Audience: Something like that.

Pamela Fox: Mark Vali, okay.

受众：V A L L YAudience: V A L L Y

Pamela Fox：V A L L Y，好吧。我的意思是，你就用那个。所以有不少评估框架，我很快会用一个更高级的。不过对于字符串检查，关键是你其实没必要用很高级的框架，因为你……Pamela Fox: V A L L Y, okay. I mean you use that one. so there's a number of eval frameworks I'll I'm going to use a fancier one soon. But for string check, but here's the thing like for string check, you don't really have to do a fancy one because you're like

受众：如果你希望它执行某个特定的操作， suliaqAudience: If you're looking for it to do a specific

帕梅拉·福克斯：是的。Pamela Fox: Yeah.

受众：对你提到的……进行并行化Audience: Do the parallelization you mentioned for

Pamela Fox：对，对。所以对于工具调用，你确实需要用更复杂一些的方法。是的，我想先从基础的讲起，但大多数时候，我们没法只靠基础方法，对吧？哦，他们这次做得确实很好。哇，不错。所以通常我们得转向更复杂的评估方法。Pamela Fox: Yeah, yeah. So for tool calls, yeah, that's where you're going to get a bit fancier, yeah. Yeah, so I wanted to start with the basic, but yeah, most of the time we can't get away with just the basic, right? oh, they did actually really well this time. Wow, nice. so usually we're going to have to move to fancier evaluation.

将 LLM 用作评判者LLM as a judge

所以这时候我们就开始引入 LLM 作为评判者。很多时候，输出会更模糊，我们需要借助 LLM 来帮我们推理这种模糊性。有很多框架都封装了这类功能。这个例子里，我决定把所有内容都直接写在代码里，这样你就能看清楚。使用 LLM 作为评判者的思路是，你需要给它一些指示，把 LLM 设定为评判模式，然后告诉它：“好的，你现在是一名评估者。这是你要评估的内容，你需要判断它是通过还是不通过。”对吧？So that's when we start to bring in LLM as a judge. so this is cuz a lot of times, you know, the output is a lot fuzzier and we need to use an LLM to help us reason about that fuzziness. And there's many frameworks that do wrap this For this example, I just decided to write it entirely in the in the code so you could see. So the idea when you use an LLM as a judge is that you need to give it, you know, put the LLM in judge mode and say, "Okay, you are an evaluator. This is what you're evaluating and you need to either say does it pass or does it fail?" Right?

我们试图让 LLM 判断这算通过还是失败？我们也希望它给出理由，对吧？让它给出理由真的非常非常有帮助。所以我们要的是通过或失败，以及理由。然后我们把查询和响应给它。We're trying to force the LLM to say does this pass or fail? and we also want it to give it reason, right? It's really really helpful to have it give it reason. So we want pass, fail, and a reason. And then we give it the query, we give it the response.

为了让它有依据，我们还想给它提供上下文，因为这有助于它真正进行推理。所以有时候，要想真正评估它，你确实需要让它获取更多信息。比如工具调用，如果你想评估它使用工具调用是否高效，就不能……你得把所有内容都给它，包括工具调用的定义。And for grounding this, we also want to give it the context cuz that helps it actually reason about it. So sometimes in order to really evaluate, you do need access to more information. So like tool calls, if you're trying to evaluate whether it was an efficient user of tool calls, you can't you have to give it everything. You have to give it the tool call definitions.

它必须提供它所进行的工具调用轨迹，对吧？所以，要构建非常好的评估器，很多时候你不只是给它输入和输出，而是把整个过程中发生的一切、展示出来的一切都给它。因此，我们会告诉 LLM：“嘿，我们将使用 GPT-5 的评审模型。它是个相当不错的评审模型。”然后我们会要求它输出一些供我们查看的 JSON。You have to give it the traces of the tool calls it made, right? So to build really good evaluators, a lot of times you're you're not just giving it the input output, you're giving it every everything that happened along the way, everything that was shown. so here we, you know, we tell the LLM, "Hey, we're going to use the judge model of GPT-5. It's a pretty good judge model." And we're going to ask for it to output some JSON that we can look at.

那么我现在就来运行这个作为裁判的 LLM。问题是，这样会更慢，因为现在每次运行评估时，我们不仅要运行自己的代码（其中会用到一个模型），还要再用另一个 LLM 来评判这段代码，对吧？所以每个评估用例实际上都需要调用两次 LLM。而且很多时候，你还会用多个 LLM 来充当裁判。So let me go ahead and run this LLM as a judge. Now, here's the thing, this is going to be even slower because now each time we're running an eval, we're both running our own code, which is using a model, and we're then using another LLM to judge that code, right? So every single eval case is actually two LLM calls. And then many times you actually run multiple LLMs as a judge.

比如你会运行五个不同的评估器。每个评估器都会单独调用一次 LLM，对吧？所以评估比较棘手的地方就在于，你不能一直运行它们。你也不一定能在每次 CI 和每次提交时都运行，因为它们运行起来相当慢。因为通常在生产环境中，比如评估 RAG 时，你一般至少需要 200 个测试用例，可能要 800 个。Like you'll have like five different evaluators that you're running. Each of those is going to be a separate LLM call, right? so that's what is tricky about evals is that you can't just run them all the time. You can't necessarily run them on every single CI and every single commit because they are fairly slow to run. Cuz normally in production, like for rag, when I'm evaluating rag, you know, you usually want like 200 test cases at least, maybe 800.

这就意味着要多次调用大型语言模型，对吧？所以会花一些时间。因此，你确实需要考虑自己要多频繁地运行评估。你可以看到，它正在处理文本，其中有许多不同的测试用例、不同的问题和答案，然后它会查看结果。And that's like a lot of calls to an LLM, right? So that takes some amount of time. so you do want to think about how often you're going to be, you know, running evals. So you can see that it's going through its text it's got a bunch of different test cases, different questions and answers. And it's looking at the results.

现在，在这个例子里，我做的就是提前把问题和答案跑了一遍。这也是一种很常见的做法。比如，提前把评估器需要的上下文都准备好，因为这样你就可以独立于应用来调整评估器，对吧？这样效率更高。因为很多时候你会发现，评估器是不正确的，对吧？Now, in this case, what I did was that I like pre-ran the questions and answers. This is a common way of doing it, too. Like get, you know, get all your get your context for the evaluator beforehand because then you can change your evaluator independently from your app, right? and that's more efficient. Cuz a lot of times what you realize is that your evaluator is incorrect, right?

就像这样，你还没有写出一个好的评估器。所以你在用 LLM 做评判时，必须弄清楚：等等，问题出在我的应用上，还是出在我的评判器上，对吧？Like that you haven't written good evaluator. so that's what you have to figure out when you're doing LLM as judge, you have to figure out like wait, is the problem my app or is the problem my judge, right?

你还希望借助人工输入来校准你的评判模型。所以，在某个领域开展工作时，找一位领域专家合作会很有帮助。他们可以帮你标注数据，并判断他们是否认同你的 LLM 评判结果：这到底算通过还是未通过？现在我们拿到了结果，可以看到各个模型及其得分。大多数模型都得了三分之二，Mistral 得了一分之二。And you also want to calibrate your judges with human input as well. So, it definitely helps me working on a domain you know, or work with a domain expert that can help you label your data and say and you know, do they agree with your LLM as a judge? Was it actually a pass or a fail? so here we get the results back and we can see you know, the model and their scores most of them got like two out of three. Mistral got one out of three.

你知道，他们在让这方面更有依据这件事上还有些工作要做。这就是使用 LLM 作为评判者的总体思路。你也看到了，有不少框架内置了 LLM 评判功能，让这件事更容易实现。不过我想说明，你也完全可以自己动手做。You know, so they're they're they have some work to do on being more grounded here. So that is the generator of using LLM as a judge. As you see, there's quite a few frameworks that make it, you know, that have built-in LLM as a judge that make it easier to do it. but I want to show that you know, you can't you can't also do it yourself.

哦，另外，如果你对评估感兴趣，我推荐看看 Hamel.dev。我不知道他这周会不会参加会议，但他有很多特别棒的资源，而且经常开设关于评估的课程。他做的梗图也很棒，我昨晚还用到了这些梗图。Oh, also I should recommend if you're interested in evaluation, Hamel.dev. I don't know if he's at the conference this week, but he has the most amazing resources and he teaches courses all the time about evals. And also he's very good memes. I actually used these last night.

所以，如果你哪天和别人聊评测，想找个评测梗图，那我昨晚在做什么来着？对，构建一个用作评判者的 LLM，这事儿很难。So, if you're ever chatting with someone about evals and you need an eval meme, what was it what I was doing last night? Yeah, building an LLM as a judge it's hard.

Pamela Fox：这是一个非常重要的资源。我们如何信任 LLM 评审？你必须校准你的 LLM 评审。好的。Pamela Fox: so very important resource here. How can we trust the LLM judge? You have to calibrate your LLM judge. Okay.

好。那么，这就是让 LLM 担任评判者。接下来，不过你也看到了，构建评判评估器很难，对吧？比如，你得在 Twitter 上关注 Hamel，参加他的课程，学习编写优质评测的各种最佳实践。All right. So, that was LLM as a judge. So now but then they as you see, it is hard to build judge evaluators, right? Like you have to like follow Hamel on Twitter and take his course and learn all these best practices for how you write a good eval.

断言：源自规范的评测ASSERT: evals from a spec

微软研究院推出了一个新工具。它非常新，大概是上周才推出的，叫作 Assert。Assert 可以帮你根据规范编写评判器。所以，你先写一份规范。So we have this new tool from Microsoft Research. It's very new, like the last week. it's called Assert. And Assert helps you write the judges based off of the spec. So, you write a spec.

你会说：“嘿，这是我正在构建的智能体，大致来说，我想让它做这些事，这是我对它应该如何工作的想法，以及它应该支持哪类用户。”然后它就会生成大量测试用例，把你的规格说明扩展成实际的评判标准，并根据所有这些内容帮你把一切都弄清楚。这一切都非常基于研究，对吧？所以，我们已经在这里为旅行规划器做了这些。那，这个……它在这个 assert eval 下面。好。You say, "Hey, this is the agent I'm building and this is, you know, roughly what I want it to be doing, my idea about how it should work, and what kind of users it should support." And then it will generate lots of test cases. It'll like flesh out your spec into into actual judging rubrics, and figure all of that out for you based off of, you know, all the it's very based in research, right? So, we've done that for the travel planner already in here. so, here's the so, it's under this assert eval. Okay.

所以，我们可以看到，我们编写了一个 YAML 文件，因为我们都希望生活中能有更多 YAML。So, what we can see is that we write a YAML file cuz we all wanted more YAML in our life.

我对 YAML 的认识是，我可能讨厌 YAML，但对于我要用它做的事情来说，它也比其他所有东西都好。所以，就这样吧。在这个 YAML 里，你知道，我们描述上下文，并说明：“好，我们要用这个评判模型。我们关心的维度包括预算，也就是预算压力。”所以，如果我们知道自己关心的是预算充足、预算紧张、预算压力不明显之类的情况，就可以提供更多信息。你知道，这样就可以对它进行分层。这就叫分层。你可以按需预先分层，也可以让它自己生成分层。然后这里还有更多维度。What I realized about YAML is that I might hate YAML, but it's also better than everything else for what I need it for. So, there we go. so, in this YAML, you know, we describe the context, and we describe, "Okay, we're going to use this judge model. these are the kind of dimensions we care about is budget, you know, budget pressure." So, we can give it more information if we know, like, "Oh, we care about ample budget, tight budget, invisible." you know, so, you can you can stratify it. This is called stratification. You can you can pre-stratify if you want, or you can just have it come up with the stratification. and then we have more dimensions here.

还有这个，还有一种技能，也就是智能体技能，可以帮你编写这个 YAML 文件，这样你就不用自己动手写了，因为没人想写，对吧？And there's this there's also a skill an agent skill that helps you write this YAML file so that you don't have to write it yourself cuz no one wants to write it, right?

然后我们有了真正进行评估的部分。这就是之前看到的 LangChain 旅行规划器。这样做只是为了确保所有操作都有追踪记录，因为它的意思是：“进行评估时，你确实需要查看每一次工具调用，这样才能真正做好评估。”好。接下来它会运行所有内容，然后我们就能看到结果。这个过程确实需要花上一段时间。And then we have the actual thing that's evaluating. This is the LangChain travel planner that we saw before. so, this just makes sure that it's tracing everything cuz it's saying, "When you are doing evaluations, you do want to see every single tool call so that you can really properly do these evaluations." All right. So, then it runs it all, and then we can see the results. It does take a, you know, a decent amount of time.

所以，我们已经把评测放进来了。我已经让它们跑起来了。还有一个小小的前端界面，可以用来查看它们。好，行。那么，这是这次搜索评测的结果。基本上，它列出了不同的行为类别，对吧？So, we've already got the evals in here. so, I've got them running. there's like a little like front end that you can use to view them. Okay. All right. So, here is the result of this a search evaluation. so, basically it you know, it came up with the different behavior categories, right?

所以，像预算不可行、违反直飞偏好规则、必须使用工具之类的，对吧？So, making like budget in infeasibility, direct flight preference rule violation, required tool use, right?

于是，它会列出哪些行为是——你知道的，允许的，哪些是不允许的。然后它会生成测试集。这个测试集真的挺有意思的。好了，开始。场景 65。So, it comes up with what behaviors are you know, allowed but not allowed. And then it comes up with the test set. The test set is really is pretty fun. and here we go. Scenario 65.

安装程序走过长途客车之前的红眼，一个周末带着六条家长短信去看 Alma。我真的很喜欢这些显然是 LLM 生成的标题，对吧？然后它生成了所有这些拿着剪贴板的教练。它可以——它会根据那个 YAML 生成所有这些场景，试着模拟用户，对吧？因为整个事情是，在投入生产之前，很难启动评估流程，但你又希望在投入生产之前就有评估，对吧？Red eye before the installer walked through the coach with six parent text one weekend to see Alma. I really like that these are obviously LLM generated titles, right? So, it came up with all of these coach with a clipboard. it can it comes up with all these scenarios based off that YAML to try and simulate users, right? Cuz the whole thing it's really hard to bootstrap your evaluation before you go into production, but you want to have evals before you go into production, right?

那么，你怎么确定自己的测试集应该是什么样的呢？这个断言框架在构建测试集和启动评测方面能帮上很大忙。So, how do you figure out, you know, what your test set is going to be? And so, this assert framework helps a lot with making that test set and bootstrapping your evals.

所以，它包含提示词和场景。So, it has prompts and scenarios.

不同模型中的 ASSERT 结果ASSERT results across models

然后我们来看结果。好，结果显示，它在 DeepSeek、Kimi 和 GPT-5.5 上都运行了一遍，得出了这里这些数字。在这种情况下，数字越低越好。所以我们可以看到，GPT-5.5 的表现比其他模型好得多。And then we have the results. Okay. So, for the results, it ran it across DeepSeek, Kimi, and GPT-5.5 and came up with the you know, with the numbers here. and in this case the lower numbers are better. And so, we can see that GPT-5.5 did way better than the other ones.

说实话，我有点惊讶，因为我们今天看到 Kimi 总体表现相当不错，我原本觉得它大致不相上下。不过至少在这个场景下，GPT-5.5 的违规次数最少，所以它在这里得分最好。嗯，没错。这就是 Assert。我得说，我刚开始接触 Assert，因为它大概一周前才推出。这是我目前为止对它的全部使用经验。不过这里的想法是，它可以帮你构建评估器、建立测试集、遵循最佳实践，并为你的评估流程打好基础，这样你就能从一些相当不错的评估开始。I was actually surprised cuz Kimi, we've seen that Kimi has generally been quite good today and I was thinking it was kind of on par, but at least for this scenario, GPT-5.5 had the lowest number of violations, and so it is it's scoring the best here. so yeah. So this is assert. I will say I'm I'm brand new to assert since it only came out like a week ago. so this is all I've I've used it so far, but the idea here is that it's helping you to build those evaluators to build that test set and to follow the best practices and bootstrap your evals so that you can start start off you know, with some really good evals.

因为我们看到的情况是，我们一直在告诉大家，比如你们得做评测，得做评测，结果根本没人做评测，或者做的都是默认的评测，比如连贯性和流畅度。千万别做这些评测。比如说，大语言模型本来就很擅长生成连贯的内容。你根本不需要检查大语言模型的内容是否连贯。Cuz what we're seeing is that we keep telling people like you got to run evals, you got to run evals, and then no one's running evals or they're running like the default evals like coherence and fluency. Never run those. Like LLMs are great at being coherent. You never have to check if an LLM is coherent.

如果有些评估器没通过，那就说明发生了很糟糕的事情。我有时确实会在 RAG 项目中使用事实依据性和相关性评估器，但大多数时候，内置评估器并不是你想要的。你需要构建自定义评估器。这正是它能帮你的地方：你需要自定义评估器，才能真正判断你的智能体是否表现良好。不过，你也需要一些帮助来搭建这些自定义评估器。If some if they're not, like something really bad has happened. I do sometimes use the groundness and relevance evaluators for my rag things, but most of the times the built-in evaluators are not what you want. You want to build custom evaluators. And that's what this is helping you with is like okay, you need custom evaluators to really see if your agent is doing a good job. but you know, you need some help bootstrapping those custom evaluators.

好的。哦，我们还剩 3 分钟。All right. Oh, we have 3 minutes left.

我说得太多了。好了，终于，之前我们在聊提示词工程：怎样调整提示词，才能让这些模型表现得更好？这相当痛苦，对吧？如果每次切换模型都得调整提示词，还要重新跑评估，那工作量可不小。I've been blabbing. Okay, finally, you know, earlier we were talking about prompt engineering like okay, how can we tweak this prompt in order to work better for these models? That's pretty painful, right? If you have to tweak the prompt every time you switch models and like rerun evals, that's like a lot of work.

哦，还有一个小计时器。谢谢。好吧，如果你每次都得调整模型，那工作量可不小，而且你也不知道该往哪个方向调整，对吧？你不了解每个模型的具体细节，也不知道怎样才能让它真正发挥出色。Oh, there's a little timer. Thank you. Okay. so if you Yeah, if you have to tweak the model every time, that's a lot of work and you don't know which way to tweak the model, right? You don't know the details of every single model, what's going to really work well for it.

所以，我添加的最后一个示例是使用 DSPy。So what I've added is this as a final example is using DSPy.

使用 DSPy 优化提示词Prompt optimization with DSPy

有人在这里用过 DSPy 吗？不错，有几个人用过。DSPy 的理念是，你可以声明自己的程序，比如说：“嘿，我正在编写一个程序，它要接收用户输入，然后将其转换成结构化的 counter bit。”也就是说，你声明程序的约束条件，然后 DSPy 会找出符合这些约束条件的提示词。所以，DSPy 会替你编写提示词。Has anyone used DSPy here? Cool, a couple people. So DSPy, the idea of DSPy is that you declare your program like you say, "Hey, I'm making this program that's expecting to take user input and then turning it into a structured counter bit." So, you declare the constraints of your program and then DSPy is going to figure out the prompt that adheres to those constraints. So, DSPy is going to write your prompt.

所以，基本上你是在把规范提交进去，然后让 DSPy 根据评估结果编写提示词，并思考：“什么样的提示词才能满足这些约束？”它实际上使用了一种很酷的算法，叫做 GEPA（遗传信条算法）。它可以利用这些遗传算法来找出下一个最值得尝试的提示词，看着它这么做真的非常有意思。So, you're basically checking your spec in and then having DSPy by write the prompts based off of doing evaluations and seeing, "Hey, what prompt is going to satisfy those constraints?" And it actually uses these this cool algorithm called GEPA, genetic credo algorithms. It can use these genetic algorithms in order to come up with the next best prompt to try, which is really really cool to watch it do that.

所以，这就是 DSPy 的理念。因此，基本上我们得设置……所以，它很不一样，对吧？你不能直接拿常规代码来用。你多少得把整个代码重写一遍，因为现在，你的 pro——我是说，你的提示词不在代码里，你的约束条件在代码里。So, that's the idea of DSPy. And so, basically we have to set So, it's very different, right? You can't just take your standard code and just use it. You kind of have to rewrite your whole code cuz now, instead of checking your pro- like your prompt isn't in your code, your constraints are in your code.

你的规范就是你的代码。这个提示是这个 DSPy 流程的输出，对吧？所以，我这里是在让它生成一个编号列表，其中每一行都包含确切数量的单词，而且每行的首字母拼起来会组成一个目标词。然后，这就是这类输入，接着我们会用一系列约束条件来检查结果是否正确，对吧？Your spec is your code. The prompt is an output from this DSPy process, right? So, here I'm doing I'm asking it to generate a numbered list where each line has exact word count and the first letter of the line spell out a target word. So, this is the kind of input and then we have a bunch of constraints that are checking to see is the result correct, right?

所以，你需要先设定好你的规范，然后设置一些检查和测试，看看：“嘿，输出是否满足这项约束？”这样 DSPy 就能不断迭代、进行爬山式优化，对吧？So, you need to set up your spec and then set up like checks, tests to see, "Hey, does the output satisfy this constraint?" And that way DSPy can keep iterating, hill climbing, right?

基本上就是不断进行爬山搜索，直到找到一个满足这些约束的提示词，对吧？所以，你把学生模型和反思模型交给它，然后它运行这个 GEPA 优化器，也就是尝试找到最佳提示词，最后我们就能拿到结果。我把结果存放在这里了，方便我们查看。这个过程确实要花不少时间。Basically hill climbing until it finds a prompt that satisfies those constraints, right? So, you give it the student model, the reflection model, then it runs this GEPA optimizer, which you know, tries to find the best prompt, and then we get back the result. So, the result I have it stored here so we can see it. It does take a fair amount of time.

这项任务花了大概 6 分钟。不过，这里的结果才是真正的说明。所以，我来把自动换行加进去。It took like 6 minutes on this task. but here, this result is the actual instructions. So, let me add word wrap into there.

用于 Mistral 的 DSPy 提示词A DSPy prompt for Mistral

没错。所以，这就是它根据那些限制条件和那些检查生成的系统提示，对吧？这挺不错的，因为这是一个适用于 Mistral 的系统提示。我选 Mistral 是因为我们之前在它身上遇到的困难最大，对吧？Right. So, this is the system prompt that it came up with based off of those constraints and those you know those checks, right? And so this is nice because this is a system prompt that works with Mistral. Cuz I picked Mistral cuz that was the one we were having the hardest time with, right?

因此，尤其是在使用小模型时，这意味着我们可以用成本最低的模型，同时获得所需的结果，对吧？Dropbox、Shopify 等公司分享过一些很棒的演讲，介绍他们如何结合使用 DSPy 和 GEPA，从而改用小模型，甚至可能将其部署在自己的基础设施上，同时获得相同的效果。关键在于尝试利用这个流程，找出最优提示词是什么。因此，这确实是一种不同的应用开发方式。让提示词成为这一流程的产出，可能会让你觉得有些不习惯，但这也可能是一种非常强大的开发方式，能让你不再那么受限于特定模型，也更灵活地使用更多模型。So especially with using smaller models, this means we can use the cheapest model and still get the results we need, right? So there's been some great talks by like Dropbox, Shopify about how they've used DSPy plus GEPA so that they can move off the larger models and use the smaller models and maybe even deploy it on their infrastructure and get the same results. But the key was trying to you know using this order to figure out what that optimal prompt is. So this is a definitely a different way of writing your apps and it can feel a little different to you know to have the prompt be the output of this process, but it's also potentially really powerful way of writing it and can give you a lot more model independence and more flexibility to be able to use more of these models.

所以我想，嗯，也确保大家知道这也是一个选项。好。行，那我想展示的内容就这些。So I wanted to yeah, make sure that people knew about that as an option as well. Okay. All right. So yeah, so that was all the things that I wanted to show.

正如你所看到的，不同模型之间有很多差异，你可以通过不同的方式改进模型输出，然后对其进行评估，再继续改进。我确实鼓励你们多折腾折腾这些模型。As you can see there's lots of differences between models, different ways you can improve the model output and then evaluate it and then improve it. I do encourage you like yeah, just just tinker with the models.

Anthropic 与引文Anthropic and citations

我总是能学到很多，每次我，你知道，试用不同的模型，看看它们有多大的差异。哦，我还没展示 Anthropic。我本来是想展示 Anthropic 的。好。It's I always learn so much whenever I you know try out different models and see how different they are. Oh, I never showed Anthropic. I meant to show Anthropic. Okay.

我也部署了一些 Anthropic 的模型。不过，使用 Anthropic 必须调用不同的 API，所以我大多数时候都没用 Anthropic。当然，你可能会对 Anthropic 感兴趣。I do have some Anthropic's deployed as well. With Anthropic you do have to use a different API. That's why I wasn't using Anthropic most of the time. But of course you might be interested in Anthropic.

他们改用 messages API，所以你确实得调整一下代码。不过说到 RAG，我其实特别喜欢这种方式，因为有个有意思的地方：Anthropic 在模型中内置了引用这一概念。所以，你实际上可以用这种结构化格式提供来源。也就是说，你可以说：“嘿，这是我的来源。”They use the messages API instead, so you do have to switch your code around. But I actually specifically like it for rag because here's interesting is that with Anthropic they actually have this notion of citations built into the models. So, you can actually you provide the sources in this structured format. So, you say, "Hey, this is my source.

“这是一个文档。这是文本。下面是文档的文本。”然后启用引文功能。接着，Anthropic 模型实际上经过训练，会返回带有引文的文本块。It's a document. It's text. Here's the text of it." And then you enable citations. And then the Anthropic models are actually trained to return back blocks of text with citation.

所以，你可以用编程的方式提取这些内容，这在构建前端时非常方便。你可以直接链接引用。这个功能真的很棒。还有一点值得关注，那就是调用模型的各种不同方式。So, you can extract those in a programmatic way, which is really nice when you're like building front ends. You can just link your citations. So, that's really cool. So, that's even another thing to look at is you know, a whole different ways of calling models, too.

好。行。是的，我们时间差不多了。Okay. All right. And yeah, we're kind of at time here.

总结Wrap-up

这个仓库会继续保留。所以，欢迎你继续在仓库上工作。我确实需要轮换我放在 that.env 里的密钥，因为我是说，我信任你，但我不知道自己有多信任你。而且 Cedric 早就说过，我这么做不好。The repo will stay up. So, you're welcome to keep working on the repo. I do need to rotate the keys that I put in that.env because I mean, I trust you, but I don't know how much I trust you. And Cedric always already said I was bad for doing that.

所以，我没法再让这些密钥保持可用太久了。不过，希望你能用这个仓库试试看。你也可以用 llama 试试。我昨天添加了对 llama 的支持。So, I won't be able to keep those keys up for much longer. But, hopefully with the repo you can try it out. You can also try it out with a llama. I added a llama support yesterday.

所以，如果你愿意的话，你知道，你也想试试 Llama 吗？如果你用的是本地模型，基本上只要把示例改成指向本地的 Llama 服务器就行了。你还得改一下模型名称，对吧？不过一般来说，它们都应该也能和 Llama 配合使用，因为 API 是一样的。So, if you want you know, do you want to try a llama, too? If you're doing local models, you basically just change the examples to point at the local llama server. And you just have to change the model names as well, right? But, generally they should all work with a llama as well because it's same API.

好的。谢谢大家。哦，还有贴纸。All right. Thank you, everyone. Oh, and there's stickers.