用 NVIDIA Data Designer 教智能体学会搜索：Dhruv Nathawani 的数据构建实战

简要说明Brief Description

Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani， NVIDIA

目录Table of Contents

• 将代理行为转化为数据管道
  Turn an agent behavior into a data pipeline
• 为什么仅有互联网数据还不够
  Why internet data is not enough
• 超越重复提示的多样性
  Diversity beyond repeated prompting
• 种子、采样器和生成参数列
  Seed, sampler, and generation columns
• 笔记本一：问答数据
  Notebook one: question and answer data
• 难度采样器与分布
  Difficulty samplers and distributions
• 提示模板和结构化输出
  Prompt templates and structured outputs
• 预览几条记录
  Preview a few records
• 评估问题质量
  Judge question quality
• 生成一批
  Generate a batch
• 笔记本二：通过 MCP 搜索
  Notebook two: search through MCP
• 检查搜索轨迹
  Inspect a search trajectory
• 为什么模型需要搜索功能
  Why models need search
• BrowseComp 与难题
  BrowseComp and difficult questions
• 将维基数据路径作为搜索谜题的线索 seed
  Wikidata paths as search riddle seeds
• 笔记本三：端到端搜索数据
  Notebook three: end to end search data
• 工具说明和示例调用方式
  Tool instructions and example calls
• 检查生成的谜题和轨迹
  Inspect the generated riddles and traces
• 种子生成与智能体轨迹生成的对比
  Seed generation versus agent trajectories
• 监督微调与蒸馏
  Supervised finetuning and distillation
• 强化学习的替代方案
  Reinforcement learning alternatives
• 筛选并评估合成数据
  Filter and evaluate synthetic data
• 过时的来源与答案核验
  Stale sources and answer verification
• 人工审核模型判断
  Human review of model judgments
• 数据生成实践及其他任务
  Data generation practices and other tasks
• 合成人物画像与人口统计分布
  Synthetic personas and demographic distributions
• 内容整理、匿名化及相关工具
  Curation, anonymization, and related tools
• 资源和角色设定生成
  Resources and persona generation

将代理行为转化为数据管道Turn an agent behavior into a data pipeline

这个研讨会的主题更加通用，也更强大。我们要探讨的是，如何使用我们开发的工具 Nemo Data Designer 来构建数据流水线，从而为模型或智能体创建或添加各种功能。那么，我们开始吧。顺便说一下，我有一个 GitHub 仓库，里面放了一些笔记本。如果你感兴趣，可以扫描二维码，也可以访问那个 GitHub 仓库。其他所有幻灯片上也会出现同一个二维码。所以如果你错过了，还有机会获取。那么，这个研讨会的主题是什么？我们会在这里学到什么？首先，我们要学习如何将你希望智能体具备的目标行为转化为一个数据问题，对吧？基本上，我们能否为这种目标智能体行为创建数据，然后训练模型，让它学会如何完成这项任务。The workshop is about something more generic and more powerful. It's about how do we build data pipelines um using the tool that we created Nemo uh data designer uh to be able to create or add any kind of functionality to a model or an agent. So let's get started. And by the way, I have a GitHub repo uh where I have some notebooks. So if you're interested, you can scan the QR code or you can go to that GitHub repo. Um and I'll have the same QR code on all the other slides as well. So if you miss it, there'll be more chances to get it. So what is the workshop about and what will we learn here? Um the first thing we're going to learn is how do you actually take a target agent behavior that you want and turn it into a data problem right? So basically can we create data for this targeted agentic behavior and then can we train a model to learn how to um uh do that task.

所以我们构建了一个名为 data designer 的工具，用来搭建合成数据管道，它正是为此而设计的。我们今天要重点介绍的应用场景是搜索。顺便说一下，这是一个非常通用、可复用的框架。所以，如果你想为搜索以外的用途创建数据，完全可以用同样的方法来做。 今天的工作坊有两种参与方式。你当然可以实时跟着做，尝试运行 notebook；不过，我们大概只有一小时，甚至不到一小时，所以你也可以先听讲，之后再运行 notebook。总之，记得获取 GitHub 链接。如果你想要幻灯片，我也把它们放在 GitHub 仓库里了。无论你是现在跟着做，还是在工作坊之后再做，都需要准备四样东西。And so we built a tool called data designer to build uh synthetic data pipelines uh which does exactly that. And the use case that we're going to target today is search. Um, and by the way, this is a very reusable framework. So if you want to create data for something apart from search, this is exactly how you would do it. And there's two ways to participate in today's workshop. So um, we have, you know, of course, you can follow live, try to run the notebooks, but since we just have like probably an hour or less than an hour, um, you could also listen now and then run the notebooks later. Um, so yeah, make sure you get the GitHub link. And if you want the slides, I also have the slides on the on the uh GitHub repo. So you're going to need four things if you're going to follow along either now or after the workshop.

显然，你需要一台支持 GPU 的机器，可以是云虚拟机，也可以是工作站。嗯，你还需要一个 Tavi Search API 密钥。Tavi 是一家为智能体提供搜索服务的公司。如果你在他们的网站上注册，他们会免费赠送你一千个积分。你还需要 Data Designer，这是我们用来创建数据的工具。它完全开源，任何人都可以下载、使用，想怎么用都可以。我在 GitHub 仓库里提供了所有笔记本。今天的议程如下：我们先花几分钟聊聊什么是 Data Designer，以及我们为什么需要这样的框架，然后再讲一些基础知识。嗯，我会用代码演示三个笔记本。第一个笔记本讲的是 Data Designer 的基础知识：如何创建你想要的数据？Obviously, you need a GPU enabled machine. So a cloud VM or a workstation. Um, you also need a Tavi search API key. Tavi is a company that does uh search for agents. If you sign up on their website, they'll give you a thousand credits for free. And you need data designer, which is our tool for creating data. It's fully open source. Anyone can download it, use it, do whatever they want with it. And all the notebooks that I've provided um on the GitHub repo. So here's the agenda for today. So we'll first spend a few minutes talking about what is data designer and why do we actually need a framework like this and then we'll talk about uh the basics. Um there's three notebooks I'm going to actually be showing you code. U the first notebook is about the basics of data designer. How do you actually create data that you want?

然后第二本笔记本讲的是模型上下文协议，也就是 MCP。具体要如何创建智能体数据呢？第三本笔记本则是……嗯，超级……And then the second notebook is about model context protocol or MCP. How do you actually uh create agentic data? And then the third notebook which is what um is the super

为什么仅靠互联网数据还不够Why internet data is not enough

搜索的一个有趣用例。我们究竟该如何教会模型进行搜索？之后我们会留一些时间来回答问题。首先，我想引用一下 Andrej Karpathy 的话：“互联网数据并不是你想要用来训练 Transformer 的数据。”这是一个非常有道理的观点，因为互联网数据可能来自维基百科这样的高质量来源，也可能是某人在 Twitter（或 X）或其他平台上发表的观点，对吧？那么，我们究竟如何知道哪些数据适合训练 AI 模型，哪些不适合呢？我们需要对这些互联网数据进行处理和筛选，然后据此创建训练数据。这就是传统的方法。而我们提出的是，我们实际上可以合成生成数据——既可以从头开始生成，也可以基于一个种子数据集生成。interesting use case of search. How do you actually teach a model to do search and then we'll leave some time at the end for questions. So I want to start off with a quote from Andre Karpati and he said internet data is not the data you want for your transformer. So this is um a very uh valid point because internet data could be you know highquality sources like Wikipedia or it could be you know someone's opinion on Twitter or X or some other platform right so how do we actually know um what data is useful for training AI models and what data is not useful so uh we need to process this internet data curate it and then uh create training data from it so that's the traditional approach approach. What we're proposing is that we could actually synthetically generate data uh either from scratch or from a seed data set.

这就是 Data Designer 的用途。Data Designer 是一个用于创建合成数据的框架。那我说的种子数据是什么意思呢？比如，你可以使用维基百科数据作为种子，然后尝试基于维基百科创建合成训练数据。这样一来，你的合成数据就不只是模型凭空编造出来的，而是以真实世界的分布和真实世界的数据为基础。Data Designer 是一个完全开源的框架。英伟达有一支规模庞大的团队在负责这项工作。你只需运行 pip install data designer 就可以安装它。我认为我们真正引以为豪的是，Data Designer 已被用于创建 6.5 万亿个训练数据词元，而且这个数字还在每天增长。我们也在持续关注 GitHub 星标数。And so that's what data designer is for. So data designer is a framework to create synthetic data. And what do I mean by um seed data? So seed data like you could use Wikipedia data as a seed and then try to create synthetic training data from Wikipedia. So your synthetic data is not just made up by a model. It is grounded in real world uh distribution and real world data. So data design is a completely open-source framework. We have a large team at NVIDIA that works on this. Um and uh you can just install it by uh running pip install data designer. And I think um what's really um what we're really proud of is data designer has been used to create six and a half trillion tokens of um training data and this is increasing every single day. Um and yeah we're keeping track of the GitHub stars as well.

所以，如果你喜欢我们的So if you like our

超越重复提示的多样性Diversity beyond repeated prompting

给我们的仓库点个星。那么，什么是合成数据生成呢？一种生成合成数据的方法是写个循环，运行一百万次。如果你想创建一个产品评论数据集，就不断提示 LLM：“给我一个产品，给我一条评论，再给我一个客户画像。”这是一种创建数据集的方法，但如果这么做，你会遇到一堆问题。那么，缺少了什么呢？多样性、多样性，最重要的就是多样性，对吧？如果你不断提示 LLM，试个 10 次、12 次，也许 20 次之后，它就会开始重复内容，给不了你需要的东西。你还缺少数据集中各项特征之间的相关性，也没有多模态能力。而且，你实际上无法控制数据集的生成，也没有验证、核验等等。repo give us a star. So uh what is synthetic data generation right? U well one way to do synthetic data generation is take a loop run it a million times and then if you want to create a product review data set just you know continually prompt a LLM saying give me a product give me a review and then give me a customer profile that's one way to create a data set but you're going to run into a bunch of issues if you do that so what's missing um well diversity diversity diversity that's the most important thing right if you continually prompt LLM after 10, 12, maybe 20 tries, it's going to start repeating stuff. It's not going to give you what you need. Um, you're also missing correlations between all the features that you're looking for in your data set. Um, you don't have multimodality. Um, you know, you're not really able to steer the creation of that data set and there's no validation, verification, uh, and so on.

而且，如果我创建了一个数据集，我其实没法把它分享给你，因为你到底要怎么用我创建这个数据集时采用的流程，来创建一个类似的数据集呢？And if I create a data set, I can't really share it with you because how do you actually take the uh process that I use to create my data set to create something similar.

种子、采样器和生成列Seed, sampler, and generation columns

所以这就是我们创建 Data Designer 的原因。那么它是如何工作的呢？呃，Data Designer 中有许多不同的列或特征。这些列本质上是一种 Data Designer 抽象。在 Data Designer 中，我们按列生成数据。首先是 seed 列，它完全是可选的。我举过维基百科的例子，它可以作为一个 seed 数据集。呃，任何数据集都可以用作 seed，比如 CSV、JSONL 或 Parquet 文件。你也可以使用自己公司的数据作为 seed。然后，你可以尝试添加 sampler 列来引入多样性。比如说，你想创建一个编程数据集。那么，一个非常简单的 sampler 列可以包含简单、中等和困难的编程问题。这就是一个 sampler 列。你可以添加一个表示难度的列。你也可以添加 subsampler。So that's why we created data designer. So how does it work? Uh we have many different columns or features in data designer. U these columns are essentially um a data designer abstraction. In data designer, we generate data column by column. So first you have your seed column which is completely optional. So I gave the example of Wikipedia which could be a seed data set. Um we could use any data set as a seed. So anything which is a CSV, JSONL, parket file. So you could use your own company data as seed. And then you can try to uh add sampler columns to inject diversity. So let's say you are trying to create a coding data set. Well, a very simpler sampler column would be like easy, medium, and difficult coding questions. So that's a sampler column. You could add which is difficulty. You could also add subsamplers.

因此，对于简单难度，你可以将它映射到某一种特定类型的问题；对于中等难度，也可以映射到某一种特定类型的问题，依此类推。我们会深入讲解这些内容。我知道这听起来有点抽象，但等我们看代码时，就会详细介绍。嗯，然后我们还有用于生成文本、生成结构化数据和生成代码的列。呃，另外，Data Designer 现在支持多模态了，所以你也可以生成图像。嗯，你还可以验证生成结果。整个思路就是构建一个 DAG 或图，用来描述你的数据集应该是什么样的；等准备好执行时，只需点击一下，就能生成数据集。今天我们会依次讲解三个不同的 notebook。嗯，我知道你们有些人可能在想：都 2026 年了，他要给我们看的应该是代码吧？So for easy, you could map it to a specific type of problem. For medium, you could map it to a specific type of problem and so on. So we'll get into all of this. I know it feels a bit abstract, but we'll get into all of this once we look at the code. Um and then we have columns for generating text, for generating structured data, for generating code. Uh we also uh data designer is now multimodal. So you can generate images as well. Um and you can validate your generations too. So the whole idea is we build a DAG or a graph which describes how your data set should be and then this DAG once you're ready to execute on it you can just hit one click and get a data set out of it. So there's three different notebooks we'll walk through today. Um I know some of you are thinking well in 2026 he's going to be showing us code right?

我的意思是，现在有 Codex、Cloud Code 以及所有这些很棒的工具。所以我觉得，只要你理解代码，就能更轻松地用 Codex 或 Cloud Code 为自己编写合成数据工作流。接下来我们简单看一下每个 notebook。顺便说一下，这是我们的代码仓库，也就是 Data Designer 仓库，它完全开源。这是我为本次研讨会创建的 GitHub 仓库，里面有一些 notebook。你可以试试 notebook 1、2、3，我们还在这里放了演示文稿。I mean now there's codeex and cloud code and all these amazing tools. So I think if you understand the code, it'll be much easier to use codeex or cloud code to write your own uh synthetic data workflows for you. Uh so we'll just take a brief look at each notebook now. And by the way, this is our repo. So this is our data designer repo. It's fully open source. Um and this is the GitHub repo that um I have created for this workshop where we have notebooks. Um so you can try out notebook one 2 3 and then we also have um the slides here

笔记本一：问答数据Notebook one: question and answer data

如果你想跟着一起操作，或者想在研讨会结束后获取幻灯片，那我们就来看第一个 Notebook。这是我们的 Data Designer 入门 Notebook。大家都能看到吗？好。需要我放大吗？这样可以。好，首先我们先设置环境。这里有一堆我们要导入的提示词。然后我们要做的就是导入 Data Designer。就是这么简单。导入之后，你会看到 Data Designer 里有一个模型提供方的概念。我们目前使用的是 VLLM，也就是本地推理。我是在一个 Nvidia GPU 集群上运行的，然后可以使用不同类型的模型，对吧？所以在我的本地集群上，我可以使用 GPTOSS 120 billion。你可以使用任何模型，或者任何开放模型来创建合成数据。if you want to follow along or if you want access to the slides after the workshop. So let's go to the first notebook. So this is our intro to data designer notebook. Can everyone see? Okay. or should I zoom in? It's good. Okay, so first things first, we set up our environment. Um there's a bunch of uh prompts in here that we're importing. And then what we do is we just, you know, import data designer. That's it literally. Uh and once you do that, you see in data designer there's a concept of model providers. Um, so we're currently using uh VLLM, so that's local inference. So I'm running this on a on a on a Nvidia GPU cluster and then we have access to different types of models, right? So on my local cluster, I have access to GPTOSS 120 billion. You can use any model or any open model to create synthetic data.

嗯，有些闭源模型会因为许可问题，禁止你从中创建数据并用这些数据来训练模型。但开源模型通常可以随意使用。所以我用的是这个模型。这里的目标是：我能不能为这个 notebook 获取一份包含维基百科主题的数据集？比如说，维基百科有很多不同的页面和主题。于是我们列出这些主题，把它们放进一个 CSV 文件，然后基本上只要用种子数据集调用我们的配置构建器就行了。然后……Um, some closed models prevent you because of licensing issues uh from creating data from it and using it for training models. But open models are usually fair game. So I'm using this model and um the goal here is can I actually take a data set of Wikipedia topics uh for this notebook. So like you know different like uh what Wikipedia has so many different pages and um uh many different topics. So we just made a list of those put it in a CSV and then we basically just call our config builder with seed data set. And

难度采样器与分布Difficulty samplers and distributions

所以本质上，我们现在是想创建一个数据设计器工作流，不过主题是维基百科。这样做之后，你就可以添加问题难度。你可以根据数据集的需求，设为简单、中等或困难。然后还可以指定权重，比如中等难度的问题可以比简单或困难的问题多。你还可以添加各种与问题难度相关的采样器列。比如，简单题可以是直接查找、简单事实或定义；而困难题则可以要求更详细或更冷门的内容。接下来，在 LLM 生成列部分，你只要调用 add column 这个函数，然后填写提示词，大致会是这个样子。我找一下。so essentially now we're um trying to create a data designer workflow but with Wikipedia uh topics. Um so once you do that you can add question difficulty. You can say easy, medium, hard, however you want your data set to be. And then there's weights you can specify. So you can have more medium questions than um you know easy or hard. And then there's all kinds of sampler columns you can add about question difficulty. So for example for easy you might do like direct lookup or simple fact or definition right but difficult might be you want something in more detail or obscure um and then once we get to the LLM generated columns you just call this function add column and then you have your prompt which is essentially going to um look something like this. Just trying to find it.

提示模板和结构化输出Prompt templates and structured outputs

是的。所以，你是一位专业的 trivia 问题撰写者。嗯，这里有一个主题难度问题类型。你在主题中使用这些 Jinja 模板时，它会在运行时把你之前创建的主题，或者你从维基百科获取的种子数据中的主题，插入到这个提示词里。这样一来，语言模型看到的每个提示词本质上都是独一无二、各不相同的。因此，当你点击“生成”时，模型就会为你生成多样化的数据。嗯，然后你还可以使用结构化列。结构化列指的是，不是尝试生成文本数据，而是尝试生成 JSON、Markdown 格式，或者你感兴趣的任何自定义格式。嗯，我们知道大型语言模型嘛，我们得对它们友好一些，得跟它们好好交流，但有时候它们可能不会给你想要的结构化格式。Yep. So, you are an expert trivia question writer. Um, so here's a topic difficulty question type. So whenever you put these uh ginger templating for topic, what it's going to do is the topics you created in the previous um or in the seed that you had from your Wikipedia, it's going to insert those at runtime into this prompt. So essentially every prompt that the LM sees is now unique and diverse. And so the model is going to give you diverse data when you actually hit generate. Um and then um you you can also have like a structured column. So a structured column is when instead of trying to generate like text data um you can try to generate like a JSON or markdown format or any custom format that you're interested in. Um and uh we know that LLMs you know we have to be nice to them. We have to talk to them but sometimes they may not give you that structured format that you want.

所以最后我们就把那些记录丢弃了。那么，嗯，数据创建的工作方式是……So then we just end up dropping those records. So the way um data creation works is

预览几条记录Preview a few records

Data Designer 中有一种叫作预览模式的功能。基本上，点击“预览”后，它会尝试为你创建三条记录，然后你就可以查看这些记录。我们来看几条记录，这样一来，一切就会清楚很多。这里的主题是万维网。嗯，问题是，或者说，某个万维网联盟提出的问题是：哪项 W3C 推荐标准使用的三个字母缩写，与挪威某座城市的机场代码相同？我肯定不知道这个问题的答案，对吧？这就是这类问题本质上很难的原因，因为你想让模型去搜索，对吧？但在第一个……we have something called preview mode in data designer. So essentially when you hit preview what's going to happen is it's going to try to create three records for you and then you can go and look at those records. So let's look at some records. So all of this will make a lot more sense. So the topic here is the worldwide web. Um, and the question or I guess some consortium of the worldwide web and the question is which W3C recommendation is identified by the same three-letter abbreviation that serves as a airport code for some city in Norway. So there's no way I would know the answer to this question, right? That's why these questions are essentially hard because you want to invoke the model to do search, right? But in this first

评估问题质量Judge question quality

Notebook，我们只是想创建一些问答对。这是一个非常简单的数据集。所以我想，这就是模型给出的答案。然后，你还可以添加一个 LLM 作为评审，对模型的表现进行评分。这是模型生成的问题，这是它给出的答案，然后这是问题质量。我们可以让另一个模型评判它对前一个模型生成的问题有什么看法，以此类推。最后，你会得到这样一个数据集。顺便说一下，我们有一个函数，可以直接把数据集推送到 Hugging Face。它用起来非常简单，而且你还可以像这样查看所有问题。然后，你可以获取一些数据集的统计信息。显然，我们这里只生成了三条记录。notebook, we're just trying to create question answer pairs. So uh which is a very simple data set. So I guess this is the answer the model came up with. And then you can also add LLM as a judge to score how your model is doing. So this is the question the model came up with. This is the answer it came up with. And then um this is the question quality. So we can have another model judge what uh it thinks of the question that the uh previous model created um and so on. So in the end you end up with a data set like this. And by the way, we have a a function that you can just push your data sets directly to hugging face. Um so it's super easy to use and um you can also visualize all of your questions here as you can see. And then you can get some stats about your data set. Obviously we generated only three records here.

所以，呃，你知道，这些统计数据没什么太大意义，不过如果你生成数百或数千个……So uh you know these stats are not very meaningful but if you generate like hundreds or thousands of

生成一批Generate a batch

对于记录，你可以这样做。然后我们有一种叫作批处理模式的功能。也就是 data designer.create create；在这里，我可以不填 100，而是填入我需要的记录数量，然后就可以创建所有这些记录。正如我之前提到的，我们是按列生成数据的，所以在批处理模式下，你先生成问题，再生成答案，对吧？这显然是一个非常简单的示例。然后还有 LLM 作为评判者，最后是你的数据集，呈现出来就是这个样子。这就是第一个 notebook。我觉得看一遍并亲自试用之后，就会发现它非常直观。我们还有第二个 notebook，更侧重于 MCP，接下来会讲到第三个 notebook。我想大家都会觉得它很有意思，因为我们会在其中实际讨论搜索。records you could do that. And then we have something called batch mode. So which is data designer.create create and here instead of a 100 I could just you know put how many other records I need and then I can go and create all these records and as I mentioned previously we generate data column by column so in the batch mode first you generate the questions and then you generate the answers right so this is obviously a very simple example and then you have the LLM as a judge um and here's your final data set that's what it looks like so That was the first notebook. I think it's quite straightforward once you look at it and try it out. Um, we have a second notebook which focuses more on MCP and then we're going to build to the third notebook which I think everyone will find interesting because that's where we'll actually talk about search.

抱歉。Sorry.

主持人：是的。Host: Yeah.

嘉宾：哦，对。所有 GitHub 链接都一样。我只是在每一张幻灯片里都放上它们。所以如果你错过了第一个，也可以……也可以找到它。不过ેણGuest: Oh yeah. The all the GitHub links are the same. I'm just putting them in every single slide. So, if you missed the first one, you can you can get it. But

笔记本二：通过 MCP 搜索Notebook two: search through MCP

如果你想浏览一下，我会在这里停一分钟。那么，我们来聊聊 MCP。MCP 是一种标准，允许智能体连接到工具，对吧？这里的工具是 tavly，它本质上是一个供智能体使用的搜索工具。当然，也有很多其他搜索工具。嗯，我们想在这里教会模型的，基本上就是如何正确地进行搜索。因此，tavly 是一个很好的工具，可以帮助模型学习如何搜索。回到第二个笔记本。这里我们会在第一个笔记本中的一些简单概念基础上继续构建，并加入如何创建智能体数据的内容。同样，这里有导入项、模型配置，然后是相同的种子数据集和采样器列，比如问题难度、问题类型等。I'll pause here for a minute if you want to scan. So, um let's talk about MCP. So, MCP is a standard which allows um an agent to essentially connect to tools, right? So the tool here is tavly which is nothing but a search tool for agents. There are many other search tools as well. Um so that's basically the um uh what what we need the mo what we're trying to teach the model here is how do you do search right so uh tavly is a good tool for the model to use to learn how to do that search. Um so going back to the second notebook. So here what we'll do is we'll just build upon some simple concepts from the first notebook and we'll add um how to create agentic data. So again you have the imports you have the the model configs and then the same seed data set the same sampler column. So question difficulty, question type and so on.

然后你实际上创建问题，最后会得到类似一个配置，对吧？我不知道为什么它打印出来的内容颜色这么浅，不过这基本上就是你正在创建的图，然后你会得到答案。所以这完全是同一回事，只不过现在你实际创建答案时，会在这里添加这个工具别名。目标是，模型不再只是尝试回答问题，而是会使用 tavly 搜索工具在互联网上查找答案，对吧？所以我们是在第一个笔记本的基础上继续构建：当时我们只是说，好，请生成一个问题，然后请生成一个答案。现在则是：请生成一个问题，然后，嘿，你可以使用这个工具。你能用它去寻找答案，而不是尝试直接生成答案吗？And then you actually create the question and then at the end you get like a config, right? I don't know why it's printing printed this in like uh light color but yeah this is essentially your graph um that you're creating and then you have the answer. So it's the exact same thing except now when you actually create the answer you add this tool alias here. And so the goal is instead of just trying to answer the question, the model is now going to use a tavly search tool to try to find the answer on the internet. Right? So that's we're build we're building upon the first notebook where we were just like okay please generate a question and then please generate an answer. Now it's please generate a question and then hey you have this tool available. Can you use this tool to go to try to find the answer rather than trying to generate the answer?

这就是第一个 notebook 和第二个 notebook 之间的区别。然后，你还有预览模式，可以生成几条记录，再查看这些记录。顺便说一下，数据创建是一个反复迭代的过程，不是一次完成就结束了。所以，创建一些数据后，你要查看一下。如果你对结果不满意，可能就需要修改工作流、调整提示词，然后不断迭代。So that's that's the difference between the first and the second notebook. And then again you have the preview mode where you can generate a few records and then you look at your records. And by the way data creation is very iterative. It's not like one and done. So when you create some data you look at it. If you're not happy with it, then probably you go change your workflow, change your prompts and then you know iterate on it

检查搜索轨迹Inspect a search trajectory

然后等你对它满意之后，就可以把它交给批量执行。我们来看看第二个 notebook 生成了什么。假设问题是：Teleifan 于 1949 年将总部迁到了德国的哪座城市？有人知道答案吗？不知道，对吧？我是说，我也不知道答案。所以我们想要提出一些比较难的问题，以便让模型进行推理并调用工具。当模型实际尝试回答这个问题时，正如你在这里看到的，它使用了 Tavily。它进行的第一个搜索是“tell if you can moved its headquarters in 1949 to which company”。本质上，它在这里使用了一个搜索工具，然后就会得到答案。这是模型发出的查询，下面是搜索结果。这些是所有的网址，每个网址都会显示标题，还会提供内容。and then once you're happy with it, you can send it off to batch execution. So let's look at what um the second notebook produced. So let's say the question is uh to which German city did Teleifan move its headquarters in 1949. Does anyone know the answer? No. Right? I mean I don't know the answer. So um what we want to do is create questions that are difficult so it invokes reasoning and tool calling from the model and so when the model is actually trying to respond to this um it's as you can see here it's using tavly so the first search it made is tell if you can moved its headquarters in 1949 to which company so essentially it's using one search tool called here and then um it's going to get an answer so this is the query from the model and then here are the results. These are all the URLs and then for each URL it's going to give you a title and then it's also giving you the content.

所以，如果你发起查询，这就是 Tavali 给我们的结果；或者说，这是智能体发起查询时 Tavali 给它的结果。然后你会得到响应，最后还可以像我们上次那样，用批处理模式创建数据集。如果你在想，他刚才是不是飞快地讲完了前两个 Notebook，那就是……So that's what Tavali gives us right if you make a query or that's what it gives the agent when the agent makes the query and then you end up with the response and then finally at the end you can create your data set with batch mode just like we did last time. And if you're thinking he just blazed through this whole first two notebooks, that's

为什么模型需要搜索功能Why models need search

因为关键内容在第三个笔记本里，对吧？嗯，这就是我今天真正想让大家重点关注的内容。接下来我们来聊聊搜索。嗯，我们刚才看了几个笔记本，了解如何使用 Data Designer，这当然没问题，但我们实际上要解决的是什么问题？嗯，就像我之前提到的，我是 NVIDIA 的一名研究科学家，也是 Nemotron 团队的一员。Nemotron 是我们打造的一系列开源基础模型。我们刚刚发布了最新模型 Ultra，此外还有另外两个模型，所以我们一共有三种不同规模的模型。在这项工作中，我们实际上必须教会 Nemotron 如何进行搜索。这非常重要，对吧？也许现在看来已经是很久以前的事了，但大概两三年前，我相信大家都记得，当时打开 ChatGPT、Claude 或 Gemini 之类的产品，再看它们的网页界面，上面会写着知识截止日期是 2023 年 12 月。because the crux of it is in the third notebook, right? So, um that's what I really want us to focus on today. So, let's talk about search. Um cuz we just went through a couple of notebooks showing us how to use data designer and that's fine, but what's the problem we're actually trying to solve? So, um like I mentioned before, I'm a research scientist at NVIDIA and we're also part of the Neotron team. So, Neimotron is a collection of found open-source foundation models um that we've created. Uh we just released our latest model Ultra, but we also have two other models. So, we have three different sizes of models. And so, as part of that work, we actually had to teach Neotron how to do search. And that's very important, right? Because maybe this feels like a long time ago now, but maybe two or three years ago, I'm sure you all remember looking at like I don't know chat GPT Claude or Gemini and then looking at the web UI and it says the knowledge cutoff date is December 2023.

所以它之所以这么说，是因为你总得在某个时候开始训练模型。也就是说，你总得在某个时候停止收集和创建数据，对吧？假设他们在 2024 年 1 月训练模型，那么到 2023 年 12 月，他们基本上就得冻结数据。所以，模型在 2023 年 12 月之后了解到的任何信息，都不属于它的训练数据。也就是说，如果你问它一个问题，比如“今天的天气怎么样？”，它就不可能根据训练数据回答这个问题，对吧？呃，问题。So the reason it said that is because well you have to start training the model at some point. So you have to stop collecting and creating data at some point, right? So let's say they're training the model in January 24. by December 23, they need to like um essentially freeze the data. So, anything that the model knows after um December 23 is not part of its training data. So, if you had to ask a question like what is the weather today, there's no way the model can answer that question from its training data, right? Uh question.

主持人：对。所以，没错，一定要扫描二维码。我们的想法是，究竟该如何教会模型了解时事？因为了解时事似乎和了解过去的事情同样重要，甚至更重要，对吧？所以，这就是搜索功能存在的原因，也是模型需要学习如何进行搜索的原因。回到这张幻灯片，教会 Neotron 模型如何使用搜索工具，就是我们的初衷。模型需要获取实时信息，才能发挥实际作用……Host: Yeah. So yeah, definitely scan the QR code. So the idea is how do you actually um teach the model about current events because learning about current events seems as important if not more important than you know knowing about uh things from the past, right? So um that's why search exists and that's why models need to learn how to do search. Um so coming back to the slide um so teaching neotron models how to use search tools that was our motivation and um um we essentially the model needs access to real-time information to be able to do useful

BrowseComp 与高难度问题BrowseComp and difficult questions

这些事情。那么我们实际上是怎么做到的呢？在讲我们是怎么做到的之前，我们需要有东西来衡量进展，对吧？所以，在基础模型领域，当你需要衡量某方面的进展时，就会使用基准测试，也就是一套标准化的测试题，但没有答案——也就是说，我们并不知道答案。搜索领域的一个基准测试是 BrowseComp，这是一个非常难的基准测试。现在 FIFA 世界杯正在举行，我不知道这里有多少足球迷，但我可是个铁杆球迷。不过我们不叫它足球，而是……1990 年到 1994 年之间的四场比赛中，哪些球队参加过一场由巴西裁判执法的足球比赛？比赛中共有四张黄牌，两队各两张，总共三张，诸如此类。我就不把问题的其余部分读完了，但这问题确实相当复杂，对吧？things. So how did we actually do this? So before I get to how did we do this? We need something to measure progress on. Right? So um in the foundation model community when you need to measure progress on something you use a benchmark which is like a standardized test set without the answers right like we don't have the answers so one of the benchmarks for search is browse comp uh which is a really difficult benchmark and uh the FIFA World Cup is going on right now so I don't know how many soccer fans there are here but I'm I'm a big soccer fan but we don't call it soccer but um between 1990 and 1994 four. Which teams played in a soccer match with a Brazilian referee? Had four yellow cards, two for each team, three in total, blah blah blah. I won't read the rest of the question, but it's it's quite complicated question, right?

我们当中没人知道这个问题的答案，或者说，也许有人特别精通这方面，答案就在他们脑子里。以前，比如三四年前，遇到这样复杂的查询时，我们会试着把问题拆解开来，然后推理该实际发起哪些查询。我们会去 Google 或其他地方搜索，获取更多信息，然后思考我们最初的目标是什么、我们获取了哪些信息，再从那里继续，最后得出答案。对吧？所以答案是爱尔兰对罗马尼亚。我之前不知道这一点。There's no way that any of us know the answer to that or maybe there's like, you know, someone who's um really wellversed in this who has the answer at the top of their head. What we would do back in the day like you know three four years ago is we'd actually try to break a problem down if we had a complicated uh query like this and then we would reason about what queries to actually make. We would go and search on Google or elsewhere uh get access to more information and then reason about what was our original goal, what information did we get access to and then proceed from there and then finally arrive at the answer. Right? So the answer is Ireland versus Romania. I I didn't know that.

将维基数据路径用作搜索谜题的种子Wikidata paths as search riddle seeds

那么，我们究竟是如何创建一个数据集，让我们能够训练模型，使它在 BrowseCom 基准测试上的成绩从 0% 提升到 30% 的呢？这就是我们今天要了解的内容。为此，我们做了一件事：我们使用了一个我之前介绍过的种子数据集，只不过我们的种子数据集不只是维基百科，而是一个本质上由维基百科主题构成的图谱，它叫作 Wikidata。我们沿着这个图谱走了几跳，这就构成了我们的种子数据集。我知道右边的示意图看起来像棵树，但它不是树，而是一张图。所以，如果它看起来像棵树，那是我的问题。呃，它应该……应该是一张图。基本上，你可以看到，右边的第一个主题是 Nvidia，Nvidia 与 Jensen 相连，他是我们的 CEO；接着，Jensen 又与俄勒冈州立大学相连，因为 Wikidata 说他在那里上过大学；然后，这所大学位于本顿县，而本顿县实际上是以一位名叫 Thomas Hart Benton 的人命名的。Um so how did we actually create a data set that allows us to train a model to go from 0% to 30% on this browsecom benchmark? Um so that's what we're going to learn about today. So to do that we took we did something um uh we we used a seed data set which I described earlier except our seed data set is not just Wikipedia it's something that um is essentially a graph of Wikipedia topics so it's called wiki data and what we did is we did a few hops along this graph and that is our seed data set and I know the diagram on the right looks like a tree it's not a tree it's a graph So um if it looks like a tree that's my fault. Uh it should it should it should be a graph. Um and essentially um you can see um on the right the first topic is Nvidia and then Nvidia is connected to Jensen who is our CEO and then that is connected to Oregon State University because Wiki Wiki data says that that's where he went to college and then that college is located in Benton County and Benton County is loca is actually named after a person called Thomas Hart Benton.

所以，如果你从维基数据中随机抽取一个实体，然后沿着它走五到八跳——我稍后会解释为什么是五到八跳——最终就会得到像这样的路径，也就是这组绿色实体，实体之间用箭头连接。这基本上就是我们用来创建搜索谜题的种子数据集。比如，我可以据此创建这样一道搜索谜题；我得看一下演讲者备注才能读出来：哪家公司是由这样一个人创立的——这个人曾就读于一所大学，而这所大学所在的县是以托马斯·哈特·本顿的名字命名的？LLM 不可能在训练数据里知道答案，对吧？因为如果给智能体作弊的机会，它就会作弊，直接把答案告诉我们。So if you took um if you randomly sampled an entity in wiki data and then did five to eight hops along it and I'll explain why five to eight you'll end up um with a hop like this like the green you know set of entities with like arrows in between. So that is essentially our seed data set to create uh search riddles. So, one search riddle that I could create from this, and I'm going to have to read this from my speaker notes, is um which company was founded by the person who went to college where the county was named after Thomas Hart Benton. So, there's no way the LLM knows the answer to that in the training data, right? Because if we give an agent the chance to cheat, it's going to cheat. It's going to just give us the answer.

它不会使用我们希望它使用的工具调用或工具，抱歉，比如搜索，对吧？所以我们希望尽可能把问题设计得难一些，也尽可能像谜语，这样就能触发模型的搜索行为。这就是我们的种子。我刚才讲了第一步：用种子创建随机游走。为什么是四到八跳，或者五到八跳？有时候你这么跳几步之后，内容就会完全跑题。比如它会说，呃，我不知道，某个人的名字，然后又说，嗯，人类。好吧，这当然没错，但你知道，这其实不太相关。基本上，跳转过程会跑题，所以我们想用一个启发式方法，把跳数控制在四到八之间；对我来说，这个范围效果不错。然后，你把这些种子转换成搜索谜题，再通过搜索工具执行搜索，工具叫 Tavly。这样就能用一个提示词生成一个数据集，其中包含搜索提示和搜索响应，对吧？it's not going to use the tool calls or the tools sorry like search that we want it to use right so we want to make the questions as difficult and as I guess um riddle like as possible so we can invoke the search behavior from the model so this is our seed so I I spoke about the first step which is create random walks with seed and why four to eight or five to eight hops sometimes it just you know if you do these hops it just goes completely off topic like It will say um I don't know like u a person's name and then it will say well human being well sure that is true but you know it's not really related like basically your hops will go off topic so that's why we want to as a heristic stay between four and eight that's what worked well for me and then you transform these seeds into search riddles and then um you execute the search via the search tool which is tavly so that's how you create a data set with a prompt and a with a search prompt and a search response, right?

然后我们评估数据。接下来，还有一点这里没有提到：我们会训练模型，对吧？因此，拿到数据集后，我们就用它来训练模型，然后在标准化基准测试上衡量模型的性能。And then we evaluate our data and then something that's not mentioned here is, you know, we train our models, right? So once you have the data set, you train the model with it and then you measure performance on a standardized benchmark.

笔记本三：端到端搜索数据Notebook three: end to end search data

那么我们来看看笔记本，看看怎么做。到目前为止有什么问题吗？有。之所以要这么做，呃，我想并不是因为问题很难，而是因为模型无法轻易在网上找到这些问题，也无法轻易在网上找到答案。比如说，如果你把我们在 BrowseComp 基准测试中用到的那一大段文本放到网上，你是找不到答案的。我的意思是，现在有了 Gemini 的 AI 搜索功能，你确实能找到答案，但我们本质上想教会模型的就是这个。对。好。那么，这就是端到端智能体搜索的教程。我们已经讨论过实现方法和如何创建数据，也讨论过如何导入这些提示词。我还写了一个函数，用 Tavily 配合 MCP。So let's look at the notebook and see how we can do this. Any questions so far? Yes. So the reason uh it was important to do I guess it's not difficult questions it's questions that the model uh cannot find on the internet readily uh cannot find answers readily on the internet too. So like if you took that full blob of text that we had in the browsecom benchmark and you put that on the web you're not going to get the answer. I mean you will get the answer because of the AI search thing with Gemini now but that's what we're trying to teach the model essentially. Yeah. Okay. So um so this is the tutorial for the end to end aentic search. So we already spoke about the way we do this and how we create the data. We already spoke about how we import all these um prompts and I wrote a function to like use Tavi with MCP.

你之后可以去看看代码。嗯，这里是我们的种子实体。这就是我给你看的那个例子。这只是一个种子实体。我的种子收集算法的工作方式是，你可以运行它一百万次，然后沿着维基数据得到一百万种不同的跳转，对吧？这些只是我们想用来构建搜索谜题的主题。另一个主题是，嗯，我们喜欢现代艺术。我其实不是什么艺术爱好者，所以这些词大多我都不懂，但基本上这就像是另一个种子跳转，你可以……You can go and look at the code later. Um and here is where we have our seed entities. So this is the example I showed you. This is just one seed entity. So the way uh my seed collection algorithm works is like you can run it a million times and you get million different hops along wiki data right so those are just topics we want to ground our search riddle creation on so another topic is um we like modern art and like I'm not really an art buff but so I don't understand most of these words but essentially this is like another seed hop that you can

工具说明和调用示例Tool instructions and example calls

所以，创建这个数据集的方法是这样的：嗯，你是一名专家级搜索代理，会使用网络搜索来准确回答问题，并提供有效的 JSON，其中包含最终答案、支持性 URL 和简短的理由。以下是你可以使用的工具。你有一个工具，就是 tavly 搜索，接下来是这个工具的一些使用指南。比如说，我们会说：“好吧，请、请不要用太多次，比如说，别调用这个工具一百次，然后花个一千美元之类的。”我想，你可以在里面加入各种各样的指令，但归根结底，嗯，开个玩笑，真正想要的是：你的代理不应该失控，为了回答一个很简单的问题就花太多钱，或者做类似的事。So the way to create this data set is um well you are an expert search agent that uses web search to answer questions accurately provide valid JSON with the final answer supporting URLs and a short justification and here are the tools you have available. You have one tool which is tavly search and then here's some usage guidelines for the tool. So, like we're like, "Okay, please please don't use more than like, you know, don't don't like call the tool like a hundred times and spend, you know, $1,000 or something." I guess those are like all kinds of instructions you can put in there, but essentially um uh jokes aside, what you want here is um your agent shouldn't go crazy and try to like you know spend way too much money answering a question that's very easy or you know something like that.

所以这就是为什么我们有这些指令，同时也有一个示例。这有点像上下文学习，对吧？你把一个示例放进提示词里，模型基本上就能从这个示例中学会如何调用搜索工具。接下来是你想要创建的提示词。这里我们收集了维基数据中的跳数。我们可以把这些跳数插入提示词中，这些就是我们的种子实体。然后我们让模型根据这些跳数中的部分或全部种子实体来提出一个问题。接着，我们让模型使用我之前展示的这个智能体解决方案图来生成回答，也就是在第二个笔记本里展示的那个。所以——So that's why we have all these instructions and we also have an example. So this is um um something like in context learning, right? So you put an example into the prompt and then the model can essentially um learn from that example how to actually call the search tool. So here's your prompt that you want to create. So here what we do is um we collected the wiki data hops. We can insert those hops into the prompt here uh which is our um you know seed entity. And then uh we ask the model to essentially come up with a question um using any or all of the seed entities from this hop. And then we ask the model to create a response using uh this agent solution draw which I showed earlier. So in the in the second notebook. So

检查生成的谜题和追踪记录Inspect the generated riddles and traces

完成这一步后，你会得到一个预览数据集。这就是我们的预览数据集。种子实体的起始节点是印度医学研究委员会。然后，它似乎位于一座名为布巴内什瓦尔的城市，这座城市是印度某个邦的首府。这就是我们的种子路径。接下来是模型生成的问题。这里没有显示这个 Asian solution draw，不过你看这里就能看到它是什么样的。它调用了不少工具，分析了这个问题，然后试图在互联网上找到答案。这就是创建一个名为 data set 的搜索工具的方式。我想先在这里停下来回答问题，然后我们再继续看接下来的几张幻灯片。有什么问题吗？如果有的话……好。once you do this, you'll end up with a preview data set. And so here's our preview data set. So Indian Council of Medical Research is the start entity for the seed. And then apparently that's located in a city called Bhuaneshwar which is a capital of some state in India. Uh so that's our seed path. And then here is the question that the model came up with. And then this Asian solution draw is not displayed in here. But if you look at it, we can we can see what it looks like here. So it did a bunch of tool calls. It reasoned about the question and then it's trying to find the answer on the internet. So that's how you create um a search tool called data set. And I guess I'll pause here for questions and then we can move on to the next few slides after that. Any questions? If something was Yes.

是啊，Yeah,

嘉宾：当然。Guest: absolutely.

主持人：是的。所以我们把鸟类基准的成绩从 26% 提高到了 41%。你问的确实是个非常精准的问题，我们也为此写了一篇博客。你可以去看看。我们用 Data Designer 创建的所有数据集都有对应的博客文章，你可以在这里看到。我也参与了这项工作。是的，这篇文章讲的是 SQL，我之后也可以把链接分享给你。总的来说，Data Designer 可以用来创建各种用于训练 AI 模型的数据集，也可以创建其他类型的数据集，比如与模型训练无关的测试数据集等等。除了 Data Designer，我们团队还开发了不少数据工具，我会在最后介绍。Host: Yes. So we moved the bird benchmark from 26 to 41%. So you asked the you know exact right question and we have a blog post on this as well. Um you can read about it. Um so all the data sets we've created with data designer um we have blog posts and you can see right here so I was involved in this work as well. Uh so yeah this talks about SQL and I can also share the link with you after um but essentially data designer can be used to create any kind of data set for training AI models possibly other data sets as well if you're looking for test data sets not related to model training and so on. Um and we have a bunch of data tools that we've created in our team apart from data designer that I'll talk about in the end.

嘉宾：所以，SODA 在 Bird 上得分是 70，是因为它具有智能体能力。我们目前的 SQL 数据集不是智能体型的。另外，基准测试中的排序结果很大程度上取决于模型。我们说的是要迁移 Neotron Nano，也就是我们的小尺寸 LLM 或 SLM。好吧，我不知道能不能称它为 SLM，因为它是 MOE，但它确实是 LLM。对于这种模型规模来说，它在板载测试中的表现相当不错。对。问题。Guest: So the soda is 70 on bird is because it's agentic. Our um current SQL data set is not agentic. And um also a sort on benchmarks is very much model dependent. We're talking about moving uh Neotron Nano which is our small size um LLM or SLM. Okay. I don't I don't know if you can call it an SLM because it's an MOE but it's an LLM. And so um for that model size it's a pretty respectable performance on board. Yeah. question.

种子生成与智能体轨迹对比Seed generation versus agent trajectories

主持人：嗯，不，这是一种确定性算法，用来创建种子数据；工具调用和智能体相关的部分，则是在你拿到问题后执行实际搜索并生成轨迹。我们称之为轨迹。所以，创建搜索轨迹这一步是确定性的。这只是我想出来的一种创建种子数据的方法。是的，你可以使用任何类型的种子数据。也许你还能想出比这更好的种子数据创建方法。Host: Yeah. No, this is a deterministic algorithm to create the seed data and then the uh tool calls and the agentic stuff is in creating the actual like performing the actual search once you have the question and creating the trajectory. We call it a trajectory. So creating the search trajectory this is deterministic. This is just something I came up with to create seed data. Yeah, you can have any kind of seed data. Maybe like you know you can come up with a better way of creating seed data than this as well.

嘉宾：哦，这个问题问得真好。我们可以这么做。Guest: Oh great question. So we could do that.

监督微调与蒸馏Supervised finetuning and distillation

嗯，我们创建了一些可用于预训练和后训练的数据集。我的主要关注点是后训练，嗯，或者说为后训练创建数据集。所以这个具体的项目，我想是后训练中的 SFT。是的，Um we have created data sets that can be used in pre-training and posttrainining. My main focus is posttraining um or you know creating data sets for post- training. So this specific uh um I guess project was SFT uh in posting. Yes,

主持人：除了我觉得英伟达有人和 Tavly 达成了协议，拿到了额度之外，没有别的原因。你可以用 Exa。Exa 是另一家搜索代理服务提供商。你也可以自己编写搜索工具。嗯，我想是吧。对，还有这一点。Host: there's no reason apart from the fact that uh I think someone at Nvidia did a deal with Tavly to get credits. You can use Exa. Exa is another search agent uh provider. You can write your own search tool. Um I guess so. Yeah, there's that as well.

嘉宾：是的，基本上是这样。Guest: Yeah. essentially.

主持人：所以，黄金响应是由 Data Designer 使用你选择的模型生成的。嗯，在第一个笔记本中，嗯，是的，抱歉，我想我讲得有点快，因为这次研讨会只有一个小时。呃，这次研讨会的上一版是两个小时。所以我用来创建数据的模型是 GPDOSS120B。嗯，但 Data Designer 不只是一个可用于创建数据的 LLM 集合。它还包含许多 LLM 和非 LLM 模型，它们协同工作，帮助你创建多样化的数据。Host: So the golden responses are generated by um tail designer using the model that you choose and so um in the first notebook um yeah and sorry I think I think um you know I went a bit quick because it's we just have an hour for this workshop uh previous version of this workshop was 2 hours. So the model that I used is GPDOSS120B to create the data. Um but data designer is not just a collection of LLMs that you can use to create data. It has a bunch of LLM and nonLLM models as well working together to help you create diverse data.

客人：再说一遍。Guest: Say that again.

主持人：这是 SC。是的。呃，你是什么意思？对。Host: This is SC. Yeah. Uh what do you mean? Yes.

嘉宾：是的。Guest: Yeah.

主持人：是的。Host: Yeah.

客人：是的。Guest: Yes.

主持人：是的，你在做蒸馏。但在某些情况下，你甚至可以超越蒸馏。这正是我想要论证的，因为我们构建了一个工具，让你能够创建多样化的数据。比如，我之前做过一个完全不同的用例。GPTOSS 在一个名为 strructal 的基准测试中取得了一定的分数，这个基准用于衡量模型在 JSON、Markdown、XML、YAML 和 CSV 格式上的表现。而 Neotron 实际上是从 GPTOSS 蒸馏而来的，但表现却超越了 GPTOSS。所以这既是蒸馏，也可以更强大，因为我们在 Data Designer 中构建了各种功能。是的。Host: Yes, you are distilling. But in some cases, you can even go beyond distillation. And that's the case that I'm trying to make because we have built this uh tool that allows you to create diverse data. So for example, a completely different use case that I was working on. GPTOSS gets some percentage on this uh benchmark called strructal which is a way to measure how models do on JSON, markdown, XML, YAML and CSV formats. And Neotron actually distilled from GPTOSS but went beyond GPOSS's performance. So it's distillation but it can also be more powerful than that because of all the things we've built into data designer. Yes.

强化学习替代方案Reinforcement learning alternatives

主持人：我该如何使用数据设计器？Host: How can I use data designer?

嘉宾：你可以使用数据设计器进行强化学习。Guest: You can use data designer for RL.

主持人：那么，举个例子，顺便说一下，我们现在是在为搜索做这件事，或者说 NVIDIA 的团队正在为搜索做这件事。所以，与其像他描述的那样尝试创建标准答案，或者照着我幻灯片上写的内容来解答搜索谜题，你只要跳过这一步，最后基本上就会得到一个包含大量提示的数据集。然后我们有一个叫 Nemo Gym 的工具，它本质上是一个 RL 奖励函数。我想，它是一个教你的模型如何进行 RL 的工具。但问题是，在 Nemo Gym 里，你要编写一个函数来给出奖励，然后就可以使用我们的任意 RL 训练框架，通过这些工具实时执行，并在 RL 中提供奖励。Host: So let's say uh and by the way we're doing this for search right now or or the team at NVIDIA is doing this for search. So instead of trying to create the golden solution like how he described it or or you know um executing the search riddles as written on my slide, you just don't do that step and then you essentially end up with a data set full of prompts and then we have a tool called Nemo Gym which is essentially um an RL um reward function. You know I guess it's an it's a it's a tool to basically teach your models how to do RL. But the thing is in Nemo gym you write like a a function that essentially gives you a reward and then you can use our uh any RL training framework to execute in real time with those tools and then provide a reward in RL.

嘉宾：这是个很好的观点。我们目前确实正在思考该怎么做。不过，要构建一个通用的东西才是挑战，对吧？我们不想只针对某一个用例构建东西。是的，是的。它用于 ETL 可靠吗？我自己还没试过，但我觉得应该可靠，因为我们用 data designer 创建了各种各样的数据集。我记得一开始我们展示了大约 6.5 万亿个 token。如果它不可靠，你们应该来找我们聊聊，看看我们能怎么帮你们。Guest: It's a great point. We're thinking about how to do this actually currently. But to build something generic is the is the challenge, right? Like we don't want to build something specific to one use case. Yeah. Yeah. Is this reliable and accurate for ETL? So I haven't tried it myself but I think it is because it it should be because um we've created a diverse set of data sets with data designer and I think at the start we showed about 6 and a half trillion tokens and if it's not reliable you should come talk to us and see how we can help you.

主持人：我听不见你说话。抱歉，抱歉，我听不到。Host: I can't hear you. Sorry. Sorry, I'm not able to hear.

嘉宾：哦，我我不太了解那个。最后我再回来找你，我们可以再再聊聊。嗯，抱歉……Guest: Oh, I I don't know about that. I I'll come back to you at the end and we can we can chat more. Yeah, sorry about

筛选并评估合成数据Filter and evaluate synthetic data

好的。嗯，我们还有大约 15 分钟。接下来我想谈谈一个非常有意思的话题：如何评估你的合成数据。评估并不容易，因为如果没有这一步，合成数据就没那么有用。 先简单介绍一下我们所说的 SDG，也就是合成数据生成，是如何运作的。我最初用了 50,000 条种子数据，最后只留下了 7,000 条高质量的端到端 SFT 记录——咳——然后用这些数据训练模型，把基准测试成绩从 0 提高到了 30。这很不错，但从 50,000 条到 7,000 条，中间需要经过大量筛选和过滤，才能挑出高质量数据。 所以我想在这里强调的是：创建数据时，数据质量是最重要的。那么，我说的质量具体指什么呢？that. Okay. Um, we have about 15 more minutes. So let me move on and we can have more questions at the end to something really interesting which is evaluating your synthetic data right and that's not an easy thing to do because without this evaluation step the synthetic data is not that useful. So when I started just to give you a sense of how SDG as we call it works synthetic data generation I started with 50,000 seeds and in the end I ended [clears throat] up only with 7,000 high quality endtoend SFT records and then I used that to train the model and we moved the benchmark from 0 to 30 right so that's great but 50,000 to 7,000 that's a lot of you know um um funneling and filter ing for high quality. So what like I'm trying to focus on here is that data quality is the most important thing when you're trying to create data and um what do I mean by quality?

它可以是特定于任务的，但也可以是通用的，比如我想说，多样性是质量的一个维度。嗯，之后我创建了这个函数，来帮助我们展示：LLM 作为评审对每条轨迹的评分是多少；你的问题或搜索查询有多少个词；以及这里是你的保留百分比，66%，等等。所以，真正……你之后可以看看这个函数，不过真正……It can be task specific but it can also be generic like diversity is one way to um you know one axis of quality I would say. Um so after that I created this function to help us um show like this is what the LLM as a judge scores for each trajectory um and what is like your um you know number of words in your uh question or search query and then here is your keep percentage 66% and so on. So what's really so you can take a look at that function later but what's really

过时的来源与答案验证Stale sources and answer verification

有意思的是，你实际上如何检查模型得出的答案是否正确。我遇到了一个非常有意思的例子，这个搜索谜题相当简单。它问的是：“哪家公司？哪个国家拥有美国钢铁公司？”我以为答案是美国。所以，嗯，我想，智能体在训练数据集中预测的答案是美国，我的验证器也说答案正确。但后来我去 Google 搜了一下，发现美国似乎几年前把美国钢铁公司卖给了日本——如果我没记错的话。所以，连我的维基数据种子也可能过时，对吧？这才是真正的挑战：你究竟该如何评估，并判断自己创建的数据是否优质、准确、相关，等等。嗯，我想，如果你去实现一个确定性的——比如像我那样的简单匹配函数，最后得到的结果大概只有 24%，然后你可能就会想：“我的数据太糟了，我为什么要费这么大劲？”但事实并非完全如此，因为这个比例低是我预料之中的；我其实不知道该如何创建那个函数。interesting is how do you actually check if the answers that the model got are correct or not. So I came across this really interesting example and the search riddle was quite simple. It said which comp which country owns US steel and I thought the answer is the US. And so um I you know I I guess the agent predicted US in the training data set and um my verifier said that it was correct but then I went and googled it and apparently the US sold US steel to Japan I think if I remember correctly a couple of years ago. So even my wiki data seed can become stale, right? And so that's the real challenge is how do you actually evaluate and um uh tell if the data that you've created is um good, accurate, you know, relevant and so on. Um so I guess if you go and implement a determinist I guess like a simple matching function like how I did you'll end up with like 24% and then you'll be like well my data is terrible why did I do all of this but that's not entirely true because here low percentage I expected because I don't know how to actually create that function.

所以，这是一个正在积极研究的领域：究竟该如何评估这些数据？在我看来，评估是一个多方面的问题。你不能只是把 LLM 当作评判者，然后指望它能给你一切。评估面临这样一个问题：如果你检查了 100 个问题，并发现这 100 个问题都不存在，那么你能得出的结论也仅仅是，数据集中不存在这 100 个问题。数据里可能还存在其他一千个问题。这就是挑战所在：究竟该如何评估这些数据？嗯，我们有几种不同的方法。嗯，其中一种就是我刚才提到的，把 LLM 当作评判者，这是一种通用方法。呃。So this is an active area of research is how do you actually evaluate this data? So I think evaluation from my perspective is a very multifaceted problem. So you can't just take LLM as a judge and put it in there and hope that that's going to you know give you everything. Evaluation is a problem where if you check for a 100 problems and if you see that those 100 problems do all you all you can tell from checking for a 100 problems is those 100 issues don't exist in your data set. there could be a thousand other issues. So that's the challenge is how do you actually evaluate the data? Um so we have a few different ways that we do that. Um one of them is like I mentioned earlier LLM as a judge which is generic. Uh

人工审核模型判断Human review of model judgments

我做过一件很有意思的事：我实际上采用了 LLM 作为评判者，然后把它的判断结果发给人工审核人员，让他们交叉核对 LLM 的判断，从而让我有信心相信 LLM 正确地评估了我的回答。这是一种做法。然后，在搜索中，你可以查看实时信息。所以基本上，你必须把这些不同评估层的结果整合起来，才能对你的数据集建立足够的信心。代码部分就到这里。我知道刚才讲得很快，但希望我已经在 GitHub 仓库里开放了所有这些内容，大家可以去看看。接下来我们聊几个其他用例。就像我之前提到的，我们是打造 Neotron 的一个更大团队中的一员，对吧？something I did really interesting is I actually did LLM as a judge with uh and I took the judgments and I sent them to human reviewers and I have those reviewers um cross-check the judgments of the LLM and essentially give me confidence that the LLM is judging my responses correctly. So that's one way to do it. And then in search you could look at real-time information. So you have to basically find a union of all these different different uh layers of evaluation that's going to essentially give you enough confidence in your data set. So that's it for the code. I know I blazed through this but hopefully I've you know um given access to all of this in my GitHub repo so you all can take a look. Um so let's talk about a few other use cases. So like I mentioned earlier, we are the um you know part of a much larger team that creates Neotron, right?

所以，嗯，我的重点，或者说我们团队的重点，基本上是创建数据集，或者创建一个用于创建数据集的框架，或者我想两者都包括。我们已经创建了各种各样的数据集。今天我们聊到的第一个是搜索工具调用。嗯，我们还有一个用于多轮工具调用的数据集。所以，我觉得搜索那个很大程度上是单轮的，对吧？就像是从提示到回复。但如果有多轮呢？或者如果有多个工具呢？嗯，你知道，不只是搜索工具，也可能有天气工具、编程工具之类的。嗯，所以我们这里正在做好几个不同的项目。另一个特别有意思的是结构化输出生成。我之前简单提到过。有一个叫 strructal 的基准测试，用来衡量你在 TOML、JSON、YAML、CSV、XML 等格式上的表现。这些都非常重要，因为每个人都想用，嗯，你知道，用智能体来做所有事情。So um my focus or my team's focus is essentially creating data sets or creating a framework for creating data sets and or I guess both of those. So there's all kinds of data sets we've created. The first one that we spoke about today, search tool calling. Um and then we also have a data set to do multi-turn tool calls. So, I think the search one was very much single turn, right? Like we're going from prompt to response. But what if you had multiple turns or what if you had multiple tools um or you know not just search tools but like you might have a I don't know a weather tool or a coding tool and so on. Um so we're working on multiple different projects here. Another really interesting one is structured output generation. I briefly spoke about it. There's a benchmark called strructal which measures your performance on toml, json, yaml, csv, xml and all of these are super important because everyone wants to use um you know agents to do everything.

现在，我相信大家都在 LinkedIn 或其他地方看过那个呃喷灯梗图，不过，如果我们要把智能体作为软件应用的核心组件，那么它们就需要可靠地为我们提供 JSON 格式或 YAML 格式之类的内容。所以，这就是我们在这方面的重点。嗯，我们还研究了 SQL——有人刚才也问到了这个——以及科学推理。嗯，我们还利用数据创建了一些预训练数据集。Now, I'm sure you all have seen the the uh the blowtorrch meme uh on LinkedIn or something, but but uh if we're going to be using agents as um the central piece in software applications, then they need to reliably give us um like a JSON format or a YAML format and so on. So, that's what we were focused on there. Um we've also worked on SQL um that someone asked about here and uh we've also worked on scientific reasoning. Um there's also pre-training data sets that we've created using data

数据生成实践及其他任务Data generation practices and other tasks

负责 STEM 性能的设计师等等。嗯，所以我可以谈谈生成数据的最佳实践。嗯，我不会详细讲这个。因为你之后随时可以查看幻灯片，但核心原则是：多样性优先于数据量。我觉得 Kim K2 论文展示了类似的结果：10 种策略各训练一个 epoch，比一种策略训练一个 epoch 更好——说错了，是比一种策略训练 10 个 epoch 更好。嗯，你总是应该使用成本最低且合适的列类型。比如，用 LLM 创建采样器就毫无意义，对吧？这完全是在浪费资源。但用 LLM 生成文本数据就很有意义。所以我们的采样器当然是用 Python 编写的。嗯，然后你要生成过量的数据，再像我之前提到的那样进行筛选，然后尽可能验证，并基于预览不断迭代。designer for stem performance and so on. Um and so I can talk about the best practices for generating data. Um I won't go over this in detail. There's because you can always look at the slides later but essentially diversity before volume. I think the Kim K2 paper showed something similar which is 10 strategies one epoch is better than one epoch. Sorry 10 strategies one epoch is better than 10 epochs one strategy. And um you always want to use the cheapest correct column type. So like using a LLM to create a sampler makes no sense, right? It's a complete waste of resources. But using LLM to generate text data makes a lot of sense. So our samplers are written in Python of course. Um and then you generate you overproduce data and then you filter down like I mentioned earlier and then you verify what you can uh iterate on the preview.

我认为，判断你的数据好不好，最好的办法就是实际看一看数据。这是最好的办法。呃，如果你能查看并审查一些样本，当然，还有“数据设计师”这个概念到底是什么意思？本质上，你是在把数据创建流程变成一种可以像代码一样进行版本控制、共享的东西。所以，我们实际上是在创建能够生成数据的代码。这样一来，你就可以把这段代码分享给任何人，因为我们已经构建好了所有这些用于分享代码、进行版本控制以及实现其他各种功能的工具。所以，如果你想做搜索，但不完全按照我做的方式来，只是稍微有些不同，你可以拿我的代码进行修改，然后用它创建自己的数据集。嗯，I think the best way to know if your data is any good is to actually look at the data. That's the best way. Uh if you can look review a few samples and um of course what is this idea of data designer? Essentially you're taking the data creation process and making it um something you know version controllable uh sharable like code. So essentially we're creating code that can create data. So now you can share this code with anyone because you know we already built all this tooling for sharing code and having version control and all of these other things. So if you're trying to do search but not exactly the way I did it and just slightly differently you take my code you modify it and you can create your own data set from it. Um,

合成人物角色与人口统计分布Synthetic personas and demographic distributions

有件事特别有意思。我开始讲一些额外的幻灯片了，没想到自己会讲到这些。我想我还有 10 分钟。好。那么，嗯，这件事真的很有意思。我们创建了一种叫作 personas 的东西，也就是 Neotron Personas USA。嗯，它的灵感还来自另一篇论文，那篇论文讲的是如何利用 10 亿个 personas 来扩展合成数据的生成。那么，什么是 persona？对吧？Persona 本质上是……我们想解决的问题是：你怎么知道自己创建的数据是否真的符合人口统计信息，或者说是否以现实世界中的某些分布为依据？对吧？这非常重要。嗯，假如我要创建一份关于美国加利福尼亚州旧金山市的医疗记录数据集，我就想确保数据集反映了旧金山的实际人口构成，对吧？so something that's super interesting. I'm getting into the extra slides which I didn't think I would. I think I have 10 more minutes. Right. So, um, this is something that's really interesting. We've created uh something called personas. So, Neotron Personas USA um it's also inspired from this other paper which is scaling synthetic data creation with 1 billion personas. So, what is a persona? Right. A persona is essentially um so we're trying to solve the problem here of how do you know if the data you've created actually um adheres to like demographic information or you know grounded in some real world distributions right that's very important u so if I were to create a data set of healthcare records in San Francisco California well I want to make sure that the uh the demograph the actual demographic of San Francisco is reflected in my data set, right?

比如说，嗯，你知道，人口统计特征有各种各样。嗯，我们使用 Data Designer 创建了一个名为 Neimatron personas 的数据集；现在它也成了 Data Designer 中的一个种子，可用于创建更多数据集。从本质上说，这让你能够从用户数据中学习，同时不会泄露任何一个人的信息。因此，现在你可以训练 AI 模型，并从人们信息的汇总数据中学习，但不必直接使用比如我的医疗记录来训练 AI 模型。我一直举医疗保健的例子，是因为我想找一个敏感行业，比如说，这里有 HIPPA 合规要求，还有各种各样我们必须遵守的法规，对吧？嗯，所以，这就是一个 persona。So like the number of uh you know there's all kinds of demographic features. Um so we created a data set called Neimatron personas uh using data designer but is also now a seed in data designer to create more data sets and um this essentially allows you to learn from user data without actually leaking any single person's information out there. So now you can train AI models and you can learn from the aggregate of um you know people's information but you don't actually have to have like for example my healthcare records you know directly um uh being used in training an AI model. I I keep using the healthcare example because I want to find a sensitive industry which is like you know there's HIPPA compliance there's all kinds of regulations there right that we have to respect. So, um, so this is a persona.

嗯，这个人是一位 50 岁的药剂师，天性自由奔放，同时又平衡着她随性放松的处事方式，诸如此类。不过说到底，这里有一个人物设定，一个职业人物设定，嗯，包括名字、姓氏、中间名、性别、年龄等等各种信息。我们有上百万个这样的设定。你可以拿来用，也可以随意把它当作种子，生成更多数据。所以，是的，关于人物设定，我大概就讲到这里。我们还剩两分钟。我会介绍一下其他工具，然后留出时间提问。哦，有个问题。好。Um, this person is a 50-year-old pharmacist with a free spirit, balances her relaxed approach, blah, blah, blah. But essentially, there's a persona, there's a professional persona, um, first name, last name, middle name, gender, age, um, all kinds of things. And we have a million of these. You can take this, you can use it, um, however you please as seed to create more data. So yeah, I think with the personas um I will probably stop here. So we have two more minutes. I'll talk about these other tools and then we'll leave time for questions. Oh, one question. Yeah.

主持人：是的。Host: Yeah.

嘉宾：对于Guest: For

主持人：对。对。对。对。所以，呃，这基本上就是我们创建它的原因：找到一种从汇总信息中学习的方法，而不必真的涉及像你我这样的个人。比如，你可以生成合成的社会安全号码。当然，它们并不会真的像你的社会安全号码，对吧？但我觉得这个想法是，比如说，呃，举个例子……呃，我不知道什么例子比较好，我……Host: Yeah. Yeah. Yeah. Yeah. So essent that's essentially why we created it is to um have a way to learn from aggregate information without actually like you know having individual people. So like you could create synthetic social security numbers. Now, they're not actually going to be like your social security number, right? But I think the idea is like if there's um for example um um I don't know what is a good example that I

嘉宾：嗯。嗯。Guest: Yeah. Yeah.

主持人：是的，是的，没错。比如说，我们实际上为一位客户创建了一些税务数据。我发现，在美国，报税收入超过每年 1,000 万美元的 18 岁以下人士恰好只有 7 个，但年龄更大、收入达到 x、y 或 z 金额的人就多得多。所以，我们创建税务数据时，比如说要生成一份合成的 W-2 表格，就得确保表格中的数值不是 LLM 随机生成的，因为如果你让 LLM 讲个笑话，它每次都会讲同一个笑话。同样，它也不会给出真实可信的数字，对吧？它总是会回归到各种数据的平均值。所以，创建数据时，你要确保这份合成的 W-2 表格、1040 表格或其他表格确实……Host: Yeah. Yeah. Exactly. So for example, we we actually created some tax data for u you know a customer and this tax data I realized there's exactly uh seven people in the US who are below 18 earning more than $10 million a year on their tax return but there's like way more people who are older and they're earning you know x or y or z amount of money and so the tax data that we create let's say we want to create a synthetic W2 form we want to make sure that that form is not just random values from the LLM because the LLM is not going to give you um if you have you asked LLM to tell a joke it's going to tell you the same joke every time. So similarly, it's not going to give you like numbers that are realistic, right? Like it's always going to come back to the average of things. And so when you create data, you want to make sure that that synthetic W2 form or 1040 form or whatever is actually um

策展、匿名化及相关工具Curation, anonymization, and related tools

遵循真实世界分布。这就是我们创建这个 persona 数据集的原因。我们有一系列工具——再讲两分钟，然后回答更多问题。不好意思。我们创建了很多工具。我想，Data Designer 只是冰山一角。不过，它目前是我们最成功的工具，因为它也是我们最早开发的工具。除此之外，NVIDIA 还有数据整理工具，比如 NeMo Curator。我相信还有其他相关演讲，我记得日程表上好像有。它很有意思。你得获取真实世界的数据，对其进行整理、修改和后处理，让数据达到可用于训练 LLM 的状态；或者你也可以创建合成数据。所以我认为这取决于你的使用场景。我不会站在这里说合成数据对所有事情来说都是最好的，但我认为有些特定场景适合整理真实世界的数据，也有些场景适合生成合成数据。我们有数据整理工具，有用于生成合成数据的 Data Designer，还有一个新工具叫 Anonymizer。adhering to real world distribution. So that's why we created this persona data set. So we have a bunch of tools uh just two minutes and then more questions. Sorry about that. Um so we have a bunch of tools that we've created. Data designer is only the tip of the iceberg I guess. Uh but it's our most successful tool so far because it's our oldest tool. Uh we also have uh you know at NVIDIA we have data curation tools. So which is Nemo curator which I'm sure there are other talks about this. I I believe I saw on the schedule. Um it is very interesting. you know you have to take data uh real world data curate it modify it post-process it get it to a state where it's usable to train in LLMs or you can create synthetic data so I think it depends on your use case like I'm not going to stand here and say synthetic data is the best thing ever for everything but I think there are specific use cases where you want to curate real world data or where you want to you know generate synthetic data um and so we have data uh curation tools we have data designer which is our synthetic data generation tool We have a new tool called anonymizer.

顺便说一下，所有这些工具都是完全开源的。因此，我们在 NVIDIA 的组织所做的一切都是完全开源的，包括 Neotron，也包括所有这些工具。嗯，所以你可以随心所欲地使用它。嗯，这个数据工具叫作 Nemo Anonymizer。它的目标是：我们能不能真正做到数据匿名化？我不知道你们怎么样，但我不知道在演讲被录制的情况下说这个合不合适，不过我曾经往自己的智能体里放过各种东西，比如有一次不小心把 API 密钥放了进去，或者放了其他一些敏感信息，对吧？还有你使用的所有智能体产生的日志数据。你需要一种办法在使用这些数据之前对其进行匿名化，对吧？这就是匿名化工具的作用。这只是一个例子，显然还有很多其他方式可以使用匿名化工具。And by the way, all these tools are completely open source. So everything that our org does at NVIDIA is fully open source, including Neotron, including all these tools. Um, so you can use it however you want. Um, so this data tool is called Nemo Anonymizer. And the goal here is can we actually anonymize data? So, I don't know about you guys, but I I don't know if I should be saying this when the talk is recorded, but I've been putting all kinds of things into my agents like you know, I accidentally put my API keys in there once or like you know some other sensitive information, right? All this log data for like all your agents that you're using. You need some way to anonymize it and before it's usable, right? So, that's what anonymizer does. So, that's just an example, but there's obviously many other ways to use anonymizer.

然后我们还有 Nemo safe synthesizer，它可以帮助我们生成敏感数据集的安全合成版本。嗯，就是这样。And then we have Nemo safe synthesizer which is allowing us to generate safe synthetic versions of sensitive data sets um as well. So yeah

资源和角色设定生成Resources and persona generation

如果你们有任何问题，欢迎来找我聊。嗯，这是我的 LinkedIn，如果你们想保持联系的话。呃，另外，你们可以使用 Data Designer。我很希望大家能试试这些 notebooks。很抱歉我们只有一个小时，所以我只能把所有代码都快速过一遍，不过你们可以自己试试这些 notebooks。如果你们对文本转 SQL 或其他一些用例感兴趣，我们也有相关博客。嗯，这展示了如何使用这些 personas。我记得有篇博客介绍了我们如何为美国创建 personas 数据集。我们获取了人口普查信息，并使用 PGM，也就是概率图模型，来建立各项信息之间的联系；基本上就是对这些人口统计信息进行处理，然后尝试据此创建一个数据集。come talk to me if you have any questions. Um here's my LinkedIn if you want to stay in touch. Uh again you can use data designer. I would love for you guys to try out the notebooks. I'm so sorry that we had to we had only one hour so I had to just like blaze through all the code but you can try out the notebooks. We also have the blogs if you're interested in text to SQL or some some other use case. So um this is this is showing us how to use the personas. Um I think there's a blog somewhere on how we created the personas data set for the US. We took the census information and we used a PGM, a probabilistic graphical model to essentially um create um like a connection between all basically take that demographic information, process it and then try to uh uh create a data set from it uh essentially.

这就是我们所做的。嗯，每个国家都会发布这些人口统计信息。所以，这并不是说……嗯，我现在在看其他幻灯片。我们最近举办了不少这样的研讨会。嗯，不过，如果你想针对美国以外的地区进行开发，或者想为日本、印度、新加坡、巴西开发产品，各种各样的用户画像都有。我相信现在我们收集的还更多了。所以，我们有各种各样的用户画像数据集。嗯，这让我们能够为你感兴趣的任何国家定制 AI。So that's what we did. Um and so every country pro uh publishes this uh demographic information. So, it's not really like um I'm looking at other slides now. We we've done a bunch of these workshops recently. Um but yeah, there's all kinds of personas if you're not if you're trying to build for something outside the US, if you're building something for Japan, India, Singapore, Brazil, I'm sure we have more now. So, we we have all kinds of personas data sets. So, um it's giving us um um a way to uh create customized AI for whichever country you're interested in.