给网页搜索加上版本控制：Exa 的 Ryan Ahern 谈历史检索与模型回测

简要说明Brief Description

Ryan Ahern 介绍了 Exa 的智能体式网络搜索和 Exa Snapshot，并解释了特定时间点的网络数据如何支持金融回测、预测信号，以及信息泄漏和过拟合检测。Ryan Ahern introduces Exa’s agentic web search and Exa Snapshot, explaining how point-in-time web data supports financial backtesting, predictive signals, and checks for information leakage and overfitting.

目录Table of Contents

• 简介
  Intro
• 作为新一代搜索者的智能体
  Agents as the new searchers
• Exa 的三项核心原则
  Exa's three core principles
• 快照搜索与实时搜索
  Snapshot vs live search
• 从信号到预测
  From signals to predictions
• 避免过拟合的回测
  Backtesting without overfitting

简介Intro

欢迎大家。很期待向大家介绍 Exa 的最新功能之一——版本控制搜索。我是 Ryan，是 Exa 的一名前沿部署工程师。Welcome everyone. Looking forward to giving you all an introduction into one of our newest features in Exa, which is version controlled search. So I'm Ryan, I'm a forward deployed engineer here at Exa.

今天我会为大家概述两件事。时间非常有限。首先，我想简单介绍一下 Exa 是什么，以及什么是智能体式网络搜索。And today I'm going to give you an overview of two things. We only have a very short amount of time. So the first one, I just want to motivate what is Exa, what is agentic web search?

你得多少相信它所描绘的愿景，才能理解为什么你会想用它来做我接下来要讲的第二件事：用它对金融模型进行回测。所以，首先我想了解一下在座各位的情况：有谁之前听说过 Exa，或者用过它？好，看来举手的人还不少。Just because you kind of need to believe in the dream of that to understand why you'd want to use it to do the second thing I'm going to talk about, which is how to backtest financial models using that. So first, just to understand where I'm at in the room, who here has heard of Exa before or used it? Okay, I've got a good show of hands.

这里有谁以前用过 Google 来做一些工作或个人用途，比如直接在 Google 里输入内容进行搜索？Who here has ever used like Google back in the day to like do some sort of work or personal use, just like typed into Google to do some sort of search?

希望在座的各位也都一样，大多数人都占有相当不错的市场份额，你们可能听说过它。最后一个问题：现在有多少人已经不太用 Google 了？你们会用 ChatGPT、Claude，或者如果你们走在技术前沿，甚至会用 Instinct 或 Muse 来搜索。Hopefully like everyone in the room, most people had pretty good market share, you might have heard of it. Last one would be, how many of you now no longer really use Google? You use ChatGPT or Claude or maybe even Instinct or Muse if you're like at the cutting edge to actually do your searches.

是啊，大多数人都是这样，对吧？所以我们要把这些东西都连接起来。最后这一点，现在很多人会说：“嘿，我现在甚至都不怎么用桌面搜索了。”Yeah, most people, right? So we're going to connect all of these things together. This last point where a lot of people now are like, "Hey, I don't even really use desktop search anymore.

“我会让智能体来做这件事。”这种情况在市场上更普遍，对吧？关键是，智能体通常很擅长理解如何搜索，因为它们可以反复尝试很多次。它们不太会累。I kind of ask an agent to do this." is something that we see more broadly in the market, right? And the idea is that like agents tend to do a pretty good job at understanding how to search because they're allowed to try multiple, multiple times. They kind of don't get tired.

他们往往非常随和，而且工作起来特别特别努力。They tend to be quite agreeable and work very, very hard.

所以，当你说“这就是我想要的东西”时，它就会开始进行许多不同的搜索，以满足你的目标。还有一点是，它们的搜索方式往往和人类不同。作为人类，你可能会输入几个与目标相关的关键词，然后希望前几个结果里就有最好的；要是没有，你也许会尝试换个搜索词，之后就差不多放弃了。So when you say like this is the thing that I want, it's going to go out and it's going to do many, many different searches in order to satisfy like your objective. And the other thing here is that like they tend to search different to humans. So as a human, you might put in like a few keywords for the thing and then hope that you get the best results like in the first couple, and otherwise like maybe you try like a different query and then you'll kind of give up.

而智能体会非常啰嗦。它们往往会非常详细地描述自己想要搜索什么，并且会执行大量搜索，以便在回应之前更全面地了解相关领域。这意味着它们平均每项任务会进行更多次搜索。Whereas agents will be very verbose. They tend to describe what they want to search in great detail and will execute many searches in order to understand the space more fully before they respond. and that means that they do way more searches on a per-task basis.

智能体成为新一代搜索者Agents as the new searchers

总体来说，这个想法大概是：“嘿，如今大多数人都会用智能体来帮忙处理搜索任务。”而 Exa 就是在这方面发挥作用。所以，当你使用各种智能体系统时，比如 Cursor 想搜索最新文档，帮助它的编程智能体完成任务；或者你在和 Gamma 对话时说：“嘿，要是能找到苹果公司 2018 年的 SEC 数据来完成我的幻灯片就太好了，那样效果会非常出色。”它就会调用 Exa，在整个网络上搜索。我们有自己的爬取基础设施、数据库、嵌入模型和检索架构，能准确找到回答查询所需的内容。这些是几家大型公司的标志。不过，既然这是 AI Engineer 大会，下面我就来介绍几个我们在金融领域合作的典型类型。Generally the idea is that like, "Hey, nowadays most humans, you're going to be using agents to kind of broker your search tasks." and that's where Exa kind of plugs in. So, when you're working with like many agentic systems. So, whenever Cursor wants to go and search latest documentation to help its coding agent complete its trajectory, whenever you're like talking to Gamma and you're like, "Hey, it'd be great if I had just this like SEC data from like Apple in like 2018 to complete my slide, like that would make it brilliant." It's going to use Exa to go out, search the full web, so we have our own crawling infrastructure, our own database, our own embedding model, our own retrieval architecture to get exactly what it needs to answer its query. this is like a couple of the logos of like the bigger companies, but because this is AI engineer, like here are a few like kind of archetypes that we work across the financial space.

我与很多对冲基金、资产管理公司、主权财富基金和风投机构合作，处理各种各样的任务，因为他们也需要最新信息。所以，如果你在对冲基金工作，你就要根据信息进行交易。你会在市场中寻找阿尔法收益，对吧？I work with a lot of hedge funds, asset managers, sovereign wealth funds, VCs across like a bunch of different tasks because they also need like the latest information. So, if you're a hedge funder, you're going to be trading on information. You're looking in the market for alpha, right?

以及能够帮助你了解如何更好地定位自身组织的信息。如果你是一家保险公司，你可能会想了解自己正在与哪些交易对手合作，或者为哪些对象承保，以便最终做出判断。因此，你要确保依据最充分的信息做出判断，并确保风险模型尽可能准确。And for information that allows you to understand how you could position your organization better. If you're a an insurer, you're probably understanding like what counterparties you're working with or who you're insuring to make sure that ultimately you're making the determination. So, you want to make sure you're making it on the best information and that your risk models are the most accurate.

或者即便你在风投行业，也会想：“我得找到最炙手可热的初创公司。我得抢先一步，在它们爆红之前就知道下一家会是谁。”Or even if you're in VC, you're like, "I need the hottest startup. I need to know the in next instant before they blow up."

而且它们不一定会存在于某种结构化数据集中。比如，我需要到网上去找到它们，无论它们存在于哪里。这就是他们来找 Exa，寻求这方面支持的原因。最后，我们来总结一下为什么选择 Exa，以及我认为为什么现在你会觉得：“好吧，我正在构建智能体，或者我对智能体感兴趣，而且我肯定需要生成式网络搜索。为什么选 Exa？”原因在于，我们从某种意义上说是一家实验室：我们对搜索市场以及智能体进行搜索的方式非常感兴趣，而且我们专注于解决这个问题。And they're not necessarily going to be in like some structured data set. Like I need to go and find them wherever they exist in the web. that's what they come to Exa to kind of support them for. and then the kind of to wrap up the last part on like why Exa and the reason why I think now you'd be like, "Okay, I'm I'm building agents or I'm interested in agents and I definitely need a generative web search. Why Exa?" and it's because like we're a lab in the sense that we're very interested in the search market and in the ways in which agents search and we're hyper focused on that problem.

Exa 的三项核心原则Exa's three core principles

有三个核心原则，接下来我们会谈谈它们如何融入版本管理，以此过渡到演讲的下一部分。不过，首先显而易见的是质量，对吧？你要确保智能体调用工具并进行检索时，能拿到它所需要的东西，因为它需要进行的工具调用越少，推理成本效益就越高，而且它也越有可能顺利完成整个执行过程。There's three like core tenets and then we'll talk about how that goes into versioning to segue into the next part of the talk. But first and quite obviously is like quality, right? So, you want to make sure that when the agent does that the tool call and the retrieval, it gets what it needs cuz the fewer tool calls that it needs to make, the more cost-effective on your inference and also just like the more likely it is to succeed on its overall trajectory.

你不会引入某些会让它产生偏差的东西。我们确实非常关注质量方面。最近我们发布了一个新基准测试，我们认为它能更准确地单独评估搜索工具的表现，而不把推理能力也算进去，因为这两者可能会相互混淆，对吧？You're not going to bring in something that like misaligns it. we're really interested in the quality space. We recently released like a new benchmark which we think more accurately assesses the performance of like the search tools separately to the inference because these two things can be confounded, right?

比如，哪个智能体或模型在调用搜索，以及哪个搜索连接到了哪个模型。这样你就可以大致了解一下我们目前最新的想法。其次，谈谈成本。Like what agent or what model is calling the search and what search is connected to what model. So, that you can kind of have a look at that to see our latest thinking there. Secondly, on cost.

所以，关注的不只是工具调用这一项，对吧？比如一次搜索要花多少钱——因为代理可以调度许多不同的搜索。这个想法既是：“嘿，这对完成你的执行轨迹有什么价值？”同时也要看它对代理完成任务的总成本有什么影响，因为这才是你真正关心的，对吧？通常来说，你不会那么在意某一次工具调用。So, it's not just the line item of the tool call, right? Like how much does one search cost because agents can broker many different searches. so, the idea is both like, "Hey, what's the value to this to complete your trajectory?" But also like what's the impact of this on the total cost of that agent completing its task cuz that's actually the thing that you care about, right? You're generally less opinionated about one tool call.

这就像是在问：“这个整体代理花了多少？”所以我们才做了动态高亮之类的功能，这意味着对于每一个结果，我们不再只是返回完整页面。It's like, "How much did this overall agent take?" and that's why like we worked on things like say dynamic highlights, which means that for every single result, we're not we're not just returning full pages.

我们不再只是针对单个页面（也就是我们之前的高亮功能），找出该页面上哪些片段最合适，并将它们发回上下文窗口，以完成整个执行过程。现在，我们会对所有搜索结果都这样处理，这意味着相比使用完整页面文本，你能以少 95% 的 token 获得更高质量的结果。我们写了一篇相关博客，非常有意思。简单来说，这能帮你在后续为智能体完成执行过程而进行推理时，节省大量输入 token 的开销。we're not even just taking one page, which is our previous highlight features, and figuring out what are the exact right snippets on that page that we need to send back to the context window to complete the trajectory. We now do that across all of the search results, which means that you can actually get higher quality results at 95% fewer tokens than you would at full page text. We have a blog about this, it's very interesting, but basically it means you'll save a lot of money downstream on all of the input tokens you'll be spending on inference for the agent to complete its trajectory.

关于延迟，最后再说一点：因为我们拥有自己的基础设施，所以可以实现极低延迟的搜索。我想简单谈谈光谱的另一端，也就是耗时很长的深度研究。Last one on latency, so because we own our own infrastructure, we can do very low latency search. I'm just going to touch on the other side of the spectrum, which is like very long-running deep research.

所以我们有 ExaAgent Ultra，这是我们不久前发布的，可能是几周前。它在很多基准测试中都处于业界领先水平，尤其是那些涉及长时间运行搜索的测试。这是因为我们能在底层做很多事情。显然，搜索框架是我们自己的，而且我们也能对它如何与智能体交互作出自己的设计，让框架执行这类需要长时间运行的研究任务。So we have ExaAgent Ultra, which we released like not too long ago, maybe a few weeks ago, which is like state-of-the-art on a lot of benchmarks around like very long-running search. And that's because we can do a lot of things under the hood. We have obviously we own the search harness, but we can also be opinionated about that how that interacts with the agent to harness do these like very long-running research task.

所以我非常建议你去看看这些东西。这只是个过渡，好让大家明白：“好，Exa 确实非常重视这个领域。我觉得他们的搜索会做得很好。”接下来要讲的内容很有意思，也是这次演讲的主题：版本管理。就在最近，我们发布了 ExaSnapshot。它包含超过 4000 亿份网页快照，你可以按日期搜索。So highly recommend you check those things out. this is just a segway to be like, "Okay, Exa really cares about this space. I think that their search is going to be good." To motivate the next part, which is really interesting is the topic of this talk, which is versioning. So just recently we released ExaSnapshot. it's 400 billion plus snapshots of the web, and you can search them by date.

这在金融服务行业确实很有意思。我们会深入探讨这一点，不过首先我想稍微谈谈这基本上意味着什么。我们是一家搜索公司，因此一直在整个网络上搜索最新信息，以便将其添加到我们的索引中。每次这么做时，我们都会检查信息是否足够新。and that's really interesting in the in the financial services industry. We'll get into that, but first I'm just going to talk a little bit about like what this basically means. So we're we're a search company, so we're constantly searching across the web for the latest information that we can add to our index. and every single time we do it, we check it for we're checking for freshness.

因为归根结底，你希望在搜索时获取最新的实时信息。但由于我们会进行所有这些检查，这意味着我们可以在检查时保存这些 URL 的一个版本，并保留版本历史。你可以通过两种方式访问这些内容。一种是：“我知道自己想查找某个 URL 在指定日期当天或之前的版本。”Because ultimately you want to have the latest live up-to-date information at the point at which you're searching. But because we're doing all of those checks, it means that we can capture a version of those URLs at the point where we checked it and have like a version history. and you can access that in two ways. One is like, "I know the URL I want to find as it was at or before a given date."

这样，你可以给我们一个 URL，并说：“这是我想要的日期。我想要该日期或之前的数据，并且请像我当时提出请求那样把它提供给我。”第二个更令人兴奋的概念是版本搜索。对于搜索 API，思路是：“嘿，如果我在之前某个时间点执行这次搜索，会发生什么？”所以，我只能得到在那个时间点或之前本来会提供给我的结果。这会带来一些非常有趣的影响。And so you can give us a URL and you say, "This is the date that I want it. I want it at that date or before and serve it to me as you would have if I had requested it at that point." The second one, which is even more exciting, is this concept of version search. So, where a search API, and the idea is like, "Hey, what would happen if I had executed that search but at a prior point, right?" So, I only get results that would have been served at that point or before. that has very like interesting implications.

我这里只做一个非常小的示例。I'm just going to do one like very small example here.

快照搜索与实时搜索Snapshot vs live search

那么，OpenAI 最近（截至 10 月 1 日）……左侧展示的是 10 月 1 日的搜索快照，右侧则是我制作这张幻灯片时，也就是几天前的实时搜索结果。不过在 10 月初，当时有些讨论认为，OpenAI 的年化营收运行率大约在 700 亿美元左右，对吧？这也导致市场开始以某些方式预测他们将如何估值，以及估值会是多少。So, OpenAI recently, as of October 1st. So, on the left-hand side here, you'll see a snapshot search as it was in October 1st. And on the right-hand side was a live search as of like a couple of days ago when I made this slide. but at that time in early October, like there was some discussions that like OpenAI had a like annualized like run rate around like the 70 billion, right? and that was leading to certain ways in which the market was projecting how and where they would be valued.

然后最近又有一些相关报道，最早是 Axios 报道的。再后来，又有一些媒体报道称：“实际上，按一些有意思的会计口径算，数字更接近 500 亿，对吧？”事实证明，他们算的是总收入，但实际上他们的一部分收入来自云服务提供商。严格来说，正确的核算方式应该是把这部分列为供应商代收代付项。And then more recently, there was some and that reporting was originally by Axios. And then more recently, there were some publications that were reporting like, "Actually, it's closer to 50 billion for fun accounting reasons, right?" Like it turns out that they were counting gross revenue when actually like some of their revenue was coming from cloud providers. And technically, the way that you should account for that would be that there would be a vendor pass-through.

所以，净年化收入这个指标更好。这就是其他一些参与者报告的情况，等等，对吧？但基本上，这意味着净年化运行率的预测有所下调，从 700 亿降到了 500 亿，对吧？So, net annualized revenue is better. That's how what some what some other players were reporting, etc., etc., right? But basically, it meant that the net annualized run rate was like kind of down forecast from 70 billion down to 50 billion, right?

这只是一个金融方面的话题，你们中有些人可能会感兴趣，也可能不感兴趣。但版本搜索的意义在于：假设你有兴趣创建一种产品、解决方案或智能体，它的用途就是寻找这些东西，对吧？现在你可以回头看看，想一想：“好吧，在大家都弄明白之前，对吧？在市场开始关注之前，也在某位金融分析师发表那篇抨击某个特定事物的评论文章、让所有人都知道这件事之前，互联网上都有哪些信息，对吧？”Now, this is just like a financial thing that some of you may or may not find interesting. But the so what for version search is like, "Let's say you're in the interest of creating a product or a solution or an agent that is made to that will that its job is to find these things, right?" You can now go back and be like, "Okay, before everyone figured it out, right? Before the market and before some financial analyst published the think piece that was like slamming down on this particular thing and everyone knew about it, what information existed on the internet, right?

比如，是否有一份 SEC 申报文件会在会计脚注中披露这件事？比如，是否有其他供应商发布的报告，你可能觉得没那么可靠，但它实际上却发现了这类事情，也许你应该换个角度看待它们。你有可能在市场上其他人发现之前就追踪到这些事情。但最重要的是，你不仅可以测试一下，还可以对它进行回测，对吧？Like was there an SEC filing that would have revealed this thing in an accounting footnote? Like was there some other reporting from some other vendor that maybe is like you would consider is like less reputable, but actually was able to find that kind of thing and maybe you should consider them differently." you can you can potentially like trace these things before like the rest of the market found it. But most importantly, like not only can you test out that you can backtest that, right?

然后你就可以调整模型的设置方式，也可以调整智能体的设置方式，让它能够回答这样的问题：“它能不能发现这个差异？”更重要的是，它能不能以一种具有泛化能力的方式发现这个差异，对吧？比如说，我们创建 50 组样本，看看它能不能在这些样本上表现良好，再用留出集检验一下，然后说：“好，我觉得我的模型现在应该很擅长发现这类特征了。”You could then adjust the ways in which you're setting up your model, you can adjust the ways that you're setting up your agent to be able to be like, "Could it have found this discrepancy?" and more importantly, could it have found that discrepancy in a way that is generalizable, right? Let's like create a set of like 50 sample sets, see if it can perform against that, check it against the holdout and be like, "Okay, I think that my model is now going to be pretty good at finding this kind of feature."

这是网络搜索领域里相对较新的东西，而且你可以对它进行回测。这只是一个小示意图，用来说明这个想法。不过，只要你用现有的任何搜索产品进行实时搜索，搜索结果就是实时的，因此包含了所有信息。比如在这个小例子里，我们要预测一家公司是否被收购。如果你进行实时搜索，就会发现这笔交易已经宣布了，对吧？因为这件事已经发生了，消息也已经传得到处都是。And that's something that's kind of new in this in the in the web search space and that you can backtest that. this is like a like a little diagram just to kind of motivate the idea, but at any like live search that you do with like any existing search product, the idea is it's live, so it has all of the information. So, in this particular little example, we're we're trying to predict whether company got acquired or not. in this case, if you do a live search, you're going to find like that the deal was announced, right? Because that happened, it got released everywhere.

你可以这样说：“嘿，我想训练一个模型，让它能够预测这家公司是否被收购。”所以，我会分别看公告发布前一天、前 7 天、前一个月的数据，看看能不能预测出来，对吧？You could be like, "Hey, I'm trying to train a model that's able to predict whether this company got acquired or not." So, I'm going to go like 1 day before the announcement, 7 days before the announcement, a month before the announcement and see whether I could or could not predict it, right?

然后你可以构建一组这样的公司，基本上训练你的模型，或者其他试图进行这种预测的系统，看看它能不能可靠地预测这个特定特征。所以我想花一点时间聊聊，目前不同的系统是如何借助 Snapshot 进行回测的。这里大致有三种类型。and you can build a set of those companies and basically train your model or whatever system it is that it's trying to do this prediction, and see like was it able to reliably like predict this particular feature. So, I'm going to spend a little bit of time here just talking about how different systems are currently being back-tested with Snapshot. So, there's like three kind of archetypes here.

第一个是，我采用的是一种非常传统的模型。可能是机器学习模型，甚至可能更简单，比如决策树之类的。我想做的就是从网上收集非结构化数据，并将其整理成某种结构化信息集。比如软件价格、一些情感分析数据，或者某个特定网站的条款与条件。The first one is I'm operating like a very traditional model. Maybe it's ML, maybe it's even more simple than that, like a decision tree or something. and all I want to do is just gather unstructured data from the version web into some sort of structured set of information. So, it could be something like software prices, it could be some sentiment analysis, it could be like the terms and conditions of a particular site.

无论是什么在驱动你的模型，并不是所有信息都会持续可用。比如说，如果你想了解软件的价格，那款软件的价格页面未必会以一种有版本记录的方式存储在网站上，让你能够可靠地访问。又或者，你可能想做一些情感分析，但这会受到一个问题的困扰：这些情绪是在什么时间点表达出来的，对吧？Whatever it is that like powers your model, not all information is continually available. Like if you're trying to understand software prices, like, you know, the page of that software price is not necessarily stored in a versioned a way for that website in a way that you could reliably access. Or potentially like you have some like, sentiment analysis that you want to do, but like that's going to be mired in like what point was that sentiment released in time, right?

从信号到预测From signals to predictions

你希望能够获得一组可靠的这类信号。然后你可以把它转换成一个信号数组，输入模型并据此进行预测。这大概就是那种基础空间。You want to be able to get like a reliable set of those signals. And then you can just transform it into some array of signals that goes into your model and you can predict it. That's like the sort of basic space.

还有些人只是在丰富他们现有模型原本无法获取的数据。所以这基本上就是在现状基础上多做一点。真正有意思的是接下来的两个领域。And some people are just like enriching data that they wouldn't have had access to for their existing models. So, that's just like sort of like the status quo plus one. The interesting space is in like the next two areas.

所以，第一个想法是添加一个智能体功能。比如说，我有一个现成的模型，它表现得相当不错，但我觉得，如果给这个模型加入一两个智能体功能，它的表现应该会更好。So, the first one is this idea of like adding an agentic feature. And here is like, okay, well, I have this existing model. It performs pretty well, but I have an idea that like if I add one or two like agentic features to this model, it would be able to perform better.

那么，假设我们要做类似负面媒体扫描这样的事情。负面媒体扫描必然要进入网络，在其中搜索所有现有信息，也可能以智能体的方式来执行，对吧？So, let's say something like an adverse media scan. Necessarily like adverse media is going out into the web, searching across it for all of the information that exists, potentially doing it in an agentic way, right?

当你进行多次搜索时，实际上是在探索多条不同的路径，然后把所有这些结果汇总成某种结构。另一个例子是股权研究，思路也很类似。比如说：“嘿，我想全面考察所有公司，找出它们可能的 ARR 营收、员工人数，以及所有重要特征。”你可以在进行预测时使用类似代理式搜索的方法，去收集所有这些信息，然后将其输入传统模型。Where you're doing multiple searches, you're exhausting multiple different trajectories, and then you're going to take all of that and then condense it into some sort of structure. another example here would be to say like equity research in much the same way. Like, "Hey, I want to go out. I want to look at the full universe of companies and I want to find like potentially all of their like estimated like rack revenue or head count or all of the features that matter." and the idea is that you'd be using something like adverse search at the point at which you're doing a prediction life to go out and gather all of that information to then send into your traditional model.

现在你可以使用类似负面快照的功能，对这些特定特征进行回测。这样你就可以回过头来说：“嘿，如果我提前一个月做这样的负面媒体扫描……”And you can now use like adverse snapshot to back test those particular features. So, you can go back and be like, "Hey, if I was to do this like adverse media scan like 1 month before."

或者说，我可能在大约四个月前就已经把一批人纳入了我的产品组合。当时在他们准备进入我的产品组合时，我会对他们说些什么？这些说法又如何与今天的情况进行对照追踪？这类功能尤其能激励很多金融服务客户，因为你不必现在就相信搜索会变得更好。你可以回过头来，针对现有流程进行回测，然后说：“嘿，效果确实更好。”所以，这可能就是下一阶段的演进，再向前迈一步。这里最右侧的部分，大概代表着未来。Or maybe I've already like onboarded a bunch of people into my portfolio like 4 months ago, what would I have said at the point where they were going to enter my portfolio? And would how is that tracked against today? and that kind of motivates like especially with a lot of financial services customers like you don't now just have to believe that like search will be better. You can actually go back and back test that against your existing processes and say, "Hey, it's better." so, that's like the maybe the next evolution like one step plus. on the far right-hand side here is kind of like maybe the future.

所以，有一些初创公司正在预测领域开展业务。这里的想法是，整个模拟或预测过程都由一个智能体来管理。So, there's some like startups that are working in this prediction space. And the idea here is that the full sort of like simulation or prediction is being managed by an agent.

所以，中间就像是一个机器学习模型，可能在进行预测，或者运行某种算法或决策树。右边则像是一个智能体，它会自行去做一堆事情。它可能会搜索内部知识库、访问数据流，或者做其他任何事情。So, in the middle is like an ML model that's maybe doing a prediction or some algorithm or decision tree. On the right-hand side like it is like an agent that's just going off and doing a bunch of things. It could be searching internal knowledge, it could be accessing data feeds, it could be doing whatever it is.

但它可能会做的一件事是调用工具并访问网络。这样一来，你就不必费力创建某种内部语料库，试图模拟访问整个网络会是什么样子。你只需要说：“嘿，我想确保自己能够控制这个功能，这样我就知道未来的信息不会直接泄漏到我的模型里。”比如，我正试图通过某种极其疯狂、会访问所有信息的智能体流程来预测公司收购事件。But one of the things that it may do is to tool call and access the web. And the idea is like now you don't have to try and create some internal corpus that tries to represent what it would be like to access the full web. You can just be like, "Hey, I want to make sure that I can control that feature so I know that future information is not just going to leak directly into my model." Like I'm trying to predict company acquisitions through some very insane like agentic process that's accessing everything.

现在你可以这样说：“好吧，我们先确认一下，它不会只是进行一次网页搜索，发现这家公司已经被收购，然后回来就以差不多 100% 的准确率预测出来。”这就是右侧大致的情况。我再举一个中间部分的小例子，给这段讨论收个尾。昨天我们举办了一场关于 Snapshot 的扩展研讨会，现场大约有 100 个人，大家都在尝试之前那个预测公司收购情况的同一个例子。有件事挺有意思：这是研讨会上某个人给出的一个示例特征，而且这个人的得分相对较高。You can now be like, "All right, let's just make sure that's not just going to have one web search that finds that it was acquired, comes back and predicts it with like 100% accuracy. so that's kind of the sort of right-hand side there. I'm just going to give like a little example of like the middle just to round off this kind of discussion. so yesterday we ran like a kind of extended workshop on snapshot and we had a bunch of around about 100 people in the room that were kind of running through this the same example we had before about trying to predict company acquisitions. one thing that was kind of interesting is that the this was like some like one of the example features from the workshop from one of the people that got a relatively high score.

但这里有一点很有意思，先说一下背景：第一个问题是，“在过去一年里，是否有投资者或重要股东公开呼吁出售整家公司，并表示支持？”第二个问题比较有意思：“这家公司的股价是否至少比过去五年的最高价低了 60%？”它的意思就是股价有些下跌。最后一点也很有意思：“这家公司是否名为 Dominion？”But a thing that was interesting here is like the just to motivate so the first one was this idea that like, "Hey, within the past year did an investor or a significant shareholder publicly call for a supported sale of the company as a whole?" And the second one, which is the interesting one, is like, "Hey, did this company's share price was at least 60% lower than high share price during the past 5 years?" Which the idea is like it's kind of dipped down. the last bit that's interesting is like, "Hey, is the company named Dominion?"

避免过拟合的回测Backtesting without overfitting

对于了解回测的人来说，这里有点过拟合的意思。这里的想法是，这可能是智能体会做的事，对吧？如果它是在一组实时数据上训练的，就有可能只是对你现有的训练数据过拟合。而 Snapshot 的想法是，你不仅可以回溯并测试这些情况，还可以回溯设置评估和智能体测试框架，以便检测这些问题；同时，你也可以尝试验证它是否具备泛化能力。Now this is a little bit of overfitting for people that know like backtesting. like and the idea here is like this could be something that like an agent could do, right? If it's trained on like a live set, it can potentially just overfit to the training data that you have. But the idea with snapshot is like not only can you go back and test for these things, you can go back and like set up your evals and your agentic harness in such a way that you can detect these things, but you can also try and check that it generalizes.

如果你受限于只能进行实时测试，比如在生产环境中测试，那这件事就会非常非常难做。而且你只能随着时间推移改变这些功能的描述方式，并根据实时情况进行验证，对吧？你可能得等上几个月，才能试着弄清楚你的智能体系统究竟是否具有预测能力，或者是以不同方式出现了过拟合。And this is something that would be very, very hard to do if you are gated by like testing live, like in production. And you can only change the ways in which these features are described over time and check against real time, right? you might be waiting for months to try and see how and why your agentic setup actually is predictive or is overfitting in different ways.

也就是说，你可以描述这些相当复杂的特征。比如说，你可以想象尝试用某种非语义搜索来重现这些特征，但在现有的 Exa 语义搜索体系里，你想把它们设计得多复杂都可以。所以你可以提出这样的需求：“嘿，我想看看，有没有关于一些声誉良好的副总裁的负面报道？他们在风投基金任职，而且上过《AI Engineer》播客，还在节目里说过一些不好的话。”对吧？你可以尽情发挥，设计你认为有预测力的特征，因为 Exa 搜索的核心功能支持这些；然后你可以对它们进行回测，看看表现是否有效。接下来我就简单总结一下。that is like you can describe these like quite complex features like these ones are like maybe you could imagine trying to recreate in like some sort of like non-semantic search, but you can go as crazy as you want within the within existing like Exa semantic search world. So, you can say things like, "Hey, I want to see if there was any adverse media against like reputable VPs that are operating like that are operating within VC funds that have like been on the AI Engineer podcast and they've said something bad about it, right? Like you can go as hard as you want on these features that you think are going to be predictive cuz that's what core Exa search allows, but then you can back test it, right? and see if it's performative or not. then I'm just going to kind of like recap.

所以，我们这里首先谈到的是使用 Exa，因为它能以高性价比提供高质量的智能体式网页搜索，并适用于各种延迟场景。我们围绕这个主题写了很多博客，所以如果你还没试过 Exa，鼓励你试试看。因为你首先想知道的是：“嘿，我真的觉得这能给我的模型带来一些好处吗？”正如我们谈到的第二点：“嘿，使用 Exa snapshot 来回测这些金融模型，或者任何风险模型、预测模型，并结合这些网页搜索功能，看看：嘿，加入它能否为你和你的智能体流程带来价值？”非常感谢。So, the first thing that we talked about here is like using Exa because it's quality cost-effective agentic web search across every latency. we have lots of blogs around this, so encourage you if you've not tried Exa to have a go at this because the first thing you want to know is like, "Hey, do I actually think that this is going to add some benefit to my model?" The second thing as we talked about is like, "Hey, using Exa snapshot to be able to like back test these financial models or whichever model of risk model prediction model with these web search features in order to see like, hey, does adding it add value to you and your agentic processes?" Thank you so much.