前沿模型为何难当买方分析师：Google DeepMind 与 LinqAlpha 谈真实工作流评测

简要描述Brief Description

Google DeepMind 和 Kaggle 探讨可复现的真实世界评估，随后介绍 LinqAlpha 的 Mandate Bench 买方股票研究案例。Google DeepMind and Kaggle discuss reproducible real-world evaluations, followed by LinqAlpha’s Mandate Bench case study for buy-side equity research.

目录Table of Contents

• Kaggle 与真实世界评估
  Kaggle and real-world evaluation
• 基准测试失效的地方
  Where benchmarks fail
• 复现并审计 65 项基准测试
  Reproducing and auditing 65 benchmarks
• 标准化基础设施和基准测试支持
  Standardized infrastructure and benchmark support
• 委托基准与投资研究
  Mandate Bench and investment research
• 重建 Sonoco 并评估推理能力
  Rebuilding Sonoco and grading reasoning
• 初步结果与即将开展的基准测试
  Preliminary results and upcoming benchmarks

Kaggle 与真实世界评估Kaggle and real-world evaluation

下午好。非常高兴今天能和大家聊聊 Evals。今天我们将探讨如何为生产环境中的真实工作流程创建 Evals 所面临的挑战、Google DeepMind 和 Kaggle 如何合作解决这一问题，以及我们与合作伙伴 Link Alpha 共同开展的一项案例研究。Good afternoon. Super excited to be talking about Evals today. So today we'll be covering the difficulty of creating Evals for real world workflows in production. How Google DeepMind and Kaggle are collaborating to solve this problem and a case study that we did with our partners at Link Alpha.

接下来，他们将介绍一个名为 Mandate Bench 的专项基准测试，主要考察模型能否胜任买方股票分析师的工作。首先，简单自我介绍一下。我叫 Jenny，是 Google DeepMind 的产品经理。So they'll present next about a specific benchmark called Mandate Bench which basically looks at how well models can perform as a buy-side equity analyst. So to start with some quick intros. I'm Jenny. I'm a product manager at Google DeepMind.

我负责我们 Kaggle 基准测试团队。Ivan 是我们 GDM 非常出色的前沿部署工程师。遗憾的是，他今天来不了。Jacob 在观众席上，他来自我们的 Link Alpha 合作伙伴团队。I lead our Kaggle benchmarks team. Ivan is our amazing forward deployed engineer at GDM. He can't make it today unfortunately. Jacob is in the crowd and from our Link Alpha partners.

所以他会和我一起做演示，遗憾的是，杰克也没能来。So he'll be presenting with me and Jack also couldn't make it unfortunately.

先简单介绍一下 Kaggle，以及我们是如何走到今天的。你们可能都知道，Kaggle 是数据科学和机器学习竞赛的聚集地。我相信你们中的很多人都接触过 Kaggle，或者过去参加过 Kaggle 竞赛。我们已经这样做 15 年了。So just starting with a quick intro about Kaggle and how we got here. So Kaggle you guys probably know Kaggle as the home for data science and ML competitions. I'm sure many of you have have interacted with Kaggle or competed in a Kaggle competition in the past. And so we've been doing this for 15 years.

本质上，我们是在现实世界中评估人们在不同方面的能力。我们已在整个平台注册了一个由 3100 万名 AI 和 ML 爱好者组成的社区。过去 15 年里举办了数千场竞赛，并建立了一个人人都非常信赖的排行榜。Kaggle 能在过去 15 年里取得成功，关键就在于我们的严谨性。Basically evaluating humans on different capabilities in the real world. And we've built up a community of 31 million AI and ML enthusiasts across the platform. Thousands of competitions over the last 15 years and one leaderboard that everyone really trusts. And what has really made Kaggle succeed in the last 15 years is our rigor.

因此，我们会使用封存的私有测试数据集、进行数据泄漏检测并部署反作弊基础设施，同时确保在 Kaggle 上举办的每一场竞赛都经过精心设计，能够有效评估人们在该竞赛旨在考察的能力方面的表现。So we have sealed private test data sets. We leak detection, anti-cheating infrastructure, and really make sure that every single competition that is put on Kaggle is designed really well and good at evaluating humans on the capability that it's supposed to be.

这就把我们带到了今天。2026年2月，Kaggle 实际上并入了 Google DeepMind，我们也将主题从评估人类转向评估前沿模型。嗯，Kaggle 基本上已成为 GDM 面向公众的基准测试平台。我们的目标是为所有前沿模型打造一个开放、中立的试验场，使用相同的基础设施、相同的任务和相同的评分标准，这样每个分数都能真实反映模型表现。我们的目标是成为行业基准测试的标准，就像 Kaggle 已经成为竞赛领域的标准一样。and so that brings us to today. So, in February 2026, Kaggle actually moved into Google DeepMind and we're moving our subject from evaluating humans to evaluating frontier models. Uh and so Kaggle is basically become GDM's public platform for benchmarks. And our goal is to make an open, neutral proving ground for every single frontier model that uses the same infrastructure, the same tasks, and the same grading so that you every score is representative. and our goal is really to become the industry standard for benchmarks the way that Kaggle already is for competitions.

那么，这到底为什么重要？我们为什么要探讨这个问题？Kaggle 为什么要着手解决这个问题？因为基准测试非常重要。基准测试是我们衡量 AI 行业进展的方式。So, why does this actually matter? Why are we getting into this problem? Why is Kaggle trying to tackle this problem? and it's because benchmarks are really important. Benchmarks are how we measure progress in the AI industry.

而现在，基准测试的现状从根本上来说已经出了问题。想想看，如今基准测试有各种各样的用途。嗯，前沿实验室用它们来训练模型、不断提升模型表现。初创公司则用它们来决定要围绕哪些模型打造产品。And right now, the state of benchmarks is fundamentally broken. So, you think about all the different ways that benchmarks are used today. Uh frontier labs are using them to train and hill climb models. Startups are using them to decide what models that they want to build their product around.

投资者正用它来评判技术方面的量化指标。而你如何衡量——比如基准是否准确衡量了你希望它衡量的内容——恰恰是模型最终会不断优化的东西。当你衡量错了东西，模型在它们本该完成的任务上反而会变得更差。Investors are using it to judge technical amounts. and how you measure like what whether the benchmark is correctly measuring what you want it to measure is exactly what the models end up hill climbing. and when you measure the wrong thing, the models actually get worse at what they're supposed to be doing.

基准测试的局限性Where benchmarks fail

所以，如今的基准测试有几种不同的失效方式。第一种是相关性问题。大多数公开基准测试实际上并不能反映你的智能体目前在生产环境中所做的真实工作。So, there's a few different ways that benchmarks fail today. the first one is relevance. So, most public benchmarks don't actually resemble the real work that your agents are doing in production today.

所以，它们中的大多数测试的是合成谜题、学术冷知识和随机数学题，而这些与智能体在生产环境中的实际工作几乎毫无相似之处。如今基准测试的第二个问题是，从算力角度来看成本非常高，而算力正是当今 AI 行业最稀缺的资——资源。So, most of them test synthetic puzzles, academic trivia, random math problems, and that bears little little resemblance to what your agent actually does in production. The second problem with benchmarks today is it's very expensive in terms of a compute perspective, which is the like most scarce re- resource in the AI industry today.

比如说，拿一个基准测试来说，假设它包含 500 个多轮智能体任务，而你想在 16 个前沿模型上运行它，那么每轮测试就要花费数万美元。每当有新模型发布——现在基本上每一两周就会有一个——你都必须重新跑一遍，才能让基准测试保持最新，否则它就会过时。我们看到的第三个问题可能最被低估，那就是验证。So, take a benchmark, let's say it has 500 multi-turn agent tasks, and you want to run it across 16 frontier models, that costs tens of thousands of dollars per sweep, and every time that a new model launches, which is basically every week or two at this point, you have to do the sweep again in order to keep your benchmark current, otherwise it becomes stale. And the third problem that we see, which is probably the most underappreciated, is verification.

所以，行业里的所有基准测试都是这样：每个人都运行自己的测试框架，报告自己的分数，并构建自己的自报式基准测试实现。不同实现得出的数字差异很大，这主要是基础设施造成的，而不是模型质量本身存在实际差异。So, everyone like all the benchmarks in the industry, everyone runs their own harnesses, they report their own scores, they build their own like self-reported benchmark implementation, and the numbers are wildly different across different implementations. And this is mostly because of infrastructure, not because of actual differences in model quality.

所以，如果你使用不同的脚手架、以不同的方式调用工具，或者只是把输入以不同的方式序列化后传给模型，最终得到的基准测试分数就会有所不同，即使使用的是完全相同的基准测试和模型。所以，我觉得最后一点，也就是验证这一块，可能是最抽象的。嗯，我想稍微深入探讨一下这一点。So, if you have different scaffolding, if you have different ways that you're calling tools, if you input if you just serialize the input into the model in different ways, you end up with different benchmark scores, even if it's the exact same benchmark on the same model. so, that last point, around verification, I think, is probably the most abstract. Uh so, wanted to deep dive into that a little bit.

复现并审计 65 项基准测试Reproducing and auditing 65 benchmarks

因此，为了真正了解基准测试行业的现状和各种问题，我们 Kaggle 花了过去两个月的时间复现了 65 个不同的行业基准测试：我们重建了所有这些基准、逐一运行，并逐条阅读了每一条执行轨迹，以真正弄清楚问题出在哪里，从而确定我们需要构建什么样的基础设施来解决这些问题。我们在这些不同的基准测试中发现了大量问题。这里列举其中一些，让你大致了解各种问题。so, we, in order to really understand the state of the benchmark industry and all of the different problems, we at Kaggle spent the last 2 months reproducing 65 different industry benchmarks, and recreated all of them, ran all of them, and read every single trajectory trace by trace to really understand what the problems were so that we could figure out like what the infrastructure we wanted to build was to fix it. And we saw tons of issues across all of these different benchmarks. So I'm listing a few of them here just to give you an idea of all of the different problems.

因此，在一个基准测试中，我们发现受评估的模型实际上在评估容器中找到了另一个模型的 API 密钥，随后调用了那个模型，并将它的回复作为自己的回复提交给评估。So for one benchmark we found the model that was being evaluated actually found the API keys in the eval container for a different model and then called different model and then use that response and submitted it as their response to the eval.

其次，大多数人在构建基准测试时，并没有真正把被评估的智能体和负责评分的智能体分开。它们共用同一个容器，而现在的模型很擅长入侵容器。我们发现，模型基本上会修改评分脚本，把分数改成自己想要的分数——也就是 100%——然后让自己通过。第三个原因纯粹就是统计方差。Second, most people when they build their benchmark, they don't actually separate the agent being evaluated from the agent being graded. They share the same container and models are really good at hacking their container now. So we found models basically modified the grading script in order to give them whatever score that they wanted which was 100% and let themselves pass. The third is just pure like statistical variance.

因此我们发现，在全部 65 项基准测试中，我们都进行了多次运行。结果发现，即使是在同一套基础设施上，多次运行完全相同的基准测试，得分也会出现高达正负 20% 的波动。再强调一下，使用的是同一个模型、完全相同的测试工具链和完全相同的基础设施。这反映的只是测试工具链本身或基准测试设计的不稳定性。So we found for all 65 benchmarks we ran all of them multiple times and we found that running the exact same benchmark even on the same infrastructure multiple times produced variance of up to plus or minus 20% in the score. And again, this is the same model on the same exact harness on the same exact infra. So this is just reflective of instability in the harness or the benchmark design itself.

我想大家都熟悉 Hugging Face 事件。我们发现，有些模型会尝试在 Hugging Face、Reddit 或 Stack Overflow 等不同论坛上查找答案。而这一点让我们非常惊讶。每个 harness 实际上都有各自默认的推理力度。所以，如果你使用默认设置而不进行调整，最终比较的其实是推理力度不同的模型。I think everyone's familiar with the hugging face incident. So we found instances of models basically trying to look up the answers online in different forums like hugging face or Reddit or Stack Overflow. And then this one we were very surprised by. So, each harness actually ships with its own default reasoning effort. so if you use the default without setting it, then you actually end up comparing models at different reasoning efforts.

所以，如果你没有进行实际配置，确保这是一次正面对比，那么可能会出现一个模型以高设置运行，另一个模型以低设置运行的情况。我们在复现时经常发现，人们没有这样做，因为我们的得分实际上与原始基准测试论文中的得分持平，但我们发现分数本身存在问题。So, you might have one model that's running on high and one that's running on low if you if you don't actually configure it in order to make sure that it's a head-to-head comparison. and a lot of the times when we were recreating it, we found that people did not do this because our scores actually matched parity with the original benchmark paper, but we found issues with the score itself.

标准化基础设施与基准测试支持Standardized infrastructure and benchmark support

因此，鉴于行业中存在这些问题，Google DeepMind 和 Kaggle 正在携手解决。为此，我们会采取几项措施。第一，我们正与企业和研究人员合作，围绕真实工作流程构建基准，这些工作流程在现实世界中具有经济可行性。为推动这项工作，我们向任何希望构建基准的人免费提供算力和前沿模型推理服务。and so given all of these problems in the industry, Google DeepMind and Kaggle are are partnering to solve this. So, we're doing a few things here. Uh the first is we are partnering with companies and researchers to build benchmarks around real workflows economically viable real real-world workflows. and to make this happen, we are offering free compute and frontier model inference to anyone who wants to build a benchmark.

我们将在第四季度提供 500 万美元的资助，用于支持基准测试的创建，或研究如何改进基准测试领域，因为这个领域还处于非常早期的阶段。第三，我们将为你安排一位来自 GDM 的前沿部署工程师，帮助你共同设计评估，确保方法确实严谨。第二，我们在上一张幻灯片中已经对此谈了很多，但这些问题中的许多都只是基础设施栈各个层面的问题。We're giving out $5 million in grants in Q4 to fund any benchmark creation or research into fixing this space of benchmarks given how early a space it is. and third, we will we will provide you a forward deployed engineer from GDM to help you co-design your eval to make sure the methodology is actually rigorous. Second, we talked about this a lot on the last slide, but a lot of these problems are just issues all up and down the infrastructure stack.

因此，我们在 Kaggle 构建的是标准化基础设施，确保每个评分都可审计、高质量且可重复。这包括对每次运行进行沙箱隔离、记录代理调用并公开每次模型调用的追踪记录，以及采用容器化评分。and so what we're building at Kaggle is standardized infrastructure so that every score is auditable, high quality, and repeatable. So, this includes sandboxing every run, having logged proxy calls and open traces for every model call, and containerized grading.

最后，我们之所以特别要做这件事，是因为我们拥有相关的专业经验。Kaggle 在设计严谨测试、防止人类作弊方面已有 15 年经验，而我们现在正将这些经验应用到模型评估领域。我们还引入了 GDM 研究人员的专业知识，帮助我们设计所有这些真实场景基准。最后，我们实际上还与 Epoch 等独立研究机构合作，对我们平台上的所有基准进行审计，这样你就不必只听我们的一面之词。And then lastly, the reason why we specifically are doing this, is for our expertise. so Kaggle has 15 years of designing rigorous tests and preventing cheating by humans, which we are now applying to the model evaluation space. we're bringing GDM researcher expertise to help us design all of these these real-world benchmarks. and lastly, we're actually partnering with independent research orgs, like Epoch to audit all of the benchmarks on our platform, so you don't just have to take our word word for it.

所以，到目前为止，我希望我已经让你相信基准测试有多重要。不过你可能会想：作为一家公司，在还有其他优先事项和事情要做的情况下，为什么还要自己构建基准测试？这里有两个重要原因。第一个是，它实际上能让你们在内部更快地进行开发。So, at this point, I hope I've convinced you, how important benchmarks are. but you're probably wondering, why should I build a benchmark as me as like a company with other priorities and other things to do? and there's a two two big reasons. the first is it actually allows you to build faster internally.

因此，拥有一个基准和一套严谨的评估体系，就能让你轻松判断：修改提示词或测试框架后，是否破坏了某些功能，或者产品是否有所改进。其次，如果你想将基准对外发布，就能鼓励各家实验室针对你的用例不断优化。大多数实验室都在寻找真实世界的基准。一旦实验室开始针对你的基准不断优化，就会投入价值数万亿美元的算力来改进基础模型，以满足你的特定用例需求，从而让你的产品随着时间推移变得更好，为用户提供更优体验。So, having a benchmark and having a rigorous evaluation suite, allows you to easily tell when when you model modify your prompt, when you modify your hardness, whether you've broken something or how you've improved your product. and then the second is if you then want to publish your benchmark externally, then labs will be encouraged to actually hill climb on your use case. like most of the labs are are looking for real-world benchmarks. and once the labs want to hill climb on your benchmark, then they'll pour trillions of dollars in of compute into improving the base models, for your specific use case, so that your product gets better for your users over time.

接下来，我将把时间交给 Link Alpha，由他介绍 Mandate Bench 的案例研究。Mandate Bench 是我们在 Kaggle 上构建的首批基准测试之一。and with that, I will pass it over to Link Alpha, to talk through a case study of Mandate Bench, which is one of the first benchmarks that we built on Kaggle.

投资授权基准与投资研究Mandate Bench and investment research

很好。大家好，我是 Suyo，Link Analytics 的 AI 负责人。我们创建了一个名为 Mandate Bench 的基准测试，它是与 Google DeepMind 和 Kaggle 共同开发的。Great. Hi everyone. this is Suyo. I'm head of AI at Link Analytics, and we create we created a benchmark called Mandate Bench. It's built co-developed with Google DeepMind and Kaggle.

在开始之前，我先简单介绍一下我们公司。Link Analytics 为机构投资者打造 AI 研究智能体，管理资产超过 5 万亿美元的机构都在使用 Link Analytics。目前有 200 多位投资者在积极使用我们的产品。这里列出了一些信任我们的机构。So before start, let me share briefly about our company. So Link Analytics builds AI research agent for institutional investors, institutions with more than $5 trillion in assets under management using Link Analytics. And more than 200 investors use it actively. There are some of the firms that trust us I listed up here.

我们也是一个研究团队。我们已在 ICLR 或 ICML 等会议上发表了 13 篇论文，并与贝莱德、黑石、摩根大通等机构合作撰写。因此，我想先介绍一下现有的金融基准测试。大多数这类基准测试都会准备标准答案，并将其与智能体的回答进行比较。And we're we're also a research team. We have published 13 papers at the venue like ICLR or ICML with co-authored with the BlackRock, Blackstone, JP Morgan, etc. So I'd like to first go through the existing financial benchmarks. So most of these benchmarks prepare a golden answers and compare it with the agent answer.

但在现实工作中，两位分析师可能得出不同的答案，而且两者都可能是正确的。因此，现有基准测试采用的任务相对简短、简单。它们并不——它们并不涵盖分析师完整的一项工作，而只涉及其中的片段。But in real world task two analysts can reach two different answers and both could be right. So existing benchmark use relatively short and simple task. They don't they don't task a full unit of analyst work. So they only get the fragments of it.

因此，Mandate Bench 旨在做两件事。首先，它的评分标准会明确评估推理过程。其次，我们认真对待现实世界中真实的工作单元。对我们的分析师而言，这个工作单元从基金的投资授权开始，并从授权一路延伸到投资备忘录。So Mandate Bench aims to do two things. First, its rubric explicit grades the reasoning path. Second, we take the real world real unit of work seriously. For our analyst, the unit goes from the funds mandate or and from the mandate all the way to an investment memo.

那么，我们先来看看基准测试中的任务是什么样的。先从一个简单的问题开始：对冲基金的分析师是如何工作的？So, let's first look at what a task in the benchmark looks like. Let me start with a simple question. how does an analyst work at a hedge fund?

重建 Sonoco 并评估推理能力Rebuilding Sonoco and grading reasoning

没人会给分析师一个简单的任务。PM把基金的投资授权和一家公司交给分析师。比如，这项投资授权是：我们是一只事件驱动型基金，专注于并购交易，持有期为3到12个月。PM问：“好，这里有一家公司A，我们知道公司A将被公司B收购。”Nobody gives an analyst a simple task. The PM gave analyst the funds mandate and a company. For example, the mandate looks like we are an event-driven fund and we trade on mergers and acquisitions with a holding period 3 to 12 months. And the PM asked, "Okay, here's a company A and we know company A will be bought by company B.

“我们应该做多、做空，还是干脆放弃这家公司？另外，请给我一份投资备忘录，附上详细证据和推理过程。”这项工作轻轻松松就可能耗费分析师两千到两千二百个小时。这就是分析师工作的一个真正的基本单位。接下来，我们深入看一个任务示例。Should we go long, short, or just pass on this company? And please bring me a investment memo with detailed evidence and the reasoning behind it." And this can take an analyst hours, 20 to 22 hundred hours easily. And this is the one real unit of analyst work. So, let's go deeper into an example task.

这是一家名为 Sonoco 的美国包装公司。最近，Sonoco 收购了一家名为 Eviosys 的公司，还出售了其名为 TFP 的一项业务。Here's a company called Sonoco and it's a US packaging company. Recently, Sonoco bought about a company called Eviosys and it's also sold one of its business called TFP.

要对Sonoco进行估值，首先需要一个基本数字：销售额。但五个不同来源给出的销售额各不相同，从55亿美元到80亿美元不等。为什么？因为每个来源统计的Sonoco口径都不一样。To value Sonoco, you first need one basic number, which is a sales. But five different sources give the different sales numbers from 5 5 billion to 8 billion dollars. Why? Because each source counts a different different Sonoco.

所以，首先，智能体必须明白这些数字对不上。其次，它还没有……它必须查明原因。然后，然后，然后第三，也还有第二，它必须查明原因；第三，它必须重建 Sonoco，使其恢复到 2025 年 5 月时的情况，全年计入 Eviosys，且不包括 TFP。这样销售额将约为 75 亿美元。So, first, the agent has to understand the numbers doesn't match. Second, it hasn't it has to find out why. And and and third, and also second, it has to find out why and third, it has to rebuild Sonoco as it was in May 2025 with Eviosys for the full year and without TFP. This it will be about 7.5 billion dollars of sales.

而这只是熟练分析师需要梳理的全部推理过程中的一小部分。那么，我们该如何评估智能体的工作？智能体会提交一份长篇投资备忘录，并附上推理图；图中的每一项事实都应指向其确切来源的工具调用。And that is just a small part of the entire reasoning to skill analyst to work through. So, then how we grade the agent work? So, agent submits an investment memo long together with a reasoning graph in the graph and then every fact should point to exact two call it came from.

接下来，我们的前买方分析师会写下任何一份优秀备忘录都应包含的关键推理步骤。每一步都是图中的一个节点。首先，无代码代理必须还原公司在 2025 年 5 月的状况，弄清新新新业务真正赚了多少钱，扣除已出售业务的影响，并重新梳理债务。Then our former buy-side analyst write down the key reasoning steps that any good memo would include. Each step is a node in the graph. First, the no-code agent must rebuild the company it was in May 2025. And what the new new new business really earns and net out what was sold and rebuild the debt.

首先，我们向人力资源部询问时，也对这份备忘录进行了“假设测试”。例如，管理层仍承诺 EVOC 能节省成本。但是，如果销售方面的节省为零呢？这个决定会改变吗？If first that we ask HR, we also attacked the memo with the what if test. For example, management pro- promised cost savings from the EVOC still. But, what if the sales savings were zero? Would the call change?

备忘录是否对各个组成部分进行了清晰的推理说明？所以，所以，所以，备忘录可以得出不同的判断，而且两者都能得高分，只要每个判断背后的工作都做到位。所以，这是结果，你可以看到……的云图。这是初步结果，只在全部 50 项任务中选取的 10 项任务上运行。And does the memo explain this with clear reasoning about the moving parts? So, the So, the So, the memos can reach different calls and the both can score well as long as the work behind the each calls are there. So, this is the result and you can see the cloud of This is a preliminary result. Only runs over 10 tasks of 50 entire task universe.

Claude 5.5 得分最高。由于时间关系，我们跳到下一页。每个模型的表现都不一样。例如，Astra 6 非常擅长进行财务建模，而 Opus 5.5 在各项不同测试中的表现则非常均衡。And the cloud with 5.5 scores best. In the matter of time, let's skip to the next page. And every model behaves differently. For example, Astra 6 really good at making a financial modeling, while the Opus 5.5 is a balanced very well over the cross different test.

好吧，我回去找肖恩。Okay, let me get back to Shane.

初步结果与即将开展的基准测试Preliminary results and upcoming benchmarks

太棒了。那么，这就是 Mandate Bench 最初的 10 项任务。完整的 20/50 项任务基准测试将于下个月，也就是 2026 年 11 月推出。这只是未来几周内将在 Kaggle 和 DeepMind 上推出的众多其他基准测试中的一部分。呃，我们已经与多家领先的 AI 初创公司合作，共同构建不同领域的基准测试。awesome. So, that was the first initial 10 tasks of Mandate Bench. The the full 20 50 task benchmark will be launching next month in November 2026. and this is part of many many other benchmarks that are launching in on Kaggle and DeepMind over the next few weeks. Uh we've partnered with a bunch of leading AI startups to build benchmarks in different domains.

呃，我们上周推出了 HeyGen 的代码转视频基准测试。我们正在与 Octava、Quadrillion 和 Rehearsals 合作开发另外几个基准测试，涉及 meta 临床工作流、研究工作流和用户研究工作流。所以，如果现场有人有兴趣为自己的公司构建一个真实世界基准测试，我们非常乐意合作。正如我之前提到的，我们将提供免费的模型算力、JDM 的全职人力支持，以及 500 万美元的资助。Uh we launched HeyGen's code code to video bench last week. we're working on a couple other benchmarks with Octava, Quadrillion, and Rehearsals on meta clinician workflows, research workflows, and user research workflows. so, if anyone in the crowd is interested in building a real-world benchmark for your company, we're we're very interested in collaborating. as I mentioned before, we're we're providing free model compute, FTE support from JDM, and $5 million in grants.

所以，请联系我们。我们非常期待与你们合作。非常感谢。so, please reach out to us. We we'd love to work together. thank you so much.