什么时候自托管 AI 模型比购买 API 更划算：imec 的任务级基准研究

简要描述Brief Description

何时运行自己的 AI 模型比付费使用 API 更划算——Steven Latré，imecWhen Running Your Own AI Models Beats Paying for APIs - Steven Latré， imec

目录Table of Contents

• 简介
  Intro
• 芯片实验室为何研究 AI 工作负载
  Why a chip lab studies AI workloads
• AI成熟度阶段
  The stages of AI maturity
• 掌控自己的技术栈，会大幅拓展设计空间。
  Owning your stack blows up the design space
• 基准测试研究
  The benchmarking study
• 测试的设置方式
  How the tests were set up
• 各模型的准确率
  Accuracy by model
• 添加硬件成本
  Adding the hardware cost
• 并发：可容纳多少用户
  Concurrency: how many users fit
• API 与租用 GPU 和购买 GPU 的比较
  API vs. renting vs. buying GPUs
• 测试框架与令牌使用
  Harnesses and token use
• 接下来是什么？
  What's next

简介Intro

我叫史蒂文，在 IMEC 负责领导人工智能部门。对于不了解 IMEC 的各位来说，我们在人工智能领域其实不太出名，但在半导体领域非常有名。我们多少被视作一颗鲜为人知的瑰宝。So my name is Steven. I'm leading the AI department at IMEC. For those of you who don't know what IMEC is, we're not that well known actually in the AI space, but we're very well known known in the semiconductor space. Been seen a little bit as the hidden gem.

我们已经存在了四十多年。我们通常做的事情是：拥有世界上最先进的研发洁净室，并与合作伙伴共同研发下一代半导体。从这个意义上说，我们在半导体领域产生的影响相当巨大。我认为，如今投入应用的几乎所有晶体管技术，都是我们在过去四十年里发明的。所以，对我这个来自比利时的人来说，我们可以说是比利时一颗鲜为人知的明珠。We exist already for about more than four decades and what we typically do is we have the most advanced R&D clean room in the world and we actually run together with our partners next generation semiconductors in that sense. And so actually the impact that we've had in that semiconductor space is pretty huge. I think every single transistor technology that was more or less that is deployed now has been invented by us in the last four four decades in that sense. So in that sense pretty much a hidden gem in Belgium where I'm where I'm from.

芯片实验室为何研究 AI 工作负载Why a chip lab studies AI workloads

那么，为什么一个半导体极客会谈人工智能呢？其实，我们做的远不止晶体管技术。我们的工作涵盖系统，甚至也深入到工作负载层面。我负责的正是如何在这个层面上，把人工智能工作负载与基础设施连接起来。So why is a a semiconductor nerd talking about artificial intelligence? Well, we do way more than just transistor technology. We go all the way up to systems. We go all the way up to the workloads as well and that's where I'm responsible from for where we're trying to connect what AI workloads does to infrastructure in that sense.

一方面，是因为过去这几年我们也亲身经历了这一点。当然，在 IMEC 内部，我们也在大量运用 AI，以实现智能体工作流。On the one hand because we've lived that as well the last few few years. Of course within IMEC we're also working very much with AI to kind of realize agentica flows as well.

AI 成熟度的阶段The stages of AI maturity

我认为，在过去几年里，我们很多人都经历了 AI 成熟度的不同阶段。2024、2025 年左右，我们一开始觉得，每个前沿模型都得用最新、最强的版本，好的还不够，我们需要越来越多。当然，与此同时我们也意识到，定价开始发生变化：从更多采用订阅制，转向更多采用基于 token 的定价。And what we've experienced I think a lot of you of us have experienced at the last few few years is kind of a different stages of AI maturity. We kind of started off in 2024 2025 thinking that we needed the latest and the greatest of every single frontier model and that the best wasn't good enough and that we needed more and more. But of course at the same time what we realized is that prices started evolve. We moved from more subscription based pricing to more token-based uh pricing.

然后，就像去年一样，我们越来越多地转向模型路由，尝试为当前任务找到最合适的模型类型。我想，过去一年左右，我们大家也都在做类似的事情。我们实际还发现，无论是在公司内部，还是在我们接触的客户那里，这个过程还有其他阶段。从这个意义上说，模型路由或许只是实现良好优化、为合适任务匹配合适模型的第一个容易入手的环节。And then like the last year, we moved more and more towards model routing where we try to find the best type of model for the task at hand. Things that I think all of us have been have been doing the last year or so as well. What we're actually seeing as well with both within the company but also with the customers that we're that we're talking about is that there's other stages to that as well. Model routing is in that sense maybe the first just low-hanging fruits in getting a good optimization of finding the right model to the right task.

随着 token 消耗量不断大幅增长，我们越来越明显地看到，人们也需要做其他事情。而做其他事情，基本上意味着要在那个单元中掌控自己的技术栈。大致就是这个意思。我们看到客户正在这样做，我们自己也在走这条路：一开始，我们只是关注封闭 API，尽可能利用能用的东西。第二阶段对我们来说则是：好吧，看看如何优化成本。We're seeing more and more as token consumption really really grows that there's a need for doing other things as well and doing other things basically means taking control of your own stack in that cell. That sense. So we're seeing customers and we're as on that path as well moving from well, first of all, we started with just looking at closed APIs, just grabbing what we could do. Then second phase was for us more, okay, let's see how we can do cost optimization.

现在我们已经进入第三阶段，重点是如何托管开放权重模型，以及从长远来看如何租用我们的基础设施，甚至可能购买基础设施。We're very much ourself now in the third phase where we're looking at how can we host open weights models and how can we lease our infrastructure in the long run maybe even buy our infrastructure as well.

拥有自己的技术栈会大幅拓展设计空间Owning your stack blows up the design space

我特别想谈谈整个过程中的第三和第四阶段，看看它们实际上意味着什么，因为运行一个 API，或者说仅仅提供一个用于消耗 token 的 API，确实相当简单，做起来也很容易。但一旦进入第三和第四阶段，设计空间就会突然变得大得多。第一阶段，我们只需要考虑：什么类型的模型适合达到所需的模型准确率，或者适合处理特定复杂度的任务。但一旦开始拥有自己的基础设施，突然就还得操心很多其他事情。I particularly want to talk about this third and fourth phase of all of this and see what it actually would would mean because running an API or just having an API for to for token consumption is pretty damn straightforward. It's pretty easy to do but the moment you move towards this third and fourth phase, suddenly the design space starts blowing up as well. In the first phase, we're just worried about what is the right type of model for the right type of model accuracy or for the right type of task complexity. But the moment you start owning your your infrastructure, you suddenly need to worry about a whole lot of other things as well.

我该选择哪种模型？我怎么才能相信这个模型？嗯，你需要考虑测试框架，但也需要考虑很多硬件配置，比如我该为此采用什么样的硬件架构？CPU 和 GPU 的比例应该是多少？What kind of model am I going to take? How can I trust this this model? Uh, you need to think about harnesses, but you also need to think about a lot of hardware configurations as well, such as what is the right hardware architecture that I would use for this? What is the CPU to GPU balance?

我该如何进行 KB 缓存优化，等等这些事情？这也是我们希望定位自身的方向，也是我接下来会继续推进的方向。当然，在这个对模型和 harness 已经非常熟悉的领域里，我们也确实看到了从模型和 harness 到基础设施之间存在的差距。How can I do KB cache optimization and uh, and all of that. And so that's where we want to position ourselves as I am going forward as well, where of course here in a in a world that is very much aware of models and harnesses, but we do see very much also a gap from those models and harnesses to that infrastructure as well.

基准测试研究The benchmarking study

嗯，我们过去几个月所做的工作，也是我今天要介绍的内容，是一项我们在过去几个月开展的基准测试研究。我们进行了数千次测试，真正量化了从使用 API 转向自建基础设施究竟能带来什么。具体来说，是从智能体 AI 的角度来看。如果你有一项智能体 AI 任务，或者对某项特定智能体 AI 任务的性能有要求，那么从使用 API 转向自建基础设施，对你来说成本会是多少？Um, so what we did in the last few months and what I'm here to present today is a kind of benchmarking study that we did in the last few months. We did uh, really thousands of runs to really really quantify what does that move from going from an API to really infrastructure really uh, give you. And so really from an Agentic AI perspective. So if you have an Agentic AI task, if you have a specific Agentic AI task performance, what would be the cost for you if you move from an API towards your own uh, infrastructure?

这也意味着从更偏向按 token 计费的订阅模式，转向真正考虑总体拥有成本。还有，嗯，那么应该如何配置基础设施呢？具体来说，哪些类型的执行框架值得考虑？需要什么样的硬件？So that also means moving from a more token-based cost subscription to really kind of total cost of ownership. And also um, how should you then configure the infrastructure? Really from what kind of harnesses are relevant? Uh, what kind of hardware would you need?

那么，所有这一切所采用的推理引擎是什么？我们是怎么做到的？What is the inference engine of choice uh, for all of this? So how did we do this?

测试的设置方式How the tests were set up

嗯，我们确实想在任务层面进行比较，所以我们专门考察了长程任务，因为长程任务确实是最理想的场景——如果你有自己的基础设施，并且想要自己掌控它的话。如果交互性很强，规模经济可能就不一样了；但对于长程任务，我们认为这是第一个理由，或者说第一个转折点：掌控自己的 AI 技术栈确实很有意义。因此，我们选取了一个具体的基准，即 SWE-bench Pro 中用于编程辅助的一系列基准任务。接着，我们对几种截然不同类型的编程执行框架进行了验证。Uh, we really wanted to compare at a task level, so we looked at uh, long horizon tasks specifically because long horizon tasks are really the perfect uh, um, if you really have your own infrastructure that you want to uh, to own. If you do do it very interactive, it might not be the economies might not be the same, but for long horizon tasks, we think that's the first reason or the first pivot point where um, taking a hold of your AI stack is really relevant. So we took one specific one being um, uh, the SWE-bench pro uh, tasks, which is set of benchmark tasks that exist um, uh, for coding assistance. And what we did is we kind of validate or kind of very different types of coding harnesses.

我们为此采用了 Codex by 和 Cloud Code，它们大致算是目前最受欢迎的三种编程辅助框架。我们也尝试了不同类型的模型，从传统的标准 API 环境开始，同时还考察了各种不同复杂度和参数规模的开放权重模型。由于这场演讲的主题是如何将这些方案迁移到你自己的基础设施上，我们当然也会将其与相应的硬件架构相匹配。We took Codex by and Cloud Code for this, which are more or less the three most popular harnesses out there for coding assistance. And then we kind of vary the different types of models that we could use on this as a as well. Starting with a traditional standard API environment, but then also looking at a wide range of different open weight models with different types of complexities and parameter sizes as a as well. But as this is a talk about translate translating this towards your own infrastructure, we of course also map this to the right hardware architecture at hand as well.

我们一直在努力寻找合适的架构类型，既能满足需求，又足以真正实现规模化，并支持多个用户同时并发使用。本质上，我们提高了硬件的复杂度，也提高了模型的复杂度，同时还考察了整体的应用性能。那么，你的任务解决率如何？延迟是多少？成本又是多少？Always trying to find the right type architecture that kind of is still sufficient enough to really do this at scale and it where you really have where you can have multiple concurrency concurrent users on on this. So in essence, we kind of increased the hardware complexity, increased the model complexity, but at the same time also looked at really the application performance of all of this. So what is the kind of the resolve rate of your task? What is the latency and what is the cost?

总而言之，总而言之，这次我们进行了超过一千次不同的测试。我们搭建了一套完整的基准测试流程，我想大致带你梳理一下我们从中得到的各种结果。All in all in all this was more than a thousands of different runs that we did. We set up a full benchmarking pipeline and I kind of want to run you run you down with the different results we got out of out of this.

各模型的准确率Accuracy by model

我们先从最明显的一点说起。不同的模型、不同的模型复杂度，以及不同的模型参数规模，实际上都会导致这些任务的准确率有所不同。我们可以把模型大致分为三大类。第一类是较小的模型，我认为参数量在几千亿以内的都属于这一类，Qwen 和 DeepSeek 就是其中的代表。So let's start maybe with the most obvious one. Kind of different models, different model complexities, different model parameters sizes actually give you different accuracies of these of these tasks. We could actually really classify three main categories of models. The more smaller ones and with smaller ones I would say up to a few hundred billion of parameters where Quen and DeepSeeker are part of.

它们的解决率大概达到 32.5% 到 40%，也就是说，这些任务确实能够被准确、正确地完成。第二个，比如 GLM，嗯，表现确实达到了 Opus 级别的水平。然后是几个月前发布的 Kimik 3，嗯，老实说，它的表现也确实非常出色，可能部分原因是它的一些内容也在训练集中，但结果依然相当显著，解决率达到了 86%。不过，这些数据也不是你能轻易在 Artificial Analysis 上找到的。They kind of got a got into a resolve rate of about 32 5 to 40% of tasks that really were able to achieve accurately correctly. Second one like GLM, um, was really at Opus level, uh, uh, range and then Kimik 3 that came out a few months ago, um, was, uh, really was really crushing it to, in all honesty, also probably because part of it has been probably in the training set as well, but still the results were quite, uh, substantial. Had a resolve rate of 86%. Now, all of this is not something that uh, is is something that you can easily find on artificial analysis as well.

这并没有真正为你提供太多额外的见解。This doesn't really give you a lot of uh, uh, additional insights.

加上硬件成本Adding the hardware cost

我觉得，如果再把基础设施和硬件这一层面也考虑进去，就更有意思了。在这方面，我们确实能看到相当大的差异。我觉得有意思的是，比如把 Qwen 和 DeepSeek 作比较，它们的整体应用性能大致处于同一水平。不过，如果再看看运行 Qwen 和 DeepSeek 分别需要什么样的硬件，你会发现差距有四倍之多；租用 GPU 的成本也更高。What I think is really more interesting is if you add the infrastructure and hardware angle to it as well. And there we do see quite a bit of differences. What I think is interesting is if you compare for example Qwen with DeepSeek that all in all the application performance of it is more or less on the same, uh, level. Although if you then look at what kind of hardware you would need for Qwen and DeepSeek, you're looking at a factor of four and in terms of renting GPUs that are is more expensive.

嗯，同样，如果你只是从 GLM 模型转向 Kimik 3 模型，准确率就能提升约 20%；但总的来说，租用 GPU 的成本只会增加 15%。这说明了什么？基本上，这说明，如果你要转向开放权重模型，就非常、非常有必要同时考虑其硬件限制，也要看看这一切的实际成本。这里显然不存在一条线性曲线。很多时候，它更像是一个阶梯函数：你可能会选择某个特定模型，而换成一个稍微更好或稍微性能差一些的模型，也可能会对硬件架构的选择产生巨大影响。Um, similarly if you just make the move from, um, a GLM model to, uh, a Kimik 3, uh, model, it gives you a bump of about 20% in accuracy, but all in all it's just giving you only a 15% increase in cost in renting GPUs as well. So what does that tell you? It basically tells you that if you move this make this move towards open weight models that it's super super important to kind of look at the hardware constraints of this as well and also look look at the actual, um, costs of, uh, of all of this and there's definitely not a linear curve to be found here. So often it's just a step wise function where you where it might be that you're choosing a specific, uh, model and that just taking a slightly better or slightly, uh, less performant model would have a huge impact on the hardware architecture of choice as well.

所以这是其中一种，主要是从应用程序性能的角度来看。So that's one and that's looking at mainly from an application performance perspective.

并发：能容纳多少用户Concurrency: how many users fit

嗯，我们换个角度来看，那就是并发。嗯，因为使用 API 时，你可以获得一定的 token 消耗性能。你大致知道 token 延迟会是多少，也知道 token 吞吐量会是多少。一旦转向更偏基础设施的解决方案，你突然就还得考虑并发问题。Um, let's take a different angle to it and that's concurrency. Um because when you're using an API, um you can um get a certain performance of token consumption. You know, you know more or less what your token latency is going to be, and you know what your token's throughput is going to be. The moment you make the move towards a more infrastructure-based solution, suddenly you need to worry about concurrency as well.

换句话说，从延迟角度来看，我能可靠地在同一台机器上支持多少用户，而不会让系统性能彻底恶化？这既涉及内存大小，也涉及延迟。内存方面，大多数系统都受内存限制，这意味着模型参数必须能装进内存里，用户的 KV cache 也一样。随着使用系统的用户越来越多，对 KV cache 的需求也会越来越大。In other words, how many users can I reliably uh put on the same uh machine without the system completely deteriorating from a latency perspective in that sense. There's a a memory size to it, but there's also a latency uh perspective to it. Uh memory means that uh on the one hand, the most systems are memory bound, and it means that on the one hand, your model parameter size need to fit your your your memory, but also your user's KV cache as a as well. And as you have an increasing number of um of users uh using this the system, your KV cache uh requirement will become bigger and bigger as a as well.

所以，我们做的是尝试查看我刚才已经展示给你们的不同类型的解决方案，并看看在那个系统中，我们实际能够很轻松地容纳多少并发用户。结果相当令人惊讶，或者至少对更大的模型来说，可能也没那么乐观。请记住，这些不同的模型各自都有最适合它们的硬件架构。但如果你再看看我们能够实现的舒适并发水平，就会发现它大幅下降了，即使使用更大的硬件架构也是如此。So, what we did is we kind of tried to look at the different types of solutions I just already showed you, and look at how many concurrent users we were actually able to fit really comfortably in that uh that system. And the results were pretty um uh surprising, or at least also maybe not that positive for really the bigger models as a as well. So, bear in mind, each of these different models have their own hardware architecture that is uh that is optimal for it. But if you then look at the comfortable concurrency levels that we were able to get, it really really dropped down very significantly, even with bigger hardware architectures as um as well.

而且在所有情况下，我们都比基准慢得多，也就是 Cloud Code。顺便说一句，我觉得你们已经都弄明白了。嗯，实际上，我谈到的每个主题都有一篇详细的博客文章。每个二维码也都能带你去看相关内容。And in all cases, we were much slower than uh uh our baseline, which was uh Cloud Code in that sense. By the way, I think you figured it all out already. Um we actually have a detailed blog post on each of the topics I'm I'm talking about. Every QR code actually leads you to that as um as well.

所以，这在某种程度上说明，在这个意义上，并发能力可能实际上甚至比准确率更重要。准确率提供了一个基准，但即使是非常大的模型，尤其是考虑代码辅助这类超长时程任务时，运行 Kimi K3 模型也只能支持大约四个用户同时稳定使用，而且延迟仍然慢了三倍。So, this kind of shows you that uh concurrency is actually probably even more important than accuracy in the in this sense. Accuracy gives you the kind of baseline, but then even for the really big models and even if especially if you look at very long horizon tasks like code assistance, running a Kimi K3 model only gives you about four concurrent users that can really run reliably on a on this and still the latency is still a factor three slower.

API 与租用 GPU 与购买 GPU 的比较API vs. renting vs. buying GPUs

因此，我们所做的是一项全面的优势区间分析，探讨在哪些情况下继续使用 API 确实最合适；在哪些情况下，租用各家模型提供商的 GPU 会更合理；又在哪些情况下，购买设备、投资自有基础设施并在本地部署会更有优势。当然，转向其他方案还可能有很多原因，但如果从总体拥有成本的角度来看，不深入讨论细节的话，我们发现，硬件架构和系统的规格越低，从 API 转向自有基础设施就越不划算。So, what we then did is a kind of full sweet spots analysis of okay, in which cases is it really interesting to stick to your API usage? In which cases would it make sense more for you to kind of move to renting out GPUs with the various model providers that are out there and in which case would you really go for would it be more interesting for purchasing and really investing in your own infrastructure and building it out on prem. Of course, there can be a lot of other reasons as well to make that switch but if we look at it from a total cost of ownership perspective. So, without going into too much details what we saw there is that the lower your hardware architecture system actually is, the worse it is to move from API to your own infrastructure.

我们看到，如果你确实使用的是较低档次的模型，因此也只需要较低档次的硬件架构，那么实际上你需要大约 89% 的 GPU 利用率，也就是说，GPU 需要有 89% 的时间满负荷运行。嗯，同样地，如果你转向更大型的硬件系统——我刚才说错了，应该是更高端的硬件系统。比如说，使用 8 张 B200，你只需要 15% 的 GPU 利用率就够了，这非常现实；而 89% 的利用率在这种情况下完全不可行。如果把这些情况平均来看，并以 GLM 类应用为例，我们确实会发现，这正是从 API 转向基础设施变得更有吸引力的最佳点，或者说门槛。We saw that if you really are at the lower tiers of models and as a result also at the lower tiers of the hardware architecture that you would need, you actually need an GPU utilization of about 89% so that means that your GPU needs to run at capacity at 89% of the of the time. Um, similarly if you actually move to more bigger hardware systems and I made a mistake there it should be a higher hardware systems. So, for example an 8 times B200s, you're okay with just 15% of GPU utilization and that's really realistic and 89% is just completely infeasible in that sense. If you then kind of average it out and if you look at for example GLM class applications, what we do see is that's really the sweet spot or the threshold of when moving from API to infrastructure makes it more interesting as a as well.

在我们专门针对代码辅助这一应用所做的全面总拥有成本分析中，我们发现成本节省大约达到 24.5%。这主要是从基础设施角度来看。And in the full total cost of ownership analysis that we did specifically for this application being coding assistance, we really saw that you get into a kind of 24 5% of cost saving. So that's really infrastructure perspective.

测试框架与令牌使用Harnesses and token use

我们把重点转到 harness 上。我们也调整了 harness 的数量，并观察应用在这三种不同 harness 下的性能。乍一看，应用性能其实没有太大变化，这可能有点令人意外。我们给每种 harness 的每项任务，它们基本上都完成了；而且对于不同类型的模型，我们在这个基准测试上得到的应用准确率也大致相同。Let's kind of switch it over to harnesses. We also varied the amount the harnesses as a as well and looked at what the application performance was for those three different harnesses. And on the first glance, the application performance actually didn't really change that much, which was maybe kind of surprising. Every single task we gave every single harness we gave you more or less got it and every single for different kind of models, we kind of got into the same kind of application accuracy on this on this benchmark.

乍一看，你可能会说，嗯，运行框架其实完全没有影响，因为它能把事情做好。如果你只通过 API 使用，这或许确实如此；但如果你转向使用自己的基础设施，并且需要为 GPU 使用付费，情况就完全不同了，因为 GPU 的实际运行时间确实很重要。我们发现，不同类型的运行框架对这次任务中的 token 消耗量有很大影响。具体来说，在 Codex 的情况下，使用 GLM 时实际生成的 token 数量只有一半。因此，使用这些 GPU 的实际运行时间也缩短了一半。So at the first glance, you could say, well, the harness actually doesn't have an impact whatsoever on that because it gets the job done. Well, that might be the case if you're going for an API only usage, but it's completely different if you're moving towards your own infrastructure and you actually pay for the GPU usage because then the wall time of your GPU really is is relevant. And what we saw there is that different kinds of harnesses actually had a huge impact on the amount of token consumption we had in the in this and then specifically in the case of Codex, we had for a GLM only half of the of tokens that were actually being being generated. And as a result, also the wall time of using these GPUs actually was half of that as well.

这意味着，尽管这些测试框架实际上都能让你获得合适的应用性能，或者说大致相当的应用性能，但最终你的 GPU 费用也会只有其他测试框架的一半。因此，这也非常清楚地表明，在这种情况下，测试框架与模型的正确搭配或许不会影响应用性能，却确实会影响 GPU 的总体拥有成本。So that means that although the harnesses actually gave you the right type of application performance or more or less a similar level of application performance, the GPU bill you would in the end have would be half of the other harnesses as well. So it really shows very clearly that in that case also the right harness to model choice might not have an impact on application performance, but it has really an impact on your total cost of ownership for your GPUs as a as well.

接下来是什么？What's next

所以，这只是我们过去几个月里做的第一项相关研究。我们接下来要往哪走？我觉得这里有很多东西值得深入探讨。很明显，把它从 API 转向基础设施，并不是一个非常简单或线性的选择。So, this is just the first study that we're that we did in the last few few months on this. Where are we going with this? I think there's a lot to unpack. It's very clear that moving this from API to infrastructure that is just not very simple or very linear choice.

我刚才展示的是单一应用——代码辅助——的情况，但真实的企业环境往往还会混合使用许多不同的应用，这实际上会让事情变得更加复杂。我们也看到，运行框架、模型和基础设施之间存在非常复杂的关系。而且我还没谈到其他问题，比如量化方面我们该怎么做？And I have showed this now for one single application being coding assistance, but a real enterprise mix would often have a combination of a lot of difference of applications as well. And that actually complicates things even even further. We also saw that there's a very complex relationship between harnesses, between the models, between the infrastructure. And I haven't touched upon other things like what do we do in terms of quantization?

我们也应用了诸如推测解码这样的特定优化，但还没有对其进行详细配置。那么，这些优化会产生什么影响？第三个维度当然是，这一切也在不断变化。新的模型不断推出，新的硬件架构选择也不断涌现。What is the impact of specific optimizations like speculative decoding that we applied as well, but that we haven't configured into to detail. And the third dimension is of course that it's very much a constantly moving target as well. New models are being being launched. New hardware architecture choices are there as well.

所以，我们实际上正在进一步完善已有的一套基准测试流程。我们也在将其扩展为一套完整的基准测试数据集，之后会向所有人公开分享，并努力持续跟踪这个不断变化的目标。但与此同时，我们无法独自完成这件事。我们是一个研究机构。So, we're actually building out a kind of benchmarking pipeline that we already have. We're building this out to a full benchmarking data set as well that we're going to share publicly for everyone and trying to keep track of this moving target moving moving forward. But at the same time we cannot do this alone. We're a research organization.

我们是一家非营利性研究机构，对我们来说，最重要的是努力把整个生态系统整合起来。我认为，软件和硬件领域也应该更加紧密地结合。因此，我们实际上正在发起一个联盟，将对这方面感兴趣的人聚集起来，包括软件领域中关注硬件优化的人。我们还会把他们与现有的客户群体联系起来，其中包括新老芯片供应商和基础设施供应商。We're a non-profit research organization and the biggest important part for us is trying to bring that ecosystem together. And I think that the software world and the hardware world deserves to be much closer together as well. So, we're actually launching a kind of alliance to bring people who are interested in this part, software people from the software who are interested in hardware optimization. We're bringing this together with our existing customer portfolio of new and existing chip providers and infrastructure providers as well.

如果你对此感兴趣，欢迎在这次演讲结束后联系我。非常感谢。And if you would be interested in this, uh free to reach out, uh after this uh this talk. Thanks a lot.