不到 500 美元训练金融智能体：4B 专用模型如何胜过 235B 模型

简要描述Brief Description

小模型，大成果：以不到500美元训练一个金融智能体——Charles Dickens，Snorkel AISmall Models， Big Results: Training a Finance Agent for Under $500 — Charles Dickens， Snorkel AI

目录Table of Contents

• 简介
  Intro
• 专业化有时胜过规模化
  Specialization can beat scale
• 关于 Snorkel AI
  About Snorkel AI
• 与加州大学伯克利分校合作
  Working with UC Berkeley
• 两个模型的故事
  A tale of two models
• 专才，而非通才
  Specialists, not polymaths
• 从 10-K 文件构建 FinQA
  Building FinQA from 10-K filings
• 三层验证
  Three layers of verification
• 模型失效之处
  Where models fail
• 使用 rLLM 进行训练
  Training with rLLM
• 训练环境
  The training environment
• 500 美元以下的培训
  Training for under $500
• 结果：4B 击败 235B
  Result: 4B beats 235B
• 它能迁移到更难的问题上吗？
  Does it transfer to harder problems?
• 通用工具的使用表现稳定
  General tool use held up
• 简单数据获胜了
  Simple data won
• 获得的简单奖励
  Simple rewards won
• 企业级智能体蓝图
  A blueprint for enterprise agents
• 如何评估智能体
  How to evaluate agents
• 开放基准测试资助项目
  Open Benchmarks Grants

简介Intro

主持人：感谢大家参加。我是 Charles，是 Snorkel AI 的研究科学家。今天我将介绍，我们如何让一个 40 亿参数的模型胜过它的 2350 亿参数版本。这是在模拟金融环境中取得的一项具体成果，Host: Everyone, thanks for joining. I'm Charles and I'm a research scientist at Snorkel AI. And today I'm going to be talking about how we got a 4 billion parameter model to outsmart its 235 billion parameter variant. And this is a specific result in a simulated financial environment,

专业化有时胜过规模优势Specialization can beat scale

但我认为，有一些普遍性的经验值得确保大家牢记。首先，对于企业级 AI 工作负载而言，可靠性和专业化实际上可能比参数规模更重要。而要做到这一点，可以依靠高质量的数据，这些数据能够真正代表你所面向的领域和任务分布。这样一来，规模更小的模型也能达到前沿模型的水平，而成本却只需其一小部分。所以，在谈到这一点之前，我先……but I think there are general takeaways that I want to make sure taken home. And the first is for enterprise AI workloads, reliability and specialization uh can actually outweigh raw parameter scale. And this is possible with high-quality data that actually represents the domain and task distribution that you're targeting. And uh this can result in smaller models that can achieve parity with frontier models at just a fraction of the cost. So, before I get to it, I'm going to take

关于 Snorkel AIAbout Snorkel AI

借此机会简单介绍一下我们自己。这是我们的团队。我们是一家前沿人工智能数据实验室。团队创始人来自华盛顿大学、威斯康星大学和斯坦福大学的实验室，此外还有一支由非常优秀的全职员工和研究人员组成的内部团队。我们很幸运，能够参与前沿人工智能领域的工作——也就是说，构建并部署数据集和环境。这项工作处于学术界与产业界研究的交汇处，同时也扎根于企业部署实践。总而言之，我们的团队致力于构建尽可能优质的数据，以推动先进人工智能的发展。我们的工作主要分为三个方面。首先是开放研究和基准测试，我们持续为开源基准测试做出贡献，而我今天要介绍的项目就属于这一方面。this opportunity to introduce ourselves a little bit. This is our team. We're a frontier AI data lab and our team, we have founders from labs at University of Washington, Wisconsin, Stanford, and an internal team of really talented FTEs, researchers, and we're in a fortunate position to play in, by that I mean build and deploy data sets and environments uh for frontier AI. And that's at the intersection of academia and industry research, but also grounded in enterprise deployments. Um and this is all to say that we have a team that's built to focus on building the best data possible to define in advanced AI. And the way we do this is threefold. First is open research and benchmarks, so we're constantly contributing to open-source benchmarking, and the project I'll be talking about today actually falls under this pillar.

嗯，不过我也想提一下，我们还致力于前沿实时数据和环境，以及企业级部署。在开放研究与基准测试方面，我们通过资助支持研究。我会在演讲最后谈谈相关机会，所以请做好准备。除此之外，我们还会与学术界或业界的合作伙伴共同撰写研究成果，并主导独立研究、发布开放成果。Um but I also want to acknowledge frontier live data and environments, as well as enterprise uh deployments are the things we work on. And under open research and benchmarks, we support research through grants, and I'll talk about opportunities here at the end of this talk, so um get ready for that. And then co-authoring with academic uh or industry partners um and leading independent research and publishing open artifacts.

与加州大学伯克利分校合作Working with UC Berkeley

那么，我今天要谈的是与加州大学伯克利分校 Sky Computing Lab 的合作。正如我之前提到的，我们构建了一个拥有 40 亿参数的 Qwen 模型，在金融任务上的表现超过了它的 2350 亿参数版本。剧透一下：我们看到这个 40 亿参数模型的 pass@1 接近 60%，而 2350 亿参数模型达到了 51%。结果令人振奋。这项工作是合作完成的，该致谢的必须致谢：Sky Computing Lab 的 Manon Rupta 和 Shujian Tang，以及 Snorkel 的 Bhavishya、Chris Glaze 和我本人。你可以在我们的 GitHub 上找到所有资源，包括训练脚本和合成数据，欢迎查看；这些内容都是开源的，可供你使用。今天我会介绍我们从企业部署经验中了解到的金融机构对 AI 的需求，以及 FinQA 开源基准。So, what I'm talking about today is uh a partnership with UC Berkeley Sky Computing Lab, and we built, like I said, a 4 billion parameter Qwen model to outperform its 235 billion model variant on financial tasks. We saw the 4 billion parameter model, spoiler alert, uh achieve almost a 60% pass at one, while the 235 billion model uh achieved 51%. So, exciting results, and this is all uh in collaboration with I got to give credit where credit's due, Manon Rupta, Shujian Tang from the Sky Computing Lab, as well as Bhavishya and Chris Glaze and myself uh from Snorkel. And you can find all the assets, the uh training scripts, synthetic data uh on our GitHub, so check those out, and this is all open-source and available to you. So, I'll cover today what uh we see financial institutions need um from AI, and this is from our enterprise deployments uh learnings, as well as the FinQA open-source benchmark.

这是经过专家验证的金融问答数据，我们用它训练并评估了模型。最后，我们来介绍 RLLM FinQA 4B 模型的训练过程。那么，开始吧。And this is expert-validated financial question-answering data that we trained our uh and evaluated our model on. And finally, uh the training process for the RLLM FinQA 4B model. So, let's get started.

两个模型的故事A tale of two models

这是唯一一张提到查尔斯·狄更斯的幻灯片，我保证。不过我想说，这是两个模型的故事。首先，是大型通用模型与小型专用模型的对比。重点在于，金融机构问我们模型或智能体的推理能力有多强时，他们通常真正想问的是：它能否在我这套复杂的架构、遗留 API 和技术债务中运行？它能否在真实且漫长的工作流程中值得信赖，而不会让错误不断累积？我们能否可靠地审计这个智能体？例如，在金融和法律领域，这一点很重要。这就引出了一个问题：我们该如何评估和改进智能体，让它们在这些现实环境中变得可靠？嗯，我们可能会本能地想要扩大规模。我们都见过规模定律的结果，但今天我想要论证的是……This is uh the one and only slide with the Charles Dickens reference, I promise, but this is a tale of two models, I'd say. First, the large generalist versus small specialist. And the point is here, when financial institutions ask us how well does the model or the agent reason, what they typically mean is can it operate in my stack of complex schemas, legacy APIs, and tech debt? And can we trust it in realistic uh and long workflows without compounding errors? And can we reliably audit the agent? That's important in financial and legal domains, for example. Which brings us to the question is uh how can we evaluate and improve agents to make them reliable in these uh real-world environments? So, um the reflex may be to scale. We've all seen scaling law results, uh but today I'm going to argue that

专才，而非博学家Specialists, not polymaths

专业化工作流程实际上并不需要通才。我们不需要博学家。你知道，如果你有幸把陶哲轩加进了联系人，也不会为了税务审计去找他。你会去找熟悉相关表格、工具和规则的专家。这就是我们开发 FinQA 的原因。它是一套经过专家验证的金融问答数据。接下来，我会介绍我们构建这一基准和训练数据的多阶段流程。specialized workflows don't actually need generalists. We don't need polymaths. You know, you wouldn't call Terence Tao for a tax audit, um if you're lucky enough to have him in your contacts. You'd call the specialist that knows the forms, the tools, and the rules. So, that's why we built FinQA. This is expert-validated financial question-answering data. And I'll walk through the multi-stage pipeline of how we built this benchmark and training data.

基于 10-K 文件构建 FinQABuilding FinQA from 10-K filings

那么，第一步是模式和数据提取。这些内容来自 10-K 报告。嗯，这是美国证券交易委员会要求所有上市公司提交的年度报告，分析师会用它们来识别风险并辅助投资决策。嗯，我们从 Edgar 系统中获取这些报告，并使用 Qwen 3 30B 模型生成表格，也就是 SQL 表，大约有 6,900 个。我们用每张表生成一组问答。生成时还会提供由熟悉这些文档的金融专家制定的问题分类体系。这些内容会作为上下文，嗯，供模型生成问答，并附带元数据；稍后我们会看到这些元数据的用途。最后，第三步是验证。这是一个三层的……So, first is schema and data extraction. This is coming from 10-K reports. Uh these are annual reports required by the SEC for all public companies, and they're used uh by analysts for identifying risks and informing investment decisions. Uh we pull these from the Edgar system and use the Qwen 3 30 billion model to produce tables, SQL tables, approximately 6,900 of those. And we use each table to generate a single question-answer pair. And this is uh produced along with the question taxonomy that was developed with financial experts that work with these documents. And these are used as context uh for the model to generate a question-answer pair along with metadata that we'll see uh what that's used for later. And finally, the third step is verification. And this is a three-layer

三层验证Three layers of verification

验证流程。首先是程序化一致性检查。我们会使用元数据，包括血缘关系、表名和列名，确保这些内容没有被臆造。其次是由独立智能体进行自动审查。最后是专家人工审查，也就是由人工逐一检查，确保问题和答案真实可信且经过验证。最终产出的是 Snorkel 的 FinQA 数据集。这是我们的第一轮尝试。数据包含需要规划、使用工具、调用工具并进行推理的真实查询，而答案都是单一、可验证的最终答案。基于原始约 7,000 张表的数据，我们得到训练集 4,000 张、验证集 500 张，以及留出基准集 290 张。数据经过拆分，确保基准集中的公司不会与任何其他数据拆分重叠。verification process. First is programmatic consistency checks. That's where we use um the metadata, uh which includes lineage and table names and columns to make sure we're not hallucinating those. The second is automated reviews from independent agents. And finally, expert manual review. So, humans actually going through these and making sure the questions and answers are realistic and verified. And the outcome is Snorkel's FinQA data. And this is our first pass at this. Uh realistic queries requiring planning, tool use, uh tool calling, and reasoning. And um answers are single, verifiable, final answers. And from this, um we get from the uh original set of roughly 7,000 tables, 4,000 in train, 500 in val, and 290 in the held-out benchmark. And the data is split so that no company in the benchmark overlaps um in any split.

而基于这些数据，我们发现，嗯，即便是前沿模型，也反复存在纪律方面的缺口，呃……And on this data, we identified um even with frontier models, uh reoccurring discipline gaps, uh

模型失效之处Where models fail

失败模式。首先是模式幻觉。模型会自以为某些表和列名确实存在，这可能是预训练数据造成的，它过去曾见过这些内容。其次是上下文过载，也就是用执行效果不佳的查询不断填充模型自身的上下文。例如，只调用 `select star`；如果执行成功，这可能会耗尽它的上下文容量。最后是错误恢复能力差。也就是说，3235 会重复同一种失败策略，而不是分析错误信息并做出调整。这些失败模式并非只出现在我今天谈到的金融领域，也出现在我们在保险核保中看到的情况里。我想特别提到这项工作，它是几周前在 CAIS 会议上发表的。failure modes. And the first is schema hallucination. So, a model will actually assume that tables exist, column names exist, and um this can be an artifact of pre-training data, something it's seen um in its past. The next is context flooding. So, actually flooding its own context um with poorly playing queries. For example, just calling select star, um and this could overwhelm its context limits if successful. And finally, poor recovery um from errors. So, when 3235 would actually repeat the same failed strategy um instead of considering the error messages and adapting. And these failure modes are not actually just seen in the financial environment that I'm talking about today, but also what we saw in insurance underwriting. And this is a work that I'm calling out that was presented at the CAIS conference um few weeks ago.

那么，现在我来实际开始训练。So, now I'll get into actually training

使用 rLLM 进行训练Training with rLLM

R LLM FinQA 40亿参数模型。呃，我们使用了 R LLM 框架，它由 R Sky Computing Lab 的合作者开发。如果你还没用过它，嗯，这是一个非常易用的工具。我想特别提到它的几个核心功能：它可以与任何智能体框架配合使用，只需通过装饰器模式，几乎不需要修改代码；采用 CLI 优先的工作流；并且经过了实战验证。嗯，我们发现 R LLM 不仅能提升这个金融案例研究中的表现，在数学推理方面也有提升。比如，它内置了多种 RL 算法，包括 GRPO、reinforce RL 和 OOO；你可以对这些算法进行自定义，而且它还支持多种训练后端。好了，接下来介绍我们的训练环境，嗯，这是训练……the R LLM FinQA 4 billion model. Uh for this we used the R LLM uh framework and this was developed by the R Sky Computing Lab collaborators. And if you haven't used this yet, um this is a really user-friendly tool and a few core features um I want to call out is it really works with any agent framework, requires near zero code changes just using uh a decorator pattern, CLI first workflow, battle-tested results. So, um we've seen R LLM actually improve not just on this finance um case study, but also in math reasoning for example. Um multiple RL algorithms are built in, GRPO, reinforce RL, OOO for example, and you can customize those and then supports multiple uh training backends. All right, now for our training environment, um this is the uh training

训练环境The training environment

我们为模型使用的环境。智能体在 ReAct 循环中运行，并可以使用工具来与我们生成的表格进行交互。该环境包含我们创建的约 7,000 张表格。最后，奖励是二元正确性，由语言模型作为评判者来判定。本例中使用的是 GPT 5 nano，并采用基于参考答案的评估。environment we used for our model. The agent runs in a react loop and has access to tools uh to interact with the tables we generated. The environment includes the roughly 7,000 tables we created and finally reward is binary correctness uh determined by a language model as judge. In this case, GPT 5 nano using reference-based evaluation.

培训费用低于 500 美元Training for under $500

训练细节列在这里。这大概会让大家感兴趣。我们的基础模型是 Qwen 3 40 亿参数模型，使用 GRPO 和二元奖励进行优化，并采用了 RL 框架。我们运行了大约 1,000 个并发环境来生成轨迹，获得这些结果的总成本不到 500 美元。细分来看，计算成本约为 420 美元，评审 API 的费用约为 40 美元。训练使用了 8 张 H100。Training details are listed here. This is uh I guess would be interesting to folks. We have the Qwen 3 4 billion model is our base model, GRPO and binary reward uh for optimization and the RL and framework. Um we ran roughly 1,000 concurrent environments to generate trajectories and this all was a cost under $500 to get the results that we saw. Um broken down between compute roughly 420 and $40 for a judge uh the judge API. And this was on eight H100s.

结果：4B 击败 235BResult: 4B beats 235B

第一个结果，嗯，就是核心问题：一个配备了专用工具并经过专门训练的40亿参数模型，能否与它更大的变体竞争？答案是肯定的。我们看到，基础模型的准确率提高了一倍多，并且表现优于它更大的变体——235参数模型。尽管它的规模只有对方的一小部分，但仍取得了这样的结果。我们使用专家筛选的290个留出样本，对这里展示的所有模型进行了评估。The first result um is the central question, can a 4 billion model uh armed with the specialized tools and training actually compete with its larger variant? And the answer is yes. So, what we saw was actually the base model's accuracy um more than doubled and it outperforms its larger variant, the 235 parameter model. And this is uh despite being a fraction of the size. And we evaluate all models shown here um on the expert curated held out 290 samples.

它能迁移到更难的问题上吗？Does it transfer to harder problems?

因此，一个很自然的担忧是：这能否真正迁移到更难的问题上？目前的任务只是基于单张表格的单轮问答。为了研究这一点，我们考察了 FinQA 推理数据。这是在我之前介绍的 FinQA 数据基础上的扩展。不同之处在于，现在需要依赖多张表格（两到五张），并进行连续的决策和规划。我们在这个数据集上进行了评估，具体划分也一并列出：训练集约 1,000 个样例，验证集 120 个，基准集 80 个。我们发现，工具使用规范确实能从简单数据上的训练直接泛化过来。实际上，无须针对多表格样例进行显式训练，也能在这个数据集的多表格变体上取得提升。So, a natural concern is whether this actually transfers to harder problems. So, this is just single question answer pairs on a single table. Um and to investigate this, we uh looked at our FinQA reasoning data. So, this is a development on the FinQA um data I presented earlier. And the difference here is that now we're having a dependencies on multiple tables, two to five tables, and requires sequential um decision-making and planning. And we evaluated on this data set. You can see the breakdown um as well. We have roughly 1,000 in train, 120 in validation, 80 in benchmarks. And what we see is the tool use discipline actually generalized directly from training on the simple data. And no explicit training on multi-table examples was needed actually to see lift in this multi-table variant of the data set.

我们将通过消融研究探究原因。但我们接下来的担忧是，这一结果是否能够泛化到……And we'll investigate why with the ablation studies. But the next concern we had was whether this result would generalize to

通用工具使用能力依然稳健。General tool use held up

更通用的使用场景。因此，我们使用了 BFCL 基准来衡量通用工具调用能力。总体准确率实际上略有提升，而且我们在多轮交互和记忆方面也取得了小幅进步。因此，在对该模型进行强化学习微调时，专业化并没有削弱它更广泛的工具使用能力。接下来，为了找出是什么促成了这一结果……more general use cases. So for this we use the BFCL benchmark and this measures general tool calling capability. And overall accuracy was actually slightly improved and then um we had also minor gains on multi-turn and memory. So specialization didn't actually erode the model's broader tool use competence when we did RL fine-tuning for this model. So to isolate what drove this

简单的数据获胜了Simple data won

在性能方面，我们对数据混合方案进行了消融研究，也就是尝试不同的方式来组合我们已有的训练数据。第一种是只用 FinQA 数据进行训练，第二种是同时使用单表和多表数据，最后一种则是采用课程式训练，先从单表数据开始，再过渡到多表数据。令人惊讶的是，只在较简单的数据集上训练，带来的提升反而最大。这在一定程度上解释了我们之前看到的泛化结果。瓶颈从来都不是推理深度，而只是工具使用能力；也就是说，模型实际出问题的地方在于工具使用，而我们提升的是这方面的可靠性。一旦掌握了基本功，它就能将这些技能组合起来。对，抱歉，我回去一下。performance, we ran ablation study on data mixtures, so different ways we can combine the training data we had. The first is just training on FinQA data and the second single and multi-table and finally a curriculum where we start with single and then move to multi. And surprisingly, just training on the simpler set resulted in the most lift. And this partially explains why the generalization result we saw earlier. The bottleneck was never reasoning depth, it was just tool use, meaning the model where it was actually failing and improving the reliability of that. And once you master the fundamentals, it could compose those skills. Yeah, so oops, let me go back.

轻松赢得奖励Simple rewards won

接下来一个自然的问题是，更具信息量的奖励信号——比如奖励中间步骤、表访问、查询完整性之类——是否真的能加快学习？嗯，这和课程学习不一样。我们提出的最复杂方案是评分标准，它采用包含多个加权组成部分的细粒度评分。这实际上是与专家共同设计出来的。嗯，尽管我们在奖励工程上投入了这些精力，但我们再次发现，越简单越好。单一的二元奖励信号，嗯，在这个任务上给我们带来了最大的提升。The next natural question is whether more informative reward signals, Um, for example, rewarding intermediate steps, uh, table access, query completeness, things like that could actually accelerate learning. Uh, so this is different from curriculum, for example. And the most sophisticated variant we came up with was, uh, a rubric. And this uses fine-grained scoring with, uh, multiple weighted components. And this is actually coming came up with with experts. Um, and despite this investment in reward engineering, we actually again saw, um, simpler was better. A single binary reward signal, um, was what got us the most lift on this

企业级智能体蓝图A blueprint for enterprise agents

数据。这让我们得以制定一套蓝图，嗯，我们一直在为企业级智能体开发这套蓝图。嗯，总的来说，我认为这项研究表明，如果有高质量数据来代表目标结果和任务，就可以通过 RL 成功训练小模型。这样一来，针对专业任务和领域的部署成本效益也会更好。嗯，我们在其他领域也看到了这一点，并反复验证了这一结果，包括医疗保健、法律和保险，嗯，这些都是我之前提到过的。所以，去看看吧。嗯，至于构建用于评估智能体的维度，我们的观点是……data. Which led us to a blueprint, um, that we've been developing for enterprise agents. Um, and generally, I'd say this study showed us that small models uh, can successfully be trained uh, with RL um, when you have quality data to uh, represent the target um, outcomes and tasks. And this can result in better deployment economics uh, for specialized tasks and domains. And um, we've seen this and repeated this result in other areas including health care, law, and insurance, um, which I pointed to earlier. So, check that out. Um, and our point of view as far as building um, axes for evaluating agents is this is

如何评估智能体How to evaluate agents

环境复杂度是最重要的维度之一。智能体所处的环境有多复杂、多真实、多动态？接下来是自主运行时长。人们是否按照实际希望使用智能体的方式，对智能体进行测试和评估，考察它们在相应的任务时长和运行周期内的表现？它们能否独立做出安全且恰当的决策？随着时间推移，它们能否真正提升作为副驾驶时的判断力？最后是输出复杂度。我们希望涵盖日常工作中各种各样的输出。我认为，目前这一方面还没有得到充分探索。虽然已有新的基准测试朝这个方向发展，但我仍然看到大家主要关注文本产物；如何在这些场景中创建更严谨的评估，是一个很大的机遇。这方面已经取得了一些进展。environment complexity is um, one of the most important axes. How complex, how realistic and dynamic are the environments uh, the agents operating in. The next is autonomy horizon. Um, are agents being tested and evaluated at the lengths and the horizon lengths in the way people actually want to use agents? Um, are they making safe and good decisions by themselves? And can they actually improve um, their judgment as a co-pilot over time? And finally, output complexity. We want to capture the wide range of outputs in our day-to-day work. Um, and I would say that this is currently under explored. New benchmarks are coming up in this direction, but uh still I see a big primary focus on um text artifacts and creating more rigorous evaluations and these settings is a big opportunity. Progress is being made here.

开放基准测试资助项目Open Benchmarks Grants

那么借此机会，我想宣传一下我们的开放基准资助项目，这个项目为本项目提供了资金支持。这项承诺投入数百万美元，用于开发这样的开放基准。我们在 Agent、Last Exam、Judgment Bench 等项目上取得了很大成功，其他项目也即将推出。这确实加快了我们衡量前沿水平、塑造前沿发展的能力。要申请的话，请访问我们的网站。你需要经历提交、筛选、评审、开发，最后是发布、上线和推广。我还想提一下，我们正在招聘专业研究人员、工程师等等，欢迎关注。你可以访问我们的网站。最后，非常感谢大家参加。你可以扫描这个二维码查看博客，以及 GitHub、Hugging Face 上所有已发布成果的链接。So, on that note, um I'm going to advertise our open benchmarks grants, which funded this project. And this is a multi-million dollar commitment to producing open benchmarks like this. We've had a lot of success agent, last exam, judgment bench, um others are coming out and this is um really accelerating our ability to measure the frontier and shape the frontier. So, to apply uh check out our website. Um you'll go through a submission selection review process, develop, and then finally publication, launch, um and promotion. And I also want to call out that we're hiring in professional research, um you name it, engineers, and um yeah, check that out. You can check out our website and finally, thank you so much for attending. Um you can use this QR code to see the blog as well as pointers to all the artifacts that released on GitHub, Hugging Face.

好的，谢谢大家。Yeah, thank you all.