即时本体：廉价智能如何改变数据转换｜Hrishi Olickel，Southbridge AI

简要描述Brief Description

即时本体：低成本智能对数据意味着什么——Hrishi Olickel，Southbridge AIJust-in-Time Ontologies: What Cheap Intelligence Means for Data — Hrishi Olickel， Southbridge AI

目录Table of Contents

• 低成本转型与俄亥俄州数据管道
  Cheap transformation and an Ohio data pipeline
• 为什么通用架构行不通
  Why a universal schema breaks down
• 为该决策构建本体
  Building an ontology for the decision
• 三种数据工作负载
  Three kinds of data workload
• 设计师、工人和优化模型
  Designers, workers and optimizing models
• 实体解析和定义检查
  Entity resolution and definition checks
• 模型先验与隐性数据错误
  Model priors and silent data errors
• 跨模型检查与共同错误
  Cross model checks and shared mistakes
• 自我优化的四项要求
  Four requirements for self optimization
• 范围受限的试点与可复用的已校验片段
  Bounded pilots and reusable checked segments

低成本转换与俄亥俄州数据管道Cheap transformation and an Ohio data pipeline

所以，我真的很喜欢 DuckDB 的 Hannis 写的这篇文章。他提出了一个问题：我们是否花了十年时间追逐分布式架构，而这些架构的前提是计算和存储既慢又贵？不过，每个时代的能力都会定义架构和工具，而工具始终滞后于时代。我们先让计算比存储更便宜，然后让人人都能使用计算资源，接着又让计算变得更快——每一次，架构都会经历巨变。如今，我们所有的数据基础设施都建立在“数据转换成本高昂”这一理念之上。可如果这种情况不再成立呢？我认为，我们已经解决了数据转换的最后一个瓶颈——通用智能，也就是判断力；而且，我们已经让它在整个 burrito frontier 上都能随取随用。这让我们正站在范式转变的前夜。So, I really love this article from Hannis at DuckDB where he asks if we lost a decade chasing distributed architectures that presumed compute and storage were slow and expensive. Capability in every era though defines architecture and tooling and the tooling lags behind consistently. We first made compute cheaper than storage and then we gave it to everybody and then we made it faster and every single time architectures undergo this massive change. All of our data infrastructure today is built on the idea that transformation is expensive. What if that wasn't true anymore? I believe we've solved the final bottleneck in transformation, general intelligence or judgment, and we've made it accessible on tap all along the burrito frontier, which puts us right on the verge of a paradigm shift.

所以，在这场演讲中，我们将探讨为什么单一本体总是会失败、模型在流水线中扮演的三种角色、如何交叉运用模型先验以获得更好的结果，以及“卷饼冲浪”。如今的模型如何同时针对成本、速度和准确度优化工作负载。下面是一个简单的例子。Jeff 上线时，我们已经有一条实体消解流水线在运行，用来处理俄亥俄州的竞选财务记录，并判断这些记录是否属于同一个人、公司或委员会。预计这项工作要花费约 10,000 美元。我们的系统接入这个新模型后，确定了它的适用环节，测试了十多种方案，并将其集成进去。首次通话后的第 10 天，我们就拿到了俄亥俄州所有金额低于 800 美元的捐赠者案例的最终真实标签。这就是……的力量。So, in this talk, we're going to cover why single ontologies always fail, the three roles models play in pipelines, intersecting model prior for better results, and burrito surfing. how today's models can tune a workload for cost, speed, and accuracy all at the same time. Here's a quick example. When Jeff launched, we already had an entity resolution pipeline running, working out Ohio's campaign finance records and whether they belong to the same person, company, or committee. It was projected to cost about $10,000. Our systems took that new model, worked out where it fit, tested over a dozen approaches, and integrated it. 10 days after that first call, we had final ground truth for every donor case in Ohio under $800. That is the power of

为什么通用架构会行不通Why a universal schema breaks down

卷饼冲浪，以及即时转换。好了，我们开始吧。本体论是企业版的思维导图。这是个把我们所有人都吸进去的沥青坑，我也不例外。两年前创立 Southbridge 时，我谈到过一个单一的通用模式。大众的看法是——这并非错，只是只看到了全貌的一小部分——迁移和本体论之所以失败，是因为缺乏认同。也就是说，项目进行到一半就被搁置，组织和人员始终没有完成切换。这些说法百分之百都是真的。但缺乏认同并不是主要原因。你看，即便你能眨眼之间把所有信息都整合并迁移到一个统一的超级模式中，你仍然必须面对一个事实：根本不存在什么超级模式。实际上有四种模式。第一种属于源头。burrito surfing and just in time transformations. So, let's get into it. Ontologies are the enterprise version of mind maps. It's this tarpet that sucks us all in, including me. I talked about a single universal schema when I founded Southbridge two years ago. The popular conception and it's not wrong, it's just small part of the picture is that migrations and ontologies fail because of a lack of buyin. That they're abandoned halfway through that organizations and people never switch over. All of that is true 100%. But buyin isn't the primary reason. See, if you could integrate and migrate all of your information into a single super schema in the blink of an eye, you still have to content with the truth that there is no super schema. There actually are four schemas. The first one belongs to the source.

无论信息来自哪里，我们以温度为例。假设我让你把温度记下来。源模式就是你能够获取到的任何信息：你的手机可以告诉你所在的区域和温度；你桌上的传感器精度较低，但位置更近；或者你可以用手表。甚至你是否有耐心完成我的请求，也属于这个源模式。第二种是存储模式。你能记下的信息总是有限的。你没法把正在使用的设备的固件记下来。你甚至没法把时间精确到秒，因为你不可能在一秒之内把所有这些信息都写完。这就是存储给你施加的模式限制。第三种属于决策。我为什么需要你把温度记下来？也许我想弄清楚什么能让你感到舒适；如果是这样，我就需要你在一天中的不同时间测量温度，而且还要确保测量的人是你。Wherever the information comes from, let's take temperature as an example. Say I ask you to write down the temperature. The source schema is whatever you have access to. your phone gives you your regions, temperature, sensor on your desk which is less accurate but it's more local or your watch but even your patience in following through on my request as part of that source schema. The second one is the storage schema. There's only so much you can write down. You can't write down the firmware of the device you're using. You can't even write the time down to the second because you couldn't finish writing all of that inside of a second. This is the schema imposed on you by storage. The third one belongs to the decision. Why do I need you to write the temperature down? Maybe I'm figuring out what makes you comfortable, in which case I need temperatures from you spread throughout the day, and I need to make sure that it's you that's doing the measuring.

也许我正在绘制天气迁移地图，在这种情况下，我不在乎是谁进行测量，但我非常在意你的准确位置、时间、所用设备，以及这些是否发生了变化。或者，我可能在决定是否安装敏感设备以及安装在哪里，而你自己就成了我需要消除影响的变量。不过还有第四种情况，我们谈到先验信息时会讲到。它就是这个近期的问题。真正的问题在于，除非正好是在做出决策的那一刻，否则你永远无法同时知道全部四种模式。在几乎所有企业和家庭中，所记录的大多数数据都很难说清将来会被用于什么。以客户为例，这是每种本体论中的第一个类别。对财务部门来说，客户就是付款的人；对支持部门来说，客户就是打电话的人。Maybe I'm building weather migration maps, in which case I don't care who does the measuring, but I care a lot about your exact location, the time, the device you used, and whether that's changed. or maybe I'm deciding whether to install sensitive equipment and wear and you yourself become this variable that I need to cancel out. There's a fourth one though which we'll get to when we talk about prior. It's this recent issue. The true problem is that you will never know all four schemas at the same time except at the moment that the decision is being made. In almost every enterprise and every home most of the data being recorded has little idea what it's going to be used for. Take a customer, the first box in every ontology. To billing, a customer is whoever pays. To support, it's whoever calls.

对合规部门来说，关键是我们要为谁承担责任。只有决策确定后，你才会知道。To compliance, it's whoever we're on the hook for. And the only time you know is when the decision is known.

为决策构建本体Building an ontology for the decision

我们应该给这笔订单退款吗？我们能给这个账户增加更多信用额度吗？突然间，你需要知道什么，以及需要知道到什么程度，就变得极其清楚了。这就是我错在哪里。不存在通用的模式。脱离了你想要回答的问题，就不存在所谓的“客户”概念。把这些模式整合起来，正是转化信息、让信息真正发挥作用的过程。把信息记录时的上下文与决策时的上下文结合起来，这需要判断力。在已知决策范围之后再进行这种整合，而不是提前完成，这就是我所说的即时本体。稍后我们会通过一个我上一份工作的故事，再谈谈判断力；在那里，我的工作大部分时候就像是确保某些数字以前根本不存在……在那之前，先简单介绍一下我和我们。Should we refund this order? Can we extend this account more credit? Suddenly, what you need to know how and to what degree becomes extremely clear. This is why I was wrong. There is no universal schema. There is no concept of a customer that's divorced from the question that you want to answer. And pulling these schemas together is the process of transformation of making information actually useful. Taking the context that it was recorded in and merging it with the context of the decision and that takes discernment. Doing that merge when the decision space is known instead of ahead of time is what I mean by a just in time ontology. So we'll come back to discernment with a story from my last job where like most of my job was to make sure that certain numbers just did not exist ever before that a little bit about me and us.

最能描述我们的词是痴迷者。把智能嵌入流水线的痴迷者。能够扩展、能够一次又一次运行，真正替我们接手工作的流水线。以及流水线We are best described as obsessives. Obsessives who embed intelligence into pipelines. Pipelines that can scale, that can run over and over again to take work truly off our hands. and pipelines

三种数据工作负载Three kinds of data workload

它们可以在成本、性能和速度方面各自独立地成熟起来。我们在数据领域所做的大部分工作，可以归为三种模式。第一种是“一次写入、运行一次”。如果你从事任何类型的开发，可以把它们想成规模庞大的即时调用函数。它们是为单一用途而构建的，比如合并数据集、挖掘信息，或回答某个特定问题，完成后就会被拆除。第二种是“多次写入、运行一次”：为一个共同的大目标构建大量大大小小的工作负载。你可能熟悉的最接近的例子是大型迁移：将多年的记录从那些没人能完全记得的系统中迁出。为一次迁移构建许多工作负载，并逐一验证；在源系统中检查它们，并与其他工作负载相互核对，因为这一切只会发生一次。第三种则是我们的看家本领，也是最常见的模式：多次写入、运行……that can mature in cost, performance and speed all on their own. And most of the work we do in data breaks into three regimes. The first is write ones run once. If you do any kind of development, think of these as monster scale immediately invoked functions. They're built for something single use like joining a data set, mining information, or answering one specific question and then they get torn down. The second is write many run once where lots of smaller and larger workloads are built for one large common goal. The closest thing you might know is like large migrations, years of records, moved out of systems nobody fully remembers. Lots of workloads built for one move and verified individually at source checked against the others because all of it only happens once. The third though and this is bread and butter is the most common is write many run.

这些是可重复运行的流水线，会随着我们不断补充信息而逐步构建和维护。在 Sautri，我们已将大量工作从主线程转移到这些可重复运行的系统中，涵盖基准测试设计、跟踪、编码，以及为我们想看的电影制作字幕。现代世界里，我们几乎所做的一切都是对数据的转换，或者受制于数据转换的难度。在所有这些情况下，我们的工作都很简单：确保人们能够信赖结果。短短两年里，我们从探索如何让早期达到 3.5 Turbo 水平的模型在医疗保健、老龄化和采矿研究等领域可靠运行，发展到构建 Hankwave——一个用于数据工作的开源工具框架——再利用它让模型能够协同处理 TB 级规模的数据。如今，我们已经拥有能够自我优化的系统。These are repeatable pipelines that are built and maintained over time as we add more information in. And at Sautri, we've moved so much work off the main thread and into these repeatable systems from benchmark design, tracking, coding to making subtitles for movies that we want to watch. Almost everything we do in the modern world is a transformation of data or it's bottlenecked by how hard it is to transform data. And in all of these cases, our job is simple. Make sure people can rely on the results. In just two years, we've gone from figuring out how to make early 3.5 turbo level models reliable on things like healthcare, aging, mining research to building Hankwave or open-source harness for data work and then using that to get models to work together at terabyte scale. Today we have systems that can self-optimize.

你把问题交给他们，给他们一种可以安全失败的方式和一个开展工作的地方，他们就会找到解决方案的形态。他们会挑选合适的模型，开展试点，然后扩大规模，一路“卷饼冲浪”，找到自己能找到的最佳方案。 说个故事吧。在我上一份工作中，我们最常听到的问题是：“我们能用数据来做这个决策吗？”有时这确实是个合适的问题，比如当时平台上有成千上万艘船。但问题是，我们只有 50 名用户时，也会问这个问题。如果你懂统计学，就能很好地解释为什么对 50 名用户做 A/B 测试是个糟糕的主意。不是因为得不出结果，而是因为你一定会得出结果。所以唯一真正可行的选择，就是干脆不产生任何数字，因为一旦有了数字，它就会为某个结果辩护。不过，今天的情况更糟。You give them the problem, a way to fail safely, a place to work, and they find the shape of the solution. They pick the right models. They run the pilots. They scale up and they burrito surf their way to the best option they can find. So story time at my last job, the question we heard most often was, can we make this decision data? And sometimes that was the right question when we had thousands of vessels on the platform. The problem is that we also asked it when we only had 50 users. And if you know statistics, you can explain really well why AB testing 50 users is a bad idea. Not because you're not going to get a result, but because you will. So the only real option was to do not produce a number at all because the moment a number exists, it argues for a result. Today though, it is worse.

交互式智能体总会给你一个答案，而且会很快给出答案。你随便要求它做个转换之类的，它就会斩钉截铁地答应：“好的，先生。”这意味着错误会迅速累积。聊天中的一个瞬间，就可能变成一场潜伏着、随时可能暴露的生产环境噩梦。这正是……的主要原因。Interactive agents will always give you an answer. They'll give it to you quickly. You ask for a transformation, whatever, and you get a contra yes sir. What this means is mistakes add up quickly. Moment in chat can be a prod nightmare forever just waiting to be discovered. The primary reason that this

设计师、工人和优化模型Designers, workers and optimizing models

实际情况是，在如今与数据工作相关的智能领域，智能承担着三种不同的角色，而编码代理会把它们混在同一个上下文窗口里。我们把它们拆开来说，因为等讲到决策模型时，这会很有帮助——决策模型恰好属于这三种角色中的一种。第一种角色是构建和设计流水线：拆解问题、编写早期规范、授权支出。第二种角色通常会和第一种混淆，也会交给同一批模型，就是在流水线内部实际执行工作：读取 PDF、写入数据库、检查结果。在讲第三种之前，我想先说明，这两个角色的需求截然不同。第一种需要更慢的模型，使用更重、更厚实的 token；需要能进行更多思考、从不同角度审视问题、消化文档、理解这些流水线是如何构建的，并在任务开始前就预测其表现的模型，对吧？happens is that intelligence as it relates to data work today occupies three separate roles that coding agents kind of mash into one context window. Let's pull those apart because it'll help a lot when we get to decision models because they fit into exactly one of these three. The first one is building and designing the pipeline which is breaking down the problem writing the early specs authorizing spent. The second one and this is usually confused with the first and given to the same models is working inside the pipeline actually doing the work reading the PDF writing to databases checking results. Before we get to the third one I want to make it clear that these two roles pull in very different directions. The first one wants slower models with heavier, thicker tokens, models that can think more, walk around a problem and see the angles, digest documentation, understand how these pipelines get built, and predict how something will do before it even starts, right?

Opus 或更高。虽然在流水线中，你想要的模型恰恰相反：你希望它们更便宜、更快、更专精、善于遵循指令，是可重复使用且缺陷不明显的工作者。自然界也有同样的现象。比如，切叶蚁群由四种不同体型的工蚁组成：小型、中型、大型和特大型。当你观察、审查或听说任何一个你设计的系统时，都要留意其中有哪些槽位，以及它们在这个层级结构中的位置——这会产生很大影响。正是因为我们有这样的区分，所以把 Jeff 引入这个新近迅速兴起的决策模型类别非常容易：它们是为执行任务而构建的，擅长快速获取信息，并被部署在大型工作负载的底层。不过，第三类模型，我们称之为爬坡者或冲浪者——它们能够先试运行一遍，然后自我优化。Opus or up. The models you want inside pipelines though are the opposite. You want them cheaper, faster, specialized, good at following instructions, repeatable workers with noable defects. You see the same thing in nature. A leaf cutter ant colony for example runs on four different sizes of worker right small medium high X high as you watch any system that you design review or hear about listen for what slots and where in this layer cake it makes a big difference because we had this distinction bringing in Jeff in this newly exploding class of decision models was very easy because they're workers built they're built for quick contact with information and they're deployed at the bottom end of large workloads So third category though, this is something we call the hill climbers or surfers models that can take a first run and self optimize.

只要掌握恰当的知识、获得明确的指导，并有一个极其稳固的立足点，他们就能尝试新事物、调整预算以控制成本风险、快速调试、做好笔记，一路探索，不仅找到正确的解决方案，还能找到最优方案。我们Given the right knowledge, instructions and an incredibly stable surface to stand on, they can try new things, vary budgets to limit cost exposure, debug quickly, take notes, surf their way not just to the right solution, but the optimal one. We

实体解析和定义检查Entity resolution and definition checks

我们把这叫作“卷饼冲浪”，会在最后讲到。但首先，来聊聊我们使用 Jev 的体验。Jev 是在一个星期二上线的，当时我们正全力投入一个实体解析流水线。你当时在处理俄亥俄州竞选财务中的捐赠者。此时，很多人已经看过这张图表。我们之前谈过结果，也根据这个数据集调整了一些正在进行的工作。所以，你可以把数据集、提示词以及所有相关内容都下载下来，自己试试。我们之前没谈的是那一支箭头背后的全部过程，以及当我们把同一个思路用在整个州的数据上时发生了什么。下面是我们尝试过的所有架构。我们先在一个小型开发集上测试了三种设计：单独使用一个廉价的推理模型；让 Jeff 在遇到不确定的问题时借助推理模型；以及由推理模型提出答案、再由 Jeff 在大约 200 个留出的家庭样本上进行核验。call that burrito surfing, and we'll cover it at the end. But first, let's talk about our experience with Jev. Jev launched on a Tuesday when we were deep into an entity resolution pipeline. You were working on donors in Ohio's campaign finance. And at this point, a lot of people have seen this chart. We talked about the results before and adapted some of the ongoing work to the data set. So, you can download it for yourself prompts and everything. What we didn't talk about is everything behind that one arrow. And what happened when we ran the same idea on the whole state? Here are all of the architectures we tried. We started with three designs on a small development set. A cheap reasoning model on its own, Jeff with a reasoning model for anything it wasn't sure about, and a reasoning model proposing answers with Jeff checking them on about 200 held out families.

单靠这个廉价推理模型，表现就能与 Fable 不相上下，而成本只有它的约 2%。Jeff 一开始答对了核心问题，但漏掉了九项家庭关联，因为它对每一项都斩钉截铁地回答了“否”。于是我们没有改动模型或阈值，只是重新措辞了问题。现在，关于家庭的问题明确说明了什么才算证据充分。Jeff 对全部 13 个发展问题的回答也从“否”变成了“是”。在将这套方法推广到全州的过程中，我们进行了十多轮这样的资格判定。每一轮都厘清一个定义：什么算委员会？申报人指的是什么？两个人住在同一地址时，又该如何界定？如今，俄亥俄州的 23,317 个案例都有了标签，说明结论依据是什么，以及核查到了什么程度。这就是我们针对这个具体问题摸索出的层级体系。The cheap reasoning model on its own did as well as Fable for about 2% of the cost. Jeff first got the core answers right, but it missed nine household legs because it had confidently said no to every one of them. So we rewrote the questions without touching the models or the thresholds. The household question now said what counts as enough evidence. And Jeff went from no on all 13 development questions to yes. And as we scaled it statewide, we ran over a dozen rounds of such qualification. Each round fixed a definition. What a committee is, what a filer means, when two people live at one address, what does that mean? Every one of Ohio's 23,317 cases now has a label that says what the answer rests on and how far it was checked. This is the hierarchy we discovered for this specific problem.

这就是我们在将决策模型纳入流水线的过程中学到的全部内容。我会把这页留在屏幕上，方便你截图。接下来看看钱都花在哪儿了。整个州级项目总共花了 7.95 美元，而预算是提前设定的 800 美元。核查的费用大约是给 545 份二次意见、处理分歧以及审计所花费用的四倍；相比之下，首次作答的费用是 128 美元。在 Jev 出现之前，我们预计这项工作要花大约 10,000 美元。现在流水线已经搭建并验证完毕，我们预计下一个州的审计预算将接近于零。最令人惊讶的是，我们的智能体 4A 先是在 Astra 上运行，后来又在 Opus 上运行，它在搭建系统、运行系统、调试系统并尝试新方法的过程中，逐步自主发现了其中大部分做法。它运行了 10 天，尝试了十多种架构。And here's everything we learned about putting decision models into pipelines. I'll leave this up so you can screenshot it. But here's where the money went. The whole statewide job came to $7.95 on an $800 budget that was set ahead of time. Checking cost about four times as much as doing 545 on second opinions, settling disagreements, and the audit against 128 for first answers. And before Jev, we expected this work to cost about $10,000. And now that the pipeline is built and verified, we expect the audit budget for the next state to be close to zero. The most amazing thing is that our agent 4A running on Astra and then later on Opus found most of these approaches incrementally on its own by building the system, running it, debugging it, and trying new things. It ran for 10 days, tried over a dozen architectures.

它的 Hanks 运行了 4,000 多次，同时还向 Jeff 发出了 90,000 次调用；它几乎尝试了 Paro 前沿领域中的每一种模型，而这些运行中的模型总共处理了大约 60 亿个 token。等这一切完成后，我们把学到的东西写回了我们的智能体模式库。这项工作借鉴了其中已有的模式，我们也在加入经过测试的新模式，比如 checking letter。对我来说，这体现了把工作从人类和智能体的主线程中移开的价值，因为每一次迭代都是一个受版本控制的 hank。它们很容易回滚、理解、调试和评议。那么，模型现在能原生地做到这一点吗？还不完全行。它们还缺少四样东西，我们接下来会逐一介绍。不过，有件事不会消失，那就是随着模型……Its Hanks ran for more than 4,000 times alongside 90,000 calls to Jeff, tried practically every model on the Paro frontier, and the models inside those runs went through roughly 6 billion tokens. And once all of that was done, we wrote what we learned back into our library of agentic patterns. This work leaned on these existing ones and we're adding new tested ones like the checking letter. This is to me the value of taking work off the main thread for humans and for agents because every single one of these iterations was a version controlled hank. They were easy to revert, understand, debug, and critique. So, can models do this natively now? Not quite. They're missing four things which we'll we'll go over. And there's something that doesn't go away though as models

模型先验与隐性数据错误Model priors and silent data errors

变得更大、能力更强。先验和人类一样，模型会把自己的先验——也就是它们认为世界应该是什么样的信念——带入每一项任务。这就是我之前提到的第四种模式：模型自身带来的模式。奇怪的是，模型越大，在这方面反而越糟。有时它们会更执着于自己的信念，甚至更加激烈地为其辩护。下面是我们遇到的几个例子。有条处理管道查找的是带下划线的 in_kind，而文件里写的却是带大写 I 和连字符的 In-Kind。因此，模型把 15,658 笔实物捐赠算成了普通货币捐赠。它也没有再次核对，贷款还款记录就这样消失了。另一个模型尝试了四种日期格式，而文件用的是第五种。结果，99.4% 的交易日期都成了空值，但每行的计数和金额总和却都匹配。get bigger and more capable. The prior just like humans, models bring their own priors, their beliefs about what should be in the world to every task. This is the fourth schema that I mentioned before, the one the model brings with it. Bigger models in a strange way are worse at this. Sometimes they hold on to their beliefs and argue for them even more vehemently. Here's a few that we ran into. One pipeline looked for inind spelled with an underscore. The file spelled it with a capital I and a hyphen. So the model counted 15,658 in kind contributions as ordinary money didn't double check the loan payment repayments also disappeared the same way. Another one tried four date formats when the file used the fifth 99.4% the transaction dates then came out empty and every row count and dollar so matched.

于是，智能体继续往下处理。几次运行都发现贷款金额是负数，便认定负号是多余的，只存储了绝对值。结果，原本净额为784,000的数值最终变成了530万的债务，而且还得到了多个模型的验证。随着模型接管数据工作，这很快就成了我在各家公司看到的最常见的一类错误。有时，只要不凭感觉行事，检查结果并运用人类直觉，就能解决这个问题。实际上，你这么做时所做的事情很有意思：在第一个例子中，你是在把自己的先验知识与模型结合起来。如果人类懂竞选财务，就会发现实物捐款缺失；或者至少会想到去核查。这就是我们发现问题的方式。但就人类而言，在这个特定的……So the agent moved on. Several runs noticed that loan amounts were negative. Decided that the sign was redundant and stored the absolute value and net of 784,000 became 5.3 million of debt in the end validated by multiple models. And this is quickly becoming the largest class of error I see across companies at data work as models take over. Sometimes you can solve this just by not vibing by checking results and applying human intuition. And what you're actually doing there when you do this is interesting. You're intersecting your priors with the models in that first example. Any human would catch missing in kind contributions if they knew campaign finance or think to check. It's how we found it. But humans speaking as a human on this particular

跨模型核查与共同错误Cross model checks and shared mistakes

任务很昂贵，速度也慢，还会让时间线暂停。它们甚至比模型更不确定，而且很讨厌被打断。不过事实证明，在我们留出的 200 个家族上，跨模型的先验交集也能帮我们取得很大进展。Jev 和 Luna 有 194 个答案一致，而且这些共同答案没有一个是错的。所有错误都出现在它们意见不一致的那六个家族中。来自不同家族、不同谱系的模型，往往比同一群体中的模型更容易意见不合，先验也不同。举个例外，我就不说具体是谁了：有些模型是在同一家公司之外蒸馏出来的。通过这类工作，你可以大幅缩小问题范围，也相应减少错误，把它们归结为一类规模小得多、但始终存在的问题——这是 LLM 作为一个物种共同面对的问题。因此，在一个基准测试中，四个不同的模型家族得出了完全相同的总结果，因为它们都采用了答案键中没有采用的同一种符号约定。task are expensive. They are slow. They pause timelines. They're even more nondeterministic than models and they hate interruptions. And but it turns out you can also get really far with cross model prior intersection on our 200 held families. Jev and Luna agreed on 194 and none of those shared answers was wrong. Every error sat in the six families where they disagree. Models from different families, different pedigrees tend to disagree more and have different priors than models in the same group. For example, the exception, I'm not going to say who are models that are distilled outside the same company. So with that kind of work, you can shrink the surface quite a bit and the errors along with it down to a much smaller but persistent class of problems that applied to LLMs as a species. So in one benchmark, four different model families produced the same total down to this end because they all shared a sign convention that the answer key didn't.

而一旦你做到这一步，唯一的办法就是And once you get that far, the only way

自我优化的四项要求Four requirements for self optimization

接下来就是把它们列出来、留意它们，并针对性地采用解决方案。那么回到正题，要实现自我优化，我们还缺哪四样东西？第一，稳定的部署界面。我们用 Hank 来实现这一点：它是一个 harness，能够适配快速变化的智能体运行时环境（如 Cloud Code 和 PI），并提供一致且经过充分测试的日志和 API 接口，同时内置检查点、回滚、预算管控等功能。它是一个可以作为构建基础的东西。不过，也有其他工具符合这一描述。公平地说，对于无头数据处理工作，我认为 Hankwave 仍然是最老牌、最稳定、最开放的。我会继续保留这个 skill，方便你想试的时候使用。第二，需要某种方式来限定实验的范围。除非能把成本、时间和访问权限作为资源加以控制，否则就无法进行测试。第三，知识和方法论。forward is to list them, look out for them, and apply specific solutions. So going back, what are the four things we're missing to be able to self-optimize? Number one is a stable deployment surface. We use Hank for this, a harness that takes the fastmoving surface of agent runtimes like cloud code and PI and that exposes a consistent welltested log and API interface with checkpointing, roll back, budget enforcement, everything built in. Something you can build on. But other things fit this description. To be fair, for headless data work, Hankwave, I would say, is still the oldest, most stable, and the most open. I'll leave the skill up if you'd like to try it. Number two is some way to bound your experiments. You can't test unless you can control cost, time, and access as resources. The third one is knowhow and methodology.

我们发现，这类新模型能够运用我们在两年的部署过程中积累的一切经验，包括智能体模式、调试技巧和提示词方法，并将其应用于复杂问题。借助恰当的抽象层，你可以用极少的 token 表达这些内容，而模型能够将其实现。第四点，也是最重要的一点，是失败面。我们的委员会说过：“给我一个立足之处，我就能撬动地球。”而对我们来说，只要有一个稳定的技术栈可供构建，并有一个失败面可供反复迭代，我们还没有遇到过无法通过不断迭代来解决的问题或约束。最后这一点最难用语言表达，也最难转化为工具和指令来帮助……We found that this new class of models can take everything we've learned from two years of deployments, agentic patterns, debugging techniques, prompting and apply it to complex problems. With the right abstraction layer, you can express it very token efficiently and the models can implement it. The fourth and the most important one is a failure surface. Our committee said, give me a place to stand and I will move the earth. And in our case, we haven't met a problem or a constraint that we couldn't iterate ourselves to solving given a stable stack to build on and a failure surface to iterate against. That last one has been the hardest to put into words and into tooling and instructions that helps

有界试点与可复用的已检查片段Bounded pilots and reusable checked segments

模型最终会达到那个水平。可以说，目前最好的模型在展现这些智慧火花方面仍然只是勉强称得上不错。不过，整个流程包含六个阶段。首先，我们明确任务规格并将其拆分。对 Jeff 来说，这意味着把大约 80 万行数据拆分成 20 万个更小、更专门的案例，再把每个案例拆分成类型明确的问题。比如捐赠者的类型、单个捐赠者，以及其他需要回答的不同问题。 然后，我们确定失败边界。在这个案例中，我们用优秀的模型处理了一小部分数据，并对其中一部分进行了人工检查，从而设定一个基准，供我们据此判断失败。在其他情况下，我们设置过自动化的失败检测系统，也设置过架构检查。我们还把行为纳入了可衡量的评估范围。但非常重要的一点是，必须在开始之前就定义好失败边界。models get there. The current best models are still only barely good at this sparks of intelligence, so to speak. The whole process though involves going through six stages. First, we spec the job and we split it. For Jeff, that means splitting some 800,000 rows into 200,000 smaller specialized cases and each of those cases into typed questions. What kind of donor, one donor, you know, different different questions that needed to get answered. Then we decide the failure surface. In this case, we ran good models on a small slice of the data with some of it being manually checked to set a baseline that we can fail against. In other cases, we've set automated failure systems. We've set schema checks. We've also covered behavior as a thing that we can measure against. But it's very important to have to define a failure surface before you ever start.

但现在才是有趣的部分。我们会运行范围最小、边界明确的试点来测试架构。一旦某些东西开始奏效，我们就把这些检查移入运行架——也就是在有界工作片段之间运行的代码，我们称这些片段为“密码子”。随后，这些密码子会被重复利用、独立循环运行、经过验证，并按顺序排列。最后，可用的流水线会部署到生产环境，在那里逐步成熟。而中间这部分，正是冲浪发生的地方。我们找到一个能产出可靠结果的架构，并以此为基准。然后，我们把模型和其他一些部分换成更快、更便宜的选项，与已知的中间结果进行比较，每次都收紧成功标准，不断向理想状态靠近，然后你只需看着前沿不断移动。能够把复杂问题拆解成若干部分，意味着每个部分都能各自移动到最适合自己的位置。But now comes the fun part. We run minimal bounded pilots to test architecture. And once something starts working, we move those checks into rigs. Code that runs between bounded work segments that we call codons. These codons are then reused, looped independently, validated, and placed in sequence. Finally, working pipelines get deployed to production where they can slowly mature. And that middle part is where the surfing happens. We find an architecture that gives a solid result. We use it as a baseline. Then we swap models and some other parts for faster, cheaper options, compare against known intermediates, tighten the requirements for success each time, moving closer and closer to the ideal, and you just watch the frontier move. Being able to break a complex problem into pieces means each piece can move its own to its own best position.

正如亚当·斯密在这样的体系中会说的那样，本体作为智能与结果之间的中介而存在，因此更容易被重复使用。这就是我所说的“即时本体”的另一半含义：工作需要时就构建它，连同相关凭据一起保存起来，等下一项工作需要同样的东西时再重复使用。有了这些本体之后，成本就能降下来。你也可以在多个数据集上并行运行它们，从而扩大规模。 最后我想道个歉：答案其实是，没有什么本体——至少从来就不存在一个普适的本体。好消息是，只要能够标注并保留本体和模式周围的上下文，每一种本体和模式就都有效。如果我们能善用这种正开始在周围世界中逐渐普及的低成本智能，那么我们很快就会迎来这样的未来：任何格式的信息都能发挥尽可能大的作用。As Adam Smith would say in a system like this, ontologies exist as intermediates between intelligence and result, which makes them easier to reuse. That's the other half of what I mean by just in time ontologies. It's built when the job needs it, kept with its receipts, and reused when the next job needs the same thing. And once you have those, you can make these things cheaper. You can you can run them in parallel across data sets to scale up. I close with an apology, right? That the answer is there is no ontology. Not a universal one, at least there never was one. The good news is that this makes every ontology and schema valid as long as you can label and preserve the context around it. And if we can harness this cheap intelligence is starting to percolate through the world around us, we're rapidly approaching a future where any information in any format is made as useful as it can be.

在 Subridge，我们正竭尽全力，让这一未来成为现实。谢谢。And at Subridge, we're working as hard as we can to make that future a reality. Thank you.