读见

为什么我们删掉了自己的 MCP 服务器并重建 — Abhi Arya,Reducto

原题Why We Deleted Our MCP Server and Rebuilt It — Abhi Arya, Reducto

约 12 分钟原文英文已翻译

视频由 YouTube 提供

看前速览

5 个要点

Reducto 产品工程师 Abhi Arya 讲述团队为何删掉按端点逐一生成工具的 MCP 服务器,并用更少但携带工作结构、可见不确定性与人类校验的工具将其重建,让智能体真正服务于用户。

  1. 智能体优先软件是一个架构问题:智能体出错时由谁付出代价

    讲者对比了三种姿态:自动模式为自主性优化,让自信的失败无声流向客户;用户优先把人的理解与智能体的能力一起封顶成聊天机器人体验;胜出的一类是服务于用户的智能体体验,提供丰富而受约束的能力,人仍然可以校验。

  2. 每个端点配一个 MCP 工具的版本在真实使用中立刻崩溃

    第一版服务器把应用的每个端点都暴露成工具。喂进半成品客户通话记录时,智能体会端到端建完整条工作流并自信宣称一切正确,但用户和智能体都解释不了建成的东西,这是架构失败,不是提示词问题。

  3. 更少但携带真实工作结构的工具取代了被删掉的服务器

    单一的分类转提取工具取代了反复调用通用创建步骤,每步之后有上下文引导,快照工具则返回带具体错误码的实时流水线状态,让无状态的智能体引用真实存在的东西,而不是臆造变量名。

  4. 让不确定性可见,把静默的运行期失败变成可校验的判断

    提取输出置信度、边界框等信息回流模型以优化 schema;评测对含糊假设直接判失败,脚手架引导智能体向用户提问而不是瞎猜,一个澄清性问题同时提供了上下文、能力和人类问责。

  5. 所有失败模式同根同源,插桩会话让系统形成自改进闭环

    一切失败都归结为智能体拥有的自主性和访问权限超过人类可验证的范围。记录每个 MCP 会话中人在哪里推翻智能体、置信度在哪里走低,并回流到长期引导中,销售团队甚至主动把工具接进了客户通话前的准备流程。

完整摘要

Reducto 产品工程师 Abhi Arya 从女朋友问他的一个问题切入:什么是智能体优先软件。他论证这本质上是一个架构问题,核心在于智能体出错时由谁付出代价。他把现有做法分成三类:自动模式为自主性做优化,让自信的错误无声触达客户;用户优先把人的理解和智能体的能力一起封顶成聊天机器人;胜出的是服务于用户的智能体体验,给模型丰富而受约束的能力,人仍然可以校验。公司第一版 MCP 服务器把每个 API 端点都映射成一个工具、塞进一个大文件,在精心策划的演示里表现尚可,一旦交给全团队接入真实输入就立刻崩溃:智能体端到端建完整条工作流,却自信宣称一切正确,用户和智能体都无法解释它到底做了什么。他一个 commit 删掉整个文件,换成更少但携带真实工作结构的工具:单一的分类转提取工具、每步之后的上下文引导,以及返回实时流水线状态和具体错误码的快照工具。由于运行期失败总是悄然而至,重建让不确定性变得可见:置信度、边界框与错误信息回流给模型,评测对含糊假设直接判失败,并奖励向用户提问而非瞎猜。他把所有失败模式归结为同一个根因,即智能体拥有的自主性和访问权限超过了人类可验证的范围;被完整记录的 MCP 会话形成自改进的引导闭环,销售团队也主动把工具接进了客户通话前的准备流程。整场是一手厂商叙事,没有量化成效数据,也没有现场演示。

章节

文字稿

查看原文

简要描述Brief Description

为什么我们删除了 MCP 服务器并重新构建——Abhi Arya,ReductoWhy We Deleted Our MCP Server and Rebuilt It — Abhi Arya, Reducto

目录Table of Contents

  • 简介
    Intro
  • 30 亿份文档
    3 billion documents
  • 什么是智能体优先软件?
    What is agent-first software?
  • 代理出错时,由谁买单?
    Who pays when the agent is wrong?
  • 自动模式、用户优先,或为用户服务
    Auto mode, user-first, or in service of the user
  • 卡在配置阶段
    Stuck in configuration
  • 版本 1:每个端点对应一个工具
    Version 1: a tool for every endpoint
  • 它立即就坏了。
    It broke immediately
  • 更少但具备真实结构的工具
    Fewer tools that carry real structure
  • 实时状态的快照
    A snapshot of live state
  • 不要只是把一个 API 交给代理使用
    Don't just hand agents an API
  • 悄无声息发生的故障
    Failures that happen quietly
  • 让不确定性显而易见
    Make uncertainty visible
  • 询问,不要猜测
    Ask, don't guess
  • 一个根本原因
    One root cause
  • 监测埋点与自我改进闭环
    Instrumentation and a self-improving loop
  • 销售团队主动采用了它。
    Sales adopted it on their own
  • “以智能体为先”真正意味着什么
    What agent-first really means

简介Intro

主持人:我叫 Avi,在 Reducto 负责产品。在马上开始之前,我想先给大家介绍一下 Reducto 的背景。嗯,我们是一个智能体式文档平台,简单来说,就是帮助团队将杂乱、非结构化的文档转化为数据,让你们的工具和 AI 智能体能够理解这些数据。我稍后会介绍智能体本身,不过我想先……Host: So, I'm Avi, and I work on product here at Reducto. And before I dive in right away, I wanted to give you guys some more context on what Reducto is. Uh we're an agentic document platform, which basically means that we help teams turn messy, unstructured documents into data that your tools and your AI agents can understand. I'll jump into the agents themselves in a second, but I'd

30亿份文档3 billion documents

我想先简单聊聊我们目前的进展。过去 3 年里,我们处理了超过 30 亿份文档,客户包括 Harvey、Scale AI,以及全球排名前五的科技公司和对冲基金。以这样的规模运行,意味着我们基本见过文档在生产环境中可能出现的各种问题,也不断迭代产品和模型,以应对几乎任何你能遇到的情况。如今,用前沿模型处理格式干净的文档,已经不算太难了。基本上,你可以把东西扔到云端,或许就能完成 60% 左右。但难点在生产环境:扫描件、页面旋转等问题都会出现。模型也会开始产生幻觉,因为企业数据中绝大多数都是非结构化的。因此,Reducto 就是位于模型与这些数据之间的一层。like to talk about where we're at first really quick. In the last 3 years, we've processed over 3 billion documents across customers like Harvey, Scale AI, and top five global tech companies and hedge funds. Running at that scale means that we've seen basically every way a document can break in production, and we've iterated our product and our models to handle basically anything that you can throw at them. And getting AI to work on a clean document with a frontier model is not super hard anymore. We basically like you can throw something in the cloud, and you might get like 60% of the way there. But the hard part is production. Scanned documents, rotated pages, etc. And the model starts hallucinating because a large majority of enterprise data is unstructured. And so Reducto is a layer that sits between your model and that data.

我们负责处理所有复杂的版式和后处理工作,因此在设计这些流水线的过程中,我们自己也会使用并构建大量智能体。好了,既然你现在对我们的工作多了解了一些,我就从一个看似无关的话题切入我们的主题。嗯,我……We handle all the complex layouts and post-processing, and that way we end up working with and building a lot of agents ourselves while designing these pipelines. So, now that you know a little more about what we do, I'm going to dive into our main topic with something seemingly unrelated. Um I

什么是智能体优先软件?What is agent-first software?

上周末我去看了我女朋友,她给了我一张《贱女孩》电影贴纸。基本上,它是智能体版的贴纸,上面写着:“上车吧,笨蛋。我们要构建智能体优先的软件。”作为一个典型的软件工程师,我当然得把这样的贴纸贴在笔记本电脑上。她是从工作单位拿到这张贴纸的,但她问我:“什么是智能体优先的软件?”因为她不从事 AI 领域。我当时一边琢磨,一边意识到自己能给出很多不同的答案:构建一个非常好的 harness,构建一个非常好的 API,编写或迭代提示词。这些都没有真正把智能体放在首位。它们更多只是在优化模型。智能体优先的软件本质上是一个架构问题。所以我认为,那种结构,那种……visited my girlfriend this past weekend, and she gave me a sticker from the movie Mean Girls. And basically, it was an agentified version of it. So, it says, "Get in, loser. We're building agent-first software." And in true software engineer fashion, I had to put a sticker like that on my laptop. And she got this from work, but she asked me like, "What is agent-first software?" cuz she doesn't work in AI. And as as I was like doing so or as I was thinking about it, I realized I have a lot of different answers. Building a really good harness, building a really good API, writing or iterating on prompts. All of these don't really um land on the agent being put first. They're more so just optimizing the model. Agent-first software is inherently an architecture problem. And so, I think that structure, that

智能体出错时,谁来承担损失?Who pays when the agent is wrong?

架构归根结底取决于一个问题:代理出错时由谁买单?对此,我认为大致可以分成三类。所以,有汽车……architecture, really comes down to one question, and that is who pays for the agent being wrong. And to that, I think there are like three buckets that you can put this into. So, you have auto

自动模式、用户优先,还是服务于用户Auto mode, user-first, or in service of the user

以自主性为优先的模式开始:你写好提示词,然后让智能体想做什么就做什么。等它出错时——它肯定会出错——你却毫不知情,因为它对自己在做什么非常自信,而最终要为它犯错向客户、用户等承担代价的是你。然后是以用户为先的模式。在这种模式下,你优先优化人机界面,把所有控制选项都交给用户;接着,你构建的智能体却几乎做不了什么,基本上就只是聊天机器人的体验。这样一来,你既限制了用户对产品的理解,因为他们无法给它写提示词,也限制了智能体自身的能力。最后,我认为真正胜出的方向,是以服务用户为目标来设计智能体体验。在这种模式下,智能体的能力经过精心设计,为模型提供丰富且范围明确的能力,让它可以完成用户真正想做的事,帮助并赋能用户;同时,人也仍然可以核验它所完成的工作。mode to start, where you optimize for autonomy, you write a prompt, and you let the agent do whatever it wants. And when it messes up, which it will, you won't have any idea because it's really confident about what it's doing, and you're the one that's paying for it being wrong to your customers, your users, etc. Then you have user-first. This is where you optimize the human interface, give the user all the knobs, and then you build on an agent that can't really do much, which is basically just a chatbot experience. You've both capped the user's understanding of the product because now they can't prompt it, and the agent's capability within as well. And lastly, the bucket that I think wins here is agent experience in service of the user. This is where the agent's capabilities are structured to give the model um rich and scoped capabilities where it can do things that the user really wants, aiding and enabling them, and even where the human can still verify the work that it's doing.

正是这种框架,让智能体体验不再那么取决于模型本身,而更多取决于上下文、能力以及整体的自我改进。This framing is what turns agent experience into less of a question of the model and more of a question of context, capability, and overall self-improvement.

卡在配置中Stuck in configuration

下面是我如何发现这些类别,并得出最后那个观点的。在 Reducto,我们一直在开发一些工具,通过流水线基本实现端到端的文档处理自动化。比如,先解析文档,再对它进行分类,或者把它拆分成不同部分,然后从中提取信息。嗯,这些工具适用于发票处理、合同管理,以及所有需要人工参与的场景。我很早就注意到,用户,尤其是我们 Reducto 的销售团队,把所有时间都花在配置上,而不是关注成果。这是一个普遍存在的更大问题,因为传统界面的设计初衷就是用来搭建这类流水线的。与此同时,在这种智能体式的工作世界里,我们越来越关注最终交付成果,以及如何尽可能快地帮助用户实现目标。And here's how I discovered these buckets and came to that last point. At Reducto, we've been building tools that basically automate end-to-end document work through pipelines. Think that we first parse a document, then we classify it, or maybe split it into sections, and then extract from it. Um and that's in use cases like invoicing, contract management, and all these things that require human input. One thing that I noticed early on was that any of the users, especially our sales team at Reducto, was spending all their time in configuration instead of outcomes. And that's a larger problem across the board because legacy interfaces were designed for kind of building that pipeline. Meanwhile, in this kind of agentic work world, we're increasingly focused on the final deliverable and how can we get the user to that point as fast as possible.

版本 1:每个端点对应一个工具Version 1: a tool for every endpoint

所以,对我们来说,要实现这一点,显而易见的做法就是搭建一个 MCP 服务器。让智能体承担繁重的工作,用户只需要考虑自己的使用场景并编写提示词。为了验证这个方向,我用 Claude Code 很快搭了一个 MVP,把驱动我们应用的每个 API 端点都做成了 MCP 工具,全塞进一个巨大的文件里。我给它写了一条非常明确的提示词,把它发到了 Slack 上。说实话,效果还挺不错。嗯,在某种程度上,它在那种环境下本来就应该奏效。我很清楚自己在做什么:我是在制作一个精心设计的演示,这也正是为什么那个独立演示根本说明不了任何问题。真正的考验是在我把它交给……的时候发生的。So, the clear move to get here for us was an MCP server. Let the agent do the heavy lifting and only burden the user with thinking about their use case and writing a prompt. To get some validation for the direction, I threw together an MVP with Claude code, and every API endpoint that drove our app was given an MCP tool in one massive file. I gave it a really pointed prompt. I put it in the Slack, and honestly, it was pretty nice. Um to some extent, it should have worked in that setting. I knew exactly what I was doing. I was building a really curated demo, and that's why the standalone demo told me nothing at all. The real test happened when I gave the

它立刻就坏了。It broke immediately

把 MCP 开放给团队里的每个人后,问题立刻就出现了。事情是这样的:有人会把客户通话中一段尚未成形的描述,或者 Notion 页面里的内容交给它,智能体就会直接开始干活。它会从头到尾构建出完整的工作流,而且还会非常自信地告诉用户,自己做的一切都没错;但无论是用户还是智能体,都说不清这个工作流实际做了什么。这才是自动模式在生产环境中带来的根本性警醒。智能体从来不会说自己不确定,而是动用手头所有工具来处理;整个过程中,也没有任何环节能让人真正验证或思考到底发生了什么。我们构建出了一个能与系统通信的东西,却没有构建出一个能真正与系统协作的东西。你可能会觉得,解决办法是把系统提示写得更详细,或者给模型更多指导,但实际上,问题出在架构上。MCP to everyone on the team, and immediately it broke. And here's how. Someone would feed it a half-baked description from a customer call or maybe a Notion page, and the agent would just start going. It'd build the entire workflow end-to-end, but very confidently, it would tell the user that everything it did was right, but neither the user nor the agent could explain what the workflow actually did. And that is the root awakening of auto mode in production. The agent never really said that it's not sure. It handled with every tool that it had, and there was nowhere for a human to really validate or think about what was going on. We built something that could communicate with our system, but we hadn't built something that could actually work with it. And you'd think that the fix here is more detail on the system prompt or more model guidance, but realistically it's architecture.

我们之前用来与 MCP 通信的整个文件,我把那个……删掉了。The whole file that we used to talk to the MCP, I deleted the

更少但具备真实结构的工具Fewer tools that carry real structure

整个东西都放在一个 commit 里。而取代它的并不是更多工具,而是更少的工具,只不过这些工具承载了实际工作的结构。比如,我没有让工作流中的某个 create 步骤被反复调用,然后只能祈祷 agent 能在流水线中正确组装好;我构建了一些能够理解我们的内部仪表盘或产品如何运作的工具。举例来说,Reducto 的一个常见用例是将我们的 classify endpoint(用于给文档分类)路由到一个 extract 节点。它基本上就是提取下游所需的信息。我没有让模型通过提示词来猜测或推断该怎么做,而是把它整合成一个单一的 classify-to-extract 工具。每完成一个步骤,agent 都会得到关于接下来该做什么的上下文指引,有些工具甚至会像这个工具一样自动串联起来。entire thing in one commit. And what replaced it wasn't more tools. It was mostly fewer tools, but they carried the structure of actual work. So for example, instead of one create step of this workflow to uh call over and over, and just praying that the agent assembled right in the pipeline, I built tools that understood how our internal dashboard or our product works. For example, a common use case with Reducto is to route our classify endpoint that classifies documents into an extract node. And what that does is basically extract information downstream. And instead of basically letting the model guess this or figure it out through the prompt, I turned it into a single classify to extract tool. And after each step, the agent gets contextual guidance on what actually comes next, with certain tools even chaining automatically like this one.

与其让智能体去猜测或思考什么是正确的,不如直接告诉它什么是正确的。现在,智能体不能再对我们确定为真的部分自由发挥。而且……Rather than the agent having to guess or think about what is right versus being told what's right. Now the agent doesn't get to freelance the parts that we know are true. And

实时状态快照A snapshot of live state

由于智能体是无状态的,我们甚至还添加了一个快照工具。通过一次调用,它基本上就能返回我们流水线的实时状态,包括整个流程的上下文,以及出现的错误和非常具体的错误代码。这样一来,它们就能参照实际存在的内容,而不是猜变量名、产生幻觉等。把上下文从提示词中移出,转而用结构化的方式呈现后,智能体不再猜测,而是开始查阅实际信息。并且,由于这些工具只能让流水线采用我们这些产品工程师所期望的形式,人类就能准确了解发生了什么,而我们也能向下游用户解释智能体的输出……since agents are stateless, we even added a snapshot tool. So this is one call that basically hands back the live state of our pipeline from the context of the entire flow to the errors that were surfaced with like really specific error codes. And that way they can reference what's actually there instead of guessing variable names, hallucinating, etc. By moving context out of the prompt and into structure, the agent stopped guessing and started referencing. And because the tools only let pipelines take the shape that we as the engineers on the product wanted, the human could follow exactly what happened and we could explain the agent's outputs to downstream users of

不要只是给智能体一个 APIDon't just hand agents an API

这个 MCP 工具。因此,为了避免自动模式,你不会直接给你的智能体一个 API,而是给它一些工具,这些工具承载着整体专业知识,例如正确的模式、状态和验证器,这样一来,正确的路径也就成了 MCP 最容易采用的路径。这样,我们现在就能构建出良好的流水线了。我们完成了,对吧?其实并没有。回想一下自动模式,智能体自信满满地犯了错,却没人看得出来。嗯,我们刚才讨论的内容解决了构建阶段的问题。这些工具让结构变得清晰易懂,人也能跟上智能体做了什么。但现在,问题出现在运行时:如果智能体试图构建的结果本身出了问题,谁能发现呢?例如,如果我们的架构设计得很糟糕,或者用户根本没有明确说明实际用例,我们怎么知道智能体做了……this MCP tool. So, to avoid auto mode, you don't hand your agent an API, you hand it tools that carry the overall expertise such as the right patterns, the state, and the validators so the correct path also becomes the simplest path that the MCP can take. So, we can build good pipelines now. We're done, right? Not really. Thinking back to auto mode, the agent was confidently wrong and no one could see it. Um that what we just talked about fixed that for building. The tools made the structure really legible and the human could follow what the agent did. But now the problem showed up at runtime. When the outcome itself that the agent is trying to build goes wrong, who can tell? For example, if we had a really bad schema or maybe the user didn't really specify the actual use case, how can we tell that the agent did

悄无声息发生的故障Failures that happen quietly

出了什么问题?而智能体出错时,往往悄无声息。也就是说,不会出现段错误或其他实际错误,嗯,整个流程会处理 100 份文档,却可能漏掉第 30 页的扫描内容;又或者,用户没有说明要将哪些类型的文档送入流程。基本上,这两种情况都是同一种失败。智能体本身会非常自信地给出错误结果,而它所具有的不确定性则完全不可见。所以,我们为了解决这个问题所做的一切,归根结底只涉及一项改变,就是让……something wrong? And with agents it goes wrong quietly. So, instead of a seg fault or some actual error, um the pipeline runs across 100 documents and maybe it misses scan on page 30 or maybe the user didn't specify what types of documents were going into the pipeline. And basically both of these things are the same failure. The agent itself is very confidently wrong and the uncertainty that the agent has is completely invisible. So, everything that we did to fix this basically came down to one uh change, which was making

让不确定性显而易见Make uncertainty visible

让不确定性真正变得清晰可见。在我们的系统中,或者使用 Reducto 时,嗯,失败也会带来信息。因此,使用 Extract 时,我们会输出置信度分数、边界框,以及所有这些关于哪里出了问题的信息;我们能够构建工具,将这些信息反馈给模型,这实际上有助于随着时间推移优化提取 schema,并生成更准确的文档输出。但那只是结果层面的不确定性。智能体也必须显露自身的不确定性。当它遇到模棱两可的情况,比如一个字段有两种解读方式,或者遇到它没见过的文档类型时,它不应该猜测,而且它……uncertainty really legible. In our system or with Reducto, uh failures carry information. So, with Extract, we output confidence scores, bounding boxes, all of this information about what went wrong, and we were able to build tools that feed that information back into the model, which actually helped optimize the extract schema over time, and get more accurate document outputs. But, that's just the outcome uncertainty. The agent also has to surface its own uncertainty. When it hits something ambiguous, um two ways to read a field or a document type that it hasn't seen, it shouldn't guess, and it

有疑问就问,别猜。Ask, don't guess

不应该只是停在原地。相反,它应该说:“这件事有两种理解方式”,然后问用户:“你的意思是什么?”所以,基本上,在我们的评测中,只要智能体做出任何站不住脚的假设,我们就会判它失败;我们还为我们的 harness 提供了提示指导,让它使用 Cloud Code 和其他前沿 harness 提供的 ask user question 工具等,而不是靠猜测。因此,我们奖励的是提问,而不是猜测。而判断力并不只存在于智能体本身。整个流程仍由用户监督,随后,智能体的置信度和推理过程会作为提示指导输出到 MCP 中。这样一来,每次你创建新内容时,它都会向用户说明自己做了什么、为什么这么做,让用户能够审核;从本质上说,这一切都是为了服务用户。shouldn't just pause exactly where it is. And rather, it should ask, "I can read this two ways," and tell the user, "What did you mean?" And so, basically in our evals, we started failing the agent for building any shaky assumptions, and we gave our harness uh prompt guidance to use the ask user question tools, etc., that Cloud Code and other like frontier harnesses offer, instead of guessing. So, we rewarded asking instead of guessing. And judgment doesn't live in the agent alone. This entire pipeline is still overseen by the user, and the confidence and the reasoning of the agent was then outputted as basically prompt guidance in the MCP. So, now every time you make something new, it explains to the user what it did and why, so the user can then review it, inherently acting in service of the user.

智能体基本上满足的是你写下的内容,而不是你想要的结果。那么,如果你让不确定性变得更清晰可见一些,同时把最终判断交由人来作出并承担责任,会怎么样?这样,一个澄清问题就能同时发挥这三种作用:为模型提供上下文和能力,也让人参与其中,真正形成判断并加以核验。The agent basically satisfies what you wrote, and not the um outcome that you wanted. So, what happens if you make the uncertainty basically a bit more legible, and you keep the final judgment where it can actually be accountable in the hands of the human, then that one clarifying question does the work of all three, giving the model context, capability, and also adding a human in the loop to actually build that judgment and verify

一个根本原因One root cause

等等这些方面也是如此。所以,回到总体上这些失效模式,我们一开始有一种自动模式,谁都无法检查。错误答案悄无声息地出现。还有一个智能体通过奖励投机或隐瞒自己的操作通过了测试。归根结底,它们的本质都一样:智能体拥有的自主权和访问权限,超出了人类能够核实的范围。而智能则体现在整个构建过程中。因此,构建流水线、生成架构,以及引入恰当的上下文,正是智能体最能发挥作用的地方;同时,一旦出错,代价也最低,因为人类就在一旁。通过围绕这个界面进行构建,我们也让用户满意度随着 MCP 的使用而提升,因为用户确实能看到发生了什么。而借助 Reducto 处理文档层,我们可以确保流水线中那些不能信任智能体处理的部分得到准确的解析和提取。things as well. So, going back to kind of all of these failure modes overall, we had initially an auto mode that nothing nobody could inspect. We had wrong answers that basically arrived in silence. And then an agent that passed the test by reward hacking or hiding what it did. And underneath, they're basically one shape where the agent had more autonomy and more access than the human could verify. And that intelligence lives in the construction overall. So, building the pipeline, generating the schema, and pulling in the right context is where the agent is more most powerful and where a mistake is the cheapest because the human is right there. By building for that interface, we were able to also have user satisfaction go up with the MCP because they could really see what was going on. And by using Reducto for the document layer, we can guarantee accurate parses and extractions for parts of the pipeline that we can't trust agents with.

智能体无论在哪里运行,都是通过可验证、正确地行事,并为用户服务而建立信任。我们之前关于整体可理解性的所有讨论,最终都会发挥作用。智能体不过是装在盒子里的智能,而包围它们的运行框架,赋予了它们交付这些成果的能力。Where the agent operates, it earns trust by being verifiable, correct, and building trust in service of the user. And the whole conversation we had about legibility overall is what inherently pays off. Agents are just intelligence in a box and the harness that you surround them with is what provides them with the ability to deliver these

监测机制与自我改进循环Instrumentation and a self-improving loop

下游结果。因此,一旦到了这一步,我们就会重点关注埋点:记录每个 MCP 会话,以及模型发送的提示词;我们甚至还请销售团队和团队中的其他成员导出他们的 Cloud Code 会话,以了解它的运行情况——人类在哪些时候覆盖了智能体的操作、模型的置信度何时降低,以及它在哪些时候提出了问题。由于我们添加了所有这些埋点信息,它们也会反过来融入 MCP 的长期整体指导中。这样一来,系统就能通过构建更多流水线,不断提升构建流水线的能力,而无需我们手动重写。这就形成了一个针对每个组织或每位用户的自我改进循环,他们都在构建这些流水线。总体而言,基础仍然是同一个决策:我们让模型受到充分引导,并且整个过程都可供检查;而围绕模型的其他一切信息,则共同构成了实际引导模型的内容。outcomes downstream. And so, once we reach this point, we focus on instrumentation where every MCP session, the prompts that were sent from the model were logged, and we even asked like our sales team and other people on the team to basically export their cloud code sessions to understand how how was behaving. Where humans overrode the agent, where confidence of the model ran low, and where it asked questions. And because we added all this instrumentation, um it flowed back into the long-term guidance that the MCP had overall. So, the system got better at building pipelines by building more pipelines without us rewriting by hand. This then became a self-improving loop per org or per user that was building these pipelines. The foundation became overall the same decision where we kept the model very guided and very inspectable, and the information around it became everything else that actually guided the model.

所以,有一件事多少让我觉得我们在这些改进上做对了,那就是我从来没有……And so, one thing that kind of told me that we were doing something right with these improvements is I never like

销售团队主动采用了它Sales adopted it on their own

我并没有明确要求销售团队使用这个智能体来构建销售管道,而只是把它作为产品工具部署在我们的 Cloud Enterprise 中。销售团队以及团队里的其他人基本上开始把它接入自己的工作流程。比如,在与客户通话之前,他们会使用 CircleBack 或 Slack 了解客户情况,然后通过 MCP 向它询问有关客户的问题,再为该客户构建后续的销售管道。因此,当智能体的使用体验真正围绕使用它的用户来打造时,我们并没有直接推动大家去用它,而是他们主动找上门来,并开始把它当作全公司的资源来使用。也正是在那时,他们开始足够信任它的输出,愿意把它展示给高价值客户,因为演示不仅运行顺利,而且有人亲口说:“嘿,这是我做的。”explicitly told the sales team to use the agent to build pipelines, but rather I just exposed it in our Cloud Enterprise as product tools. And the sales team and various other people on the team started basically wiring it into their workflows. So, before they had a customer call, they would use CircleBack or Slack and basically learn about the customer and then use the MCP to basically ask them questions about the customer and then build a pipeline for that customer downstream. So, when an agent's experience was built really in service of the users that were using it, we weren't pushing people directly toward it and rather they reached for it and actually started using it as a company-wide resource. And that's where they also started to trust the output enough to put this in front of high-value customers because the demo didn't just work, but the human actually said, "Hey, I built this.

“来看看吧。”而且这在这些销售电话中确实奏效了。我也希望 Slack 上的反应能帮助衡量这是否是一个……Come check it out." And it actually paid off in these sales calls. And I hope the Slack reactions can also help gauge that a

“以智能体为先”真正意味着什么What agent-first really means

一点点。所以总的来说,几个月前我会告诉你,所谓 agent 优先,就是拥有最好的模型,以及模型最快的推理速度或最高的智能。而随着模型不断进步,从闭源走向开源、从昂贵变得便宜,我觉得它现在的含义几乎和字面听起来的意思相反。在某种程度上,agent 并不是优先的。要打造以 agent 体验为先的产品,就意味着你必须考虑让 agent 与那些最终需要为工作结果负责的人一起参与其中。所以,问题不只是 agent 能连接到什么?如今每个人的 agent 都能连接到各种东西,而且 Claude 可以马上为你创建一个新的连接器。但除此之外,如果你的 agent 非常自信却犯了错,谁会发现?他们发现得有多快?如果你围绕这个问题来构建,就能打造出人们真正愿意托付实际工作的产品。little bit. So, overall, a few months ago I would have told you that agent first is means that you have the best model, the best inference speed or intelligence of the model. And with models getting better from closed source to open source, from expensive to cheap, I think it now means almost the opposite of what it sounds. To some extent, the agent doesn't go first. And to build for agent experience means you have to consider the agent being in the loop with the people who have to answer for the work downstream. So, it's not just what does the agent connect to? Everyone's agent connects to everything nowadays, and Claude can make a new connector for you immediately. But, regard Beyond that, it's when your agent is really confidently wrong, who finds out and how fast did they find out? If you build for that, then you can build for something that people will actually trust with real work.

差不多就是这些。谢谢大家。另外,我们在 Reducto 的 P8 展位,正在招聘很多职位。欢迎来 reductoai.notcurious 看看。谢谢。And yeah, that's about it. Thank you, guys. And also, we're at Reducto's at a booth P8, and we are hiring for a lot of positions. So, please come check it out at reductoai.notcurious. Thank you.

值得记下

智能体优先软件本质上是一个架构问题。
我们造出了能和系统通信的东西,却没造出能真正与之协作的东西。
把上下文从提示词移进结构之后,智能体不再瞎猜,而是开始引用。
不要把 API 直接交给智能体,要交给它携带正确模式、状态与校验器的工具,让正确路径同时成为最简单的路径。
真正要问的是,当你的智能体自信满满地出错时,谁会发现,又有多快发现?