从实验走向生产：Credit Karma 如何治理企业智能体

简要说明Brief Description

为生产环境铺平道路：受治理的企业级 AI 智能体——Harish Gaggar，Credit KarmaPaving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar， Credit Karma

目录Table of Contents

• 从实验到企业级智能体
  From experiments to enterprise agents
• 缺失的生产基础设施
  Missing production foundations
• 智能体及其支撑框架
  The agent and supporting frameworks
• 注册、本地开发与铺就的路径
  Registration, local development and the paved path
• 复习、容器与 Kubernetes 部署
  Review, containers and Kubernetes deployment
• 已审核的自定义 MCP 服务器
  Reviewed custom MCP servers
• MCP 网关身份验证与路由
  MCP gateway authentication and routing
• 日志、链路追踪和网关指标
  Logs, traces and gateway metrics
• LangGraph 策略与人工审批
  LangGraph policies and human approval
• 线程、运行与内存检查点
  Threads, runs and memory checkpoints
• 智能体跟踪与评估记录
  Agent tracking and evaluation records
• 筛选敏感输入
  Filtering sensitive inputs
• 上下文优化与 Token 减少演示
  Context optimization and token reduction demos
• 围绕每个智能体构建框架
  Build the framework around every agent

从实验到企业级智能体From experiments to enterprise agents

大家好。嗯，我想带大家了解一下，我们如何将 AI 智能体从实验阶段过渡到真正的企业环境。开始之前，我先简单介绍一下自己。我叫 Sharish Gokar，是 Intuit Credit Karma 的高级首席软件工程师，在那里负责构建并维护 AI 分析基础设施和服务。除此之外，嗯，我还是顾问、作家、研究人员，也担任多个组织的项目委员会成员，同时指导年轻的研究人员。好的，接下来我们来看看今天的议程。首先，我们将讨论构建 AI 智能体并将其部署到生产系统时会遇到哪些挑战，以及如何构建不同的框架来应对这些挑战。Hello everyone. Um, I would like to walk you through that how we can transition in AI agents from an experiments to the real enterprise environment. So, before I begin, a quick introduction about me. My name is Sharish Gokar. I'm a senior staff software engineer at Intuit Credit Karma where I built and own the AI analytics infrastructure and services. Um, and beside that, um, I'm advisor, uh, author, researchers, um, and also a program committee members for various organizations where I mentor the young researchers as well. All right. So, let's, uh, walk through the, um, agenda here. So, first we would like to talk about what are the challenges we have when we build an AI agents and deploy into the production systems and how we, uh, can build a different framework to address that, uh, different challenges here.

嗯，然后讲讲它的架构是什么样的。嗯，接着讲讲如何构建自己的 MCP 服务器和 MCP 网关，以及如何在网关之上进行监控。嗯，之后我们还会讨论可以使用哪些不同类型的智能体框架，以及一些用于管理记忆、跟踪和评估的其他框架；如何拦截发送给 LLM 的 PII 数据；还有如何通过上下文优化器来节省 token，对吧？Um, then how the architecture looks like for it. Um, then, um, how you can build your own MCP servers, um, and MCP gateway. And then some monitoring on top of that gateway as well. Um, then further we're going to talk about what are the different type of agent framework you can have, um, and then, um, some additional framework around managing the memories, tracking and eval, um, how you can block the PII data, uh, going before the LLMs, and then context optimizer to save your token, right?

缺少生产环境基础设施Missing production foundations

那么，事不宜迟，我来带你了解一下这个挑战。嗯，当然，很多智能体通常会在生产环境中失败，因为它们缺少基础控制。嗯，背后的关键原因是，你知道，大多数人都缺少一个网关，无法确保智能体先通过正确的身份验证提供商进行身份验证，与网关通信，然后再被重定向到正确的 MCP 服务器，对吧？其次，MCP 服务器缺少执行查询所需的正确工具。第三，我们有没有足够的跟踪和评估系统，确保发送给智能体的每个查询都能被跟踪，并且评估响应质量，对吧？我们有没有内存管理系统？比如说，你和智能体进行了很长时间或持续多天的交互，它能不能回忆起你过去问过什么，对吧？So, without delay, let's, um, let me walk you through the challenge. Um, of course, like there are many agents, uh, uh, generally get failed in productions because their foundation controls are missing. Um, and the key reason behind is that, um, you know, like, oh, most of the people are missing the gateway to make sure that agent can, you know, communicate to the gateway to get authenticated first with the right auth providers so that they can redirect to the right MCP servers, right? Second, that, um, MCP servers are missing the right tool to execute your query. Third, do we have like enough tracking and evaluation system to make sure that every query when you are sending to the agents, whether they've been tracked or not, and are you evaluating the quality response or not, right? Do we have any in memory management system or not in terms of let's say you are doing the interaction with agent for long or for many days, is it able to recall that what you have been asked in the past, right?

当然，还有其他框架要考虑。比如，你是否制定了节省令牌的措施？是否建立了内部生产机制，以确保任何机密信息都不会进入 LLM、跟踪或追踪系统，或任何内存数据库之类的地方？所以，仍有很多很多悬而未决的问题。为了解决这些问题，我们不会在这个演示中构建什么神奇的智能体，而是会讨论可以用来应对这些问题的不同框架。And then there are other framework, of course, like do you have a context of the measures to save the token, or do you have inner production in place to make sure that your any secrets should not go into LLM or tracking or tracing or any memory DB or so, right? So, a lot of lot of open questions. So, in order to resolve this, we are not going to, you know, like build a magical agent here in this demo. We're going to talk about like different framework that you can build to address this.

智能体及其支撑框架The agent and supporting frameworks

所以，首先是智能体框架。我们选择使用由 LangChain 社区提供的 LangGraph 框架。它可以帮助我们根据你的问题构建特定任务的节点，并且便于接入各种市场插件、技能和提示词，确保智能体能够根据你的查询高效地作出响应。 其次，我们也在构建 MCP 网关，以确保智能体发起调用时，或智能体使用的服务账号发起调用时，都能通过安全认证，然后被重定向到正确的 MCP 服务器。 第三是定制化 MCP，也就是说，我们会构建自己的 MCP 服务器，并配备合适的工具来执行你的查询。 最后，还会对你实际执行的操作进行一些审计，对吧？So, first is the agent framework. This is where like we use the like we choose to use the LangGraph framework, which is provided by LangChain community. This is help us in building the task-specific node based on your questions, and which is easy to plug in any like marketplace plugins or skills and prompts to make sure that agents respond efficiently based on your query. Further, the second part is we are also building the MCP gateway to make sure that whenever agent make the call or any service account from that agent make the call, make sure that they are securely get authenticated to that MCP gateway, and then get redirected to the right MCP servers. Third is the customized MCP, which means that you are building your own MCP servers with the right tool in place to execute your query. And then finally some audit on top of what actually operation you're performing, right?

然后我们会讨论一些额外的框架，比如如何屏蔽 PII 等敏感信息，如何将追踪和评估数据存储到云端，以及如何处理记忆，尤其是每个上下文中的线程和检查点，然后还包括如何……And then we talk about some additional framework around you know like how you can block the PII secrets, how you going to store the tracking eval into the cloud, and how you going to process the memories, especially any threads and checkpoints on every context, and then also how you can

注册、本地开发与铺就的路径Registration, local development and the paved path

整体优化你的上下文，确保运行查询时不会消耗太多 token，好吗？那么，我们现在来谈谈 Agent 基础设施。首先，我们会介绍本地开发。整体思路应该是这样的：每当你构建一个 Agent 时，首先要把它注册到某个系统中。通常你需要说明：“看，这是我的用例，这些是这个特定垂直领域或系统要用到的用例。”然后提交所有这些信息，系统就会为你的 Agent 生成一个 ID，对吧？有了 ID 之后，你就可以使用开发者门户或内部门户，录入所有详细信息，例如：“看，这是我的 Agent 用例，这是我的 Agent ID，我要使用的框架是 LangGraph、LangChain，或者其他框架。”optimize overall your context to make sure that you don't don't burn too many tokens when you run your query. All right? So, we talk about now the agent infrastructure. So, the Firstly, we'll go through the local development. The way the approach should be is that you know like whenever you build an agent, right? First you have to register your agent into some system. You usually tell that hey look, this is my the use case and these are the use case going to use for this particular vertical or system, and you feed all those information and generate a ID for your agent, right? Once you have that, then you can have a dev portal, internal portal where you can ingest all the details saying that hey look, this is my agent use case, this is my agent ID, this is the framework I'm going to use either LangGraph, LangChain or so.

这里是我将使用的语言，我会选择 Python 或 TypeScript，然后你可以选择技能或提示词之类的内容，再提交你的信息，对吧？提交之后，开发者门户就会创建一个代码仓库，其中不仅包含智能体框架，比如你的 LangGraph 智能体框架，还会包含记忆框架、追踪与评估框架，对吧？以及上下文优化框架和 PII 屏蔽框架。之所以这样做，是因为并不是所有开发者都会负责所有这些框架，对吧？他们有时只会创建通常用于 AI 助手工具的框架，或者只专注于一两个框架，对吧？所以，我们只是想确保在公司推广这类框架时，它们能够保持一致，并且可以统一集中管理，这样你也能直观地查看它们的结果，对吧？These are my language I'm going to use either Python or TypeScript, and then you can choose you know like your skills or prompts, and then you submit your information, right? To this. Once you submit this, the dev portal is going to create a repository, which is going to contains not just the agent framework like your LangGraph agent framework, but is also going to contains your memory framework, your tracking eval framework, right? and then your context optimizer framework, and your PII blocking framework as well. So, the reason for that is because not necessarily all your developers are taking care of all this framework, right? They just sometimes you know create the framework what generally like AI assistant tools are generally the right? Or they stick to one or two framework, right? So, we just want to make sure that when you roll out such framework, make sure they are consistent across the company um and they can take care of uh and they can be centralized in one place so you can visualize their results as well, right?

因此，拿到这个代码仓库后，开发者基本上只需要克隆仓库，就能将自己的智能体构建为 Docker 镜像和容器，并通过查询测试智能体的输入和输出。他们还可以在本地内存中查看追踪记录，也可以使用云端的 span 评估器。此外，你也可以将这些追踪记录和评估结果存储在某个数据库中，比如 SQLite 或 BigQuery，以确认智能体在本地运行正常，对吧？所以，So, once you have that repository, so developers basically, you know, clone that repository, they can build their agent as a Docker image and containers and test their agent input and output through the query. And they can also traces on their local memory, maybe cloud span evaluator here. And you can also store this tracking and eval in some of your DB, maybe SQLite or maybe BigQuery um to see to make sure that agents are fully working on your local, right? So,

回顾、容器与 Kubernetes 部署Review, containers and Kubernetes deployment

一旦你的智能体在本地运行起来，你就可以创建一个 PR。等你获得两个 PR 审批后，就可以进入 CI/CD 流水线了，对吧？PR 首先会将你的智能体注册到某个集中式智能体注册系统中，并提供一个指向它的 URL 链接，确保你能看到所有追踪记录。然后，CI/CD 流水线会把你在本地构建的 Docker 容器部署到生产系统中。它会扫描该镜像，确保不存在任何漏洞。如果一切正常并通过安全审批，就可以把镜像推回注册表，对吧？此外，你还会有自己的部署平台或编排平台，用于将该智能体部署到 Kubernetes 集群的指定命名空间中，对吧？once your agents are running in local, um then you're ready to go ahead and generate a PR and then you can once you get your two PR approval, then it's go to the CICD pipeline, right? So, the PR is basically going to firstly is going to uh register your agent into some centralized agent registry system, which provide a length with URL to it to make sure that you see all the traces. And then the CICD pipeline is basically, you know, whatever the local Docker container you built here is going to build that container into production system. Uh it will scan that image, make sure that there's no vulnerability. Um if everything looks great, security approved, you can move that image back into the registry, right? Further, you have your own deployment platform or orchestrator platform which going to deploy that agent into the Kubernetes cluster in a given namespace, right?

所以这里的好处是，这个开发者门户不只是创建代码仓库，对吧？它还会创建一系列其他生产资源，例如工作负载身份，对吧？还有内存数据库、跟踪和评估表，以及智能体密钥管理，对吧？所以，它会根据你提供的名称创建的不只是代码仓库，还有这些不同的资源。这样一来，当这个内部编排器部署智能体时，智能体就已经知道自己应该使用哪个工作负载身份，对吧？相应的绑定应该已经完成。因此，一旦完成 K 到 GSA 的绑定，智能体就会开始与所有这些资源通信。一切就绪后，智能体还会进一步与其他端点服务通信，例如 AI 网关、MCP 服务器、用于追踪的 Langfuse，或者其他一些 API 端点。So, the good part here is that this dev portal is not just creating the repository, um right? It also do create um bunch of other production resources, for example, the workload identity, um right? And then memory DB or tracking and eval tables and then agent secret management, right? Um so, it based on whatever the name you provide, it not just again it like not just repository, it all these different resources as well. So, what happened is when this internal orchestrator when it deploy the agents, the agent already know that hey, what workload identity I supposed to work with, right? That binding should already happen. So, once this K to GSA binding happens, the agent will start communicating to all these resources. Once everything in place, um agent is further going to communicate to the rest of your endpoint services like for example, AI gateway or MCP servers or length fuse for tracing or some other API endpoints.

经过审核的自定义 MCP 服务器Reviewed custom MCP servers

对吧？那么首先，我们也来谈谈 MCP 服务器。我们先聊聊架构，然后再看不同的组件。MCP 服务器是你可以使用的外部资源。不过，如果你在气隙环境中工作，强烈建议你构建自己的 MCP 服务器。我就不讲定义了，大家都知道。这里采用的框架是，你可以借助 FastMCP 服务器包或封装来构建 MCP 服务器，并在 FastMCP 封装中创建一系列不同的工具，从而搭建整体服务器。然后要确保，每当你在 MCP 服务器中定义一个工具，都必须经过内部安全团队的全面审查，以确保只有经过授权的工具才能投入生产。Right? So, now firstly, we talk about the MCP server as well. So, we talk about architecture. Now, we're going to the different component. So, MCP server is something that you can definitely use something which is outside available, but if you're working on air gap environment, it's highly recommended that you build your own MCP servers. So, um yeah, uh and you can use the uh I will not go with the definition. Everyone knows about it. So, the way the framework has been built here is that uh you you can leverage the fast MCP server package um or wrapper to um you know, build that MCP servers and you can create a different set of tools inside that a fast MCP wrapper to construct your um you know, the overall servers. And then you make sure that, you know, like every tool when you define in MCP server, they had to be fully reviewed by your own internal security team to make sure that only authorized tool can be go in production.

这样你就可以设置一些过滤器和限制，并屏蔽那些实际上不需要发送给 LLM、也不该让你获取相关信息的字段，对吧？嗯，第三点是，当然，你需要确保以这样的方式构建 MCP 服务器：当流量从 MCP 客户端经过 MCP 网关到达 MCP 服务器时，无论请求来自用户账户还是服务账户，都必须先通过 MCP 网关的身份验证，这样他们才能访问你的 MCP 工具并获取正确的信息。And that's how you can you know, enforce some filters, limits, and block the fields which is really not required to go to LLM and get you the information which is you're not supposed to get, right? Um and third part is like a of course, you need to make sure that you build your MCP server in a way that whenever the traffic is coming from your MCP client to MCP gateway to MCP servers, either it's coming from user service account, they has to be authenticated to the MCP gateway first and so that they can be access your MCP tool to get the right information.

MCP 网关身份验证与路由MCP gateway authentication and routing

好。现在来说网关部分，也就是网关的工作方式。同样，如果你在气隙环境中工作，可以自行构建内部 MCP 网关。它的工作方式是：你使用类似本地代码辅助工具的云端助手，或者使用你的智能体，对吧？当你首次向智能体提交查询时，它会向 MCP 网关发送一次性身份验证请求。MCP 网关会识别并接收该请求。首先，它会将会话状态保存到 Redis 中，然后将 JWT 访问令牌返回给客户端。客户端收到 JWT 令牌后，就会知道需要调用哪个 MCP 服务器。接着，它会向指定的命名空间提交工具调用请求，例如 Artifactory/MCP 或 GitHub/MCP，并在请求中携带 JWT 身份验证承载令牌。All right. Now, the gateway part. So, the how the gateway works. So, this is again, you can build it your own internal MCP gateway if you are again working on your air gap environment. So, the way it work is you are working on cloud assist like local, you know, code assistance tool or maybe your agent, right? So, the moment you submit your first query to your agent, it will send that one-time authentication request to the MCP gateway, right? MCP gateway recognized and take your request. First, it will save your session state into the Redis, right? And then it will, you know, return back the JWT access token back to the your client, right? Once client will receive that JWT token, it will know that what MCP server needs to call, right? So, it will submit that tool {slash} call request on a given name space {slash} MCP, meaning let's say Artifactory or GitHub {slash} MCP with the auth bearer JWT token into that.

所以，MCP 网关收到请求后，就会识别需要查找的命名空间，也就是相应的 MCP 服务器；然后解开 JWT 令牌，将用户令牌附加到该请求中，并将请求发送到相应的 MCP 服务器。MCP 服务器收到请求后，会向相应服务的端点发起 API 调用，接着交换该 ID 令牌，并根据用户获准查看的范围返回响应，对吧？用户收到令牌后，MCP 服务器基本上会解开其中的所有数据，生成与 MCP 网关兼容的负载，最后将最终负载发送回客户端，以便客户端查看数据点。这说明，我们构建这个 MCP 网关，是为了打造一个集中式的身份验证和路由枢纽，确保任何编码助手工具或智能体都能通过这个 MCP 网关和 MCP 服务器安全地连接到端点。So, once MCP gateway receive that, it will understand that what name space, meaning MCP server needs to look up for, and then it will unwrap that JWT token, and then it will attach that user token in that request and send it to the respective MCP servers. Now, once MCP server receive it, it will make that respective API calls to that endpoint of that, you know, to the service and then it will exchange that ID token and return the response based on whatever scope that user is authorized to see, right? So, once the user receive the token, you know, MCP server basically, you know, unwrap that all the data, make the payload which is compatible for MCP gateway to receive, and then finally the final payload will send back to the client to see the data points. So, what it shows is that, you know, like we build this MCP gateway to make sure that we have a centralized authentication and routing hub that securely connect any coding assistant tool or agents to the endpoint via this MCP gateway and MCP server.

这个 那里This There

日志、追踪信息和网关指标Logs, traces and gateway metrics

不应该在任何端点上提供直接访问。它们必须先通过适当的身份验证，经过 MCP 网关，才能到达 MCP 服务器和端点，好吗？那么接下来是网关监控，对吧？我们再来谈谈架构、MCP 服务器和 MCP 网关。现在要讲的是监控。再说一次，AI 智能体不能在缺乏持续监控、评估和反馈的情况下被信任，对吧？这就是我们要构建 MCP 网关监控的原因。监控通常是这样工作的：你向智能体提交任何调用时，它都会把所有运行日志发送到日志存储中。你可以使用 ELK、Loki 或 CloudWatch，对吧？通常，这些日志包含诸如“显示我最近 10 分钟调用中所有 401 错误”之类的信息，对吧？should not be direct access provided um on any endpoints. They has to go through the MCP gateway through proper authentications uh before they're reaching to the MCP server and the endpoints. All right? So, now the part is the gateway monitoring, right? We talk about again, we talk about architecture, we talk about MCP server, MCP gateway. Now, it's a monitoring. Again, the AI agent cannot be trusted without continuous monitoring, evaluation, and feedback, right? This is why we are building the MCP gateway uh monitoring. So, the way the monitoring work is usually like the moment you submit any calls to your agent, uh it will send all the operational log into the log store. You can use ELK, Loki, or CloudWatch also, right? And generally, this contains the information like show me like all last 401s in my last 10 minutes call, right?

那么，当令牌刷新失败时，OAuth 做了什么，对吧？这类请求可以发送到日志存储中。现在，还有一些跟踪和指标数据，也就是实时数据，也会发送到 OpenTelemetry。OpenTelemetry 同样是一个开源解决方案，它会收集所有跟踪和指标数据，处理这些数据，然后将其转发到后端系统，对吧？在这个例子中，跟踪数据会发送到 Tempo。这也是一个开源解决方案，用来存储所有跟踪数据。这里的跟踪数据指的是一些信息，比如这次请求中哪个工具比较慢、这次调用是来自代理还是 IDE、这次失败调用的具体错误是什么，等等，对吧？这类数据都会存储到 Tempo 中。And what did OAuth do when the token refresh failed, right? So, those kind of requests can goes to the log store. Now, there are some, you know, traces and metrics data um like real-time data that goes to the OpenTelemetry as well. Um OpenTelemetry again is open source solutions, which going to collect all your traces um and metrics and it's going to process the data and redirect to the back-end system, right? And in this case, the traces will go to the Tempo. This is again an open source solutions, which going to store all your traces in terms of like traces in the sense like, you know, like information around like which tool was slow on this request, whether this call is coming from agent or IDE, uh what was the exact error um on this failed call or so, right? So, these kind of data are going to store into the Tempo.

我的错。好。然后说到指标，Prometheus 主要关注的是相关信息，比如某个工具的 P95 延迟是多少，对吧？某个时间段内我的身份验证拒绝率是否突然飙升，对吧？这类指标会存储在 Prometheus 中。等你拿到所有这些数据点，比如 traces 和指标之后，所有这些聚合数据实际上都会保存到图表中，以便进行可视化。My bad. Okay. And then the metrics is about the Prometheus is all about the information about like what are P95 latencies for a given tool, right? Whether my auth rejection is spiking on a particular time or not, right? So, those kind of metrics is going to store in Prometheus. So, once you get all these data point like traces and metrics, this entire aggregate data actually save into the graph now for your visualizations.

LangGraph 策略与人工审批LangGraph policies and human approval

好。MCP 服务器网关和监控部分已经介绍完了。现在我们来谈谈框架，也就是智能体框架。大多数情况下，我们使用 LangGraph。LangGraph 由 LangChain 社区提供，采用基于图的架构，支持循环、分支和状态管理，适用于复杂的多智能体 AI 工作流。它的工作方式是：当你提交第一个查询时，我们会先解读你的问题，判断问题的类型以及你想要什么样的响应，然后据此制定计划。接着，系统会将请求发送到分类环节，判断它属于只读操作、修改类变更，还是删除操作。All right. So, the MCP server gateway and monitoring is covered. Now, we're going to talk about the framework. So, the agent framework, right? So, we use LangGraph for most of the cases and is basically LangGraph again it's a provided by LangChain community. It's provide a graph based architecture that support loops, branching, state management for complex multi-agent AI workflows, right? And the way it work is like the moment you submit your first query, we do interpret you know, like your question first, like what type of questions it is and what response is looking for and based on that it's create a plan, right? Then it will send for classify classify area where it will determine that hey, whether it is just like a read operations or whether it is just like a you know, modification level changes coming up or whether you really want to delete something, right?

或者一些敏感操作。它会根据这一点调整整体结构，然后据此制定计划。下一步的计划本身就是一个子图，你会在其中执行大部分操作。也就是说，在计划中，一旦确定要执行什么操作，就会进入相应的步骤，比如“获取证据”环节：它会针对你在这里提出的问题，在完成解析和分类后，调用 MCP 网关，确保通过身份验证，然后使用正确的工具调用 MCP 服务器并获取数据，接着生成相应的证据，对吧？验证完所有发现后，它就会检查策略，确认这是否为只读操作，然后继续执行，并将所有记录点保存到我们的审计日志中。Or some sensitive operations. Based on that it adjust the overall structure and then plan accordingly. The next step is plan is actually itself is a sub graph where you do the major of your operations. Meaning that in the plan like once you know what operation you're performing, it goes to the like let's say in a fetch evidence section where it actually call that whatever the question you ask here, after all interpret and classify, um it actually go ahead and make the call to the MCP gateway, uh make sure they get authenticated, um and then it make the call to the MCP server using the right tool, um and get the data, and then generate that evidence, right? Once you validate all your finding, then it will check the policies. Uh it will check whether it is a read-only, then go ahead and execute, and I will save all the record points into our audit logs.

但如果是写入和敏感操作，肯定会经过 HITL 流程，也就是等待审批。如果获批，就继续执行；如果被拒绝，无论哪种情况，都会记录到审计日志中。对吧？那么，如果把这些都结合起来，比如 MCP 网关、MCP 服务器，还有 Agent，整体看起来就是这样。最终，无论你构建的是什么 Agent，也无论你提交的是什么调用，都会先经过 MCP 网关进行身份验证，然后被转发到相应的 MCP 服务器，以便在指定工具上执行查询，调用相应端点的 API 并获取响应。对吧？好。如果从基础设施的角度来看，整体架构就是这样：Agent 与 MCP 网关通信，再转到 MCP 服务器，最后到达端点。But if it is a right and sensitive operations, definitely it will go through the HITL process where you wait for an approval. If it is approved, you go ahead and execute. If it is get rejected, in either case it's going to save into the audit log. Right? So, if we combine all this together, like MCP gateway, MCP server, um and the agent, right? This is how things looks like. So, ultimately, whatever the agent you're building, whatever the call uh you're submitting, it will go through the MCP gateway for authentications, and then it will redirect to the right MCP servers to execute uh you know, your query on a given tool, uh so that it make the API call to that endpoint and get the response. Right? All right. If it If we think from the infrastructure perspective, this is how it looks like. So, agent is communicating to the MCP gateway, goes to the MCP server, and to the endpoint.

好吧？那么，现在，呃，我们已经谈了很多关于 MCP 网关、MCP 服务器、嗯，监控以及代理框架的内容。现在我们来谈……All right? So, now the uh we talked a lot about MCP gateway, MCP servers, um the monitoring, um and the agent framework. Now we talk

线程、运行与记忆检查点Threads, runs and memory checkpoints

在构建端到端生态系统时，我们还需要关注大约三到四个额外的框架，对吧？其中一个是智能体内存管理。它基本上是一个能够随时间保存和恢复工作流状态的过程，对吧？这样可以确保每次查询运行都具有持久性，并能无缝恢复。它的工作方式是：每当你向智能体提交查询时，系统实际上都会创建一个线程，对吧？在这个线程中，当你提出第一个问题时，系统会为它创建一个运行 ID。之后，当你提出一些更深入的问题时，系统会在 LangGraph 中进入特定节点并执行该查询。然后，它会在该节点保存相应的检查点，对吧？about uh three and four additional framework that we should have to take care of when building the end-to-end ecosystem, right? One is them is agent memory management. Again, it's a uh you know, it's a it's refer It's basically a process where you can save and restore the state of your workflow over the time, right? Ensuring that it's provide you a durability and seamless recovery across across your every query run. So, the way it work is like whenever you submit any query to your agent, it actually generate a thread, right? On that given thread, when you ask your first question, it will going to create a run ID for that. So, the moment you ask some deeper questions, it will go on a particular node in that langraph and execute that query. And on that node, it going to save that respective checkpoint here, right?

现在，如果你在同一个线程中提出后续问题，就会生成另一个 run ID。而且，根据你的后续问题，它会转到特定节点，并在那里保存检查点，对吧？对于本地开发，你可以使用 SQLite 数据库；但在生产环境中，如果是在 GCP 上，可以使用 Cloud Spanner。相应地，在 Azure 和 AWS 上也可以使用各自的数据库，对吧？接下来，我来快速演示一下这里的记忆功能。比如说，我在本地构建了这个智能体，然后尝试在 SQLite 数据库上运行这个查询，看看里面有没有数据。最开始当然没有任何数据。现在我提交查询，你会看到我收到了一条记录，上面写着：“嘿，这是我的线程 ID。”Now, if you ask a follow-up questions on the same thread, it going to create another run ID. And again, depend on your follow-up question, it go to a specific node and save the checkpoint over there, right? And all this data for your local development, you can use the SQLite SQLite DB, but in production, unless in GCP, you can use Cloud Spanner. And in respective databases, you can use in Azure Azure and AWS as well, right? So, just walk you through the quick demo here on the memory. So, for example, I have, you know, like just build this agent locally and then, you know, trying to run this query on SQLite DB to see if there is any data or not. So, I'm just, you know, initially I don't have any data. Now, I submit my query, you will see that I received the one record, which is saying that, "Hey, this is my thread ID.

这是我被问到的问题，以及查询执行的确切时间。现在，如果我运行下一个查询，看看“嘿，我在某个给定线程中生成了多少条消息？”，就可以通过运行后续查询来获取这些信息，对吧？如果我想查看查询的所有元数据级信息，就可以再次运行查询，并查看详细说明，例如调用了多少个工具、我的线程 ID 是什么、运行 ID 是什么，等等。如果我还想知道运行这个查询时执行了哪些工具，也可以查看相应 MCP 服务器上的具体工具名称。除此之外，我还想了解 LangGraph 框架中的进度是如何推进的，这也可以通过在 memory DB 上运行这类查询来查看。This is the question I have been asked." And exact time, you know, when this query was executed. Now, if I run the next query to see that, "Hey, how many messages I've been produced on a given thread?" You can get this information by running the further queries, right? Now, if I would like to run to see that, you know, like all the metadata level information on my query, I can run my query again and And can see the detail description around, you know, like how many tool I get invoked, what is my thread ID, what is my run ID, um you know, like and all those informations. Now, if I want to also see that, hey, what tool have been executed when I run this query, um you can go ahead and see the exact tool name as well on that given MCP servers. Now, further I will also want to see that how the progression was happened on my lang graph framework, that also you can see by um running this type of query on your memory DB.

你可以在这里看到，当那个查询实际运行时，会调用不同类型的节点，嗯，你知道的。You can see that here like different kind of, you know, nodes been called when actually, um you know, that query has been run.

代理跟踪与评估记录Agent tracking and evaluation records

好吧？现在说一下跟踪和评估。为了确保这里不会产生混淆：就像我们之前谈到的 MCP 网关监控一样，这是代理级别的跟踪。我们这样做，是因为我们确实想跟踪你向代理提交了什么查询、收到的响应是什么、使用了什么模型、token 成本是多少、消耗了多少 token，以及你的评估标准是什么样的。你的响应质量如何，对吧？所以，你需要从第一天起就存储这些数据。想象一下，如果你有一千多个代理，每个代理都会有自己的代理跟踪与评估项目和表，对吧？因此，如果合并起来，比如你有 100 个代理，就会有 100 个不同的项目，每个项目都有各自的跟踪与评估表。All right? Now, the tracking and eval. So, just to make sure that um there is no confusion here like the we talked about earlier the MCP gateway monitoring, this is the agent level tracking. So, we do this because we really want to track that what query you have been submitting to the agent um and what the response is coming, what alarm model you have been used, what's the cost of your token, um how many token you have been burned, and what's your, you know, evaluation criteria looks like. What is the quality of your response or not, right? So, you need to store that data from the day one. And imagine that if you have a thousand plus agent, every agents will have their own um you know, um agent tracking and eval um project and the tables, right? So, if you combine, let's say if you have 100 agents, you will have 100 different projects and each project has their own respective tracking and eval tables.

现在、明天，或者六个月之后，如果有人问你 Agent 的表现如何、结果怎么样、质量如何、消耗了多少 token，你就可以汇总所有这些数据，并围绕它创建高管仪表板，看看构建这么多 Agent 的 ROI 是多少，对吧？那么，我来带你看一下这个跟踪和评估的演示。比如说，如果我看这里这一项，首先你可以看到这里列出了六张表。一张是作业跟踪表，其中包含了大部分这类信息，比如你的查询是什么、响应是什么，等等。此外还有其他一些表，比如与质量相关的表、工具调用表、节点级执行表，等等。Now, tomorrow, 6 months down the line, if somebody ask you that how your agent has been performed, what was the response, what's was the quality, how much token you have been burned, you can aggregate all this data and you can create executive dashboards around it to see um with to see the what is the ROI in building so many agents, right? So, let me walk you through the the tracking and eval um demo here. So, um here like if I look at uh this one, so first like, you know, like if you see that there are six tables are listed here. One is a job tracking table, um and which is contains most of this like information around what query like what is your query, what's the response and everything. And there are a bunch of other tables as well like your quality related and your tool invocations and then node level executions and all that.

所以，在这个例子中，我要在我的 agent 上提交第一个查询，你会看到，一开始不会有任何数据，因为这需要一些时间。所以我会再刷新一次。这次刷新后，我会看到，对于某个查询，我能获取到请求 ID、会话 ID、agent 名称、提出的问题、该问题的输出、是否已完成、耗时、所使用的模型，以及一些中间层级的信息，对吧？此外，如果我查看质量指标，会发现每次运行有八项质量检查，对吧？你可以看到，每一行对应一项检查，比如成功率、延迟、成本、工具准确性、安全性等。So, in this case I'm going to submit my first query on my agent um and you will see that um there will not be uh any data in the first place uh because it takes some time. So, I will refresh this again. And this time I will see that when I refresh, I do see that for a given query I get my request ID, my session ID, my agent name, um you know, what question I've been asked, um and what was the output um of that uh question, uh whether it has completed or not, and then what is the time it took, what is the model I've been used, um and then some of the middle level uh middle level informations, right? Further, if I look for the quality, again there is a eight quality checks per run, right? And you can see that this one row each for, you know, like success, like latency, cost, tool accuracy, safety, etc.

就像这些内容都存储在这里了，对吧？接下来是评估分数。这部分同样会显示为一行。它会是你这次运行的记分板，分数基本在 0 到 1 之间，也就是说，各项质量维度的分数都会显示在同一行里，对吧？接下来是质量指标。这部分会深入分析这一次运行。基本上，它会用一个嵌套字段来展示响应质量、安全性、延迟、成本以及工具统计信息，对吧？接下来是工具调用，也就是每次工具调用、运行的是哪个 MCP 工具、耗时多久，以及是否成功，对吧？然后是工具代理的执行轨迹，讲述这次运行的逐步过程，按顺序展示每次 LLM 调用和图状态，比如规划、总结等。好，那么现在……Like all this has been stored here, right? Now, the eval scores. This is again this going to be a one row. This is going to be a scoreboard for your given run, which is basically varies between zero to one, which is scores uh you know, like all the quality dimension in a single row, right? Now, the quality metrics. So, again it's a deep dive on that particular one run. Basically a nested field for the response quality, safety, latency, cost, uh and the tool stat in one white row, right? Now, the tool invocations, so every tool call, which MCP tool ran, how long it took, and whether it succeeded or not. Right? And now the tool agent trajectory, which is talk about step-by-step story of the run, each LLM call, and the graph state in order, like plan, summarize, etc. All right. So, now

过滤敏感输入Filtering sensitive inputs

接下来是 Agent PIR 处理。这非常非常重要，因为在使用智能体或这类功能时，有时可能会将一些机密信息作为查询的一部分输入，对吧？比如说，你上传了一份包含敏感信息的文档，或者有时在输入内容时，可能会带上 API 密钥之类的信息。因此，我们有一个框架，用于定义不同的数据类型及其严重级别。如果严重级别超过 90% 左右，我们就会立即拦截并编辑这些机密信息。这样可以确保这些机密信息永远不会进入 LLM 模型，对吧？它们既不会进入你的内存数据库，也不会进入跟踪和评估表或追踪系统。这里的目标就是在这些敏感数据刚被输入时，立即将其拦截。uh the next is the agent PIR treatment. This is very, very important because many times it happens that when you're working in agents or uh this thing, um sometimes there's a possibility that there are some secrets being ingested as a part of your query, right? Let's say you're uploading some doc which is has some sensitive informations um and sometimes while typing you may have some API keys or so. So, we have a framework which define that different sort of data types and their severity level. If your severity is above 90% or so, uh we will block and redact that uh secrets uh immediately. So, that uh make sure that that secret is never goes to LLM models, right? Neither it will goes to your memory DB, or nor it will go to tracking and eval tables, or tracing uh systems as well. So, the goal here is to just block this sensitive data as it is the moment it get ingested, right?

你打字的那一刻，那一刻。Moment moment you type.

上下文优化与令牌精简演示Context optimization and token reduction demos

好。嗯，最后一个框架讲的是上下文优化器。这是我最喜欢的领域之一。你知道，每次提交内容都非常重要，对吧？如果一次提交太多查询，有时你会上传大量数据点和一堆文件系统之类的内容，对吧？这时候，确保每次查询都节省 token 就非常重要。因此，我们构建了一个上下文优化器框架，你可以在查询发送给 LLM 之前对其进行调整和精简。在精简过程中，我们会根据 token 限制筛选负载。然后我们会重新排序：“在你提交这么庞大的上下文时，哪些信息最相关、最值得重点关注？”接着，我们会去重，移除所有重复项，并创建一组独特且经过精心筛选的数据——也就是提供给 LLM 的核心数据。All right. Um the last framework is all about now is the context optimizer. This is one of my favorite area where like um you know, uh whenever you submit, it's very important, right? Whenever you submit too many queries, sometimes you upload a lot of the um data points and you upload bunch of file systems or so, right? It's very important that time is um you know, like you make sure that you save your token in every query. So, we build in context optimizer framework where you actually shape your query and then you prune it before it go to LLM. In the pruning system, uh we do select the payload based on the token limit. Then we re-rank that, "Hey, which is the most relevant relevant information that we should target for when you submit this huge giant, you know, context?" And then we dedupe that, uh remove all the duplicates, and create a very unique set of uh very curated data like uh I mean the main data for the LLM.

然后，我们会在将这段大型负载发送给 LLM 之前，先对它进行压缩。接下来看看这个演示，很快就好。比如说，我提交第一个查询：“嘿，帮我找出 JFrog Artifactory 中的所有制品。”你看，在我提交任何查询之前，这里没有数据。提交第一个查询后，如果我刷新一下查询记录，就能看到新增了一条记录。可以看到，这个问题使用了 24,000 个 token。我稍微返回一下，给你看看。你可以看到，这个问题使用了 24,000 个 token。现在，我在后台启用上下文优化器，然后再次运行完全相同的查询。这次你会看到，对于相同的查询，刷新这次运行记录后，可以看到只用了 2,400 个 token。And then we finally compress this large payload before it goes to the LLM. So, let's see this uh demo here. Uh this will be quick. So, for example, let's say I will submit my first query, "Hey, find me all the artifacts in my JFrog Artifactory." Um you see that first when before even I not submitted any query yet, there is no data here. So, the moment I submit this first query, and if I refresh my query here, um and then you can see that the record has been added. So, you can see there is a 24,000 token has been used. I will go back a little bit um just to show the thing. So, you can see this 24,000 token has been used for given questions. Now, if I ask uh now I enable the context optimizer behind the scene and I'm rerun the exact same query again. So, this time you will see that uh you know, for the same query, if I refresh this run, and you will see that uh this time it used just 2,400 token.

对吧？对于完全相同的查询，节省的比例几乎达到 90%。我也可以在 Slack 机器人上重新运行同一个演示：我会问 Slack 机器人：“嘿，告诉我 Artifactory 中所有存储库里的全部 Docker 镜像。”运行这个查询时，没有数据。现在，我要提交查询。等这个查询运行完后，嗯，如果我在这里再次运行查询，你会看到记录被创建了。这个特定查询再次用了大约 24,000 个 token。接下来，我要在后台启用优化器，然后在 Slack 机器人上重新运行同一个查询。嗯，如果我再次运行这个查询，它会显示只用了大约 1,500 个 token，节省的比例同样接近 90%。在大多数情况下，我们能节省大约 70% 到 78% 的 token。想象一下，如果你有成千上万个智能体，每个每天都要运行很多次查询，你就能想象通过这个框架可以节省多少 token。Right? For exact same query. So, which is almost like 90% saving. I can rerun the same demo on the Slack bot as well where I'm asking Slack bot, "Hey, tell me all my Docker images across all the repositories in Artifactory." When I run this query, there is no data. Now, I'm going to submit my query. So, once this query will run, um and if I run my query again here, and you will see the record get created. It's again used like 24,000 for this this particular query. Now, what I will do is I will turn on my optimizer behind the scene and I rerun the same query in the Slack bot. Um and then if I run this query again, um it will show like it exactly used 1,500 almost token, which is again like 90% saving. In most of the cases, we get like 70 to 78% token saves. So, imagine that you have thousands of um you know, like um um if you have thousands of like agents, which is running like you know, many queries per day, you imagine how much token you can save with this framework.

好的。那么，现在的……All right. So, now the

围绕每个智能体构建框架Build the framework around every agent

这里的关键要点是，对我们来说，要确保构建的是一个企业级框架，而不只是一个智能体，对吧？要确保底层具备完整的框架。不要急着快速构建一个智能体，然后就部署上线。要确保整个过程都经过完整的框架，这样你就不会到后来才意识到：“我一开始就应该把这个框架搭好。”对吧？要从一开始就做到这一点。 第二，要确保有一套完善的网关系统。确保每个工具调用请求，无论是从本地发起，还是通过智能体发起，都必须经过某种集中式 MCP 网关。确保请求在到达 MCP 服务器之前经过身份验证。然后还要确保 MCP 服务器能够控制你应该使用哪个工具来执行特定查询，对吧？key takeaway here is uh you know, like um for us is like make sure that we build an enterprise framework, um not just an agent, right? Which make sure that you have all the framework underneath. Don't try to be in rush to building some agent quickly and deploy ship it. Make sure that it has to go through all the framework all together, so that you should not realize a letter state that, you know, I should have this framework in place. Right? In in the very first place. Second is like your Make sure that you have a proper gateway system. Make sure that every tool call request either you are doing this from your local or through agent, it has to go through some sort of centralized MCP gateway. Uh make sure it authenticated before it reaching to the MCP servers. And then make sure that your MCP servers take a control on what tool um you know, uh you should use uh to execute a particular query, right?

并确保针对这些工具执行设置了监控和完整的审计。当然，你知道，对于每一项操作、每一项高风险事项，都要始终让人工参与其中。确保具备可观测性，也要做好审计。最后但同样重要的是，确保始终配备上下文优化器，以便证明你的智能体所花费的成本是合理的，对吧？我只是想分享一下，我最近在亚马逊发布了我的 MCP 书籍。欢迎下载，Kindle 版是免费的，也有纸质版。好了，非常感谢大家观看这段视频。如果你有任何问题，欢迎通过 LinkedIn 或 Twitter 联系我，我很乐意解答。再次感谢。And make sure that you have monitoring in place and complete auditing in place for that tool executions. And of course, you should um you know, keep the always human in the loop process for every um every action items, every high-risk items. Um make sure that there is a observability um and you know, um auditing has to be in place. And last but not least, make sure that context optimizer should always be there to justify your cost on your agents, right? So, just want to share is that I released my MCP book recently on Amazon. So, feel free to download. It's free in Kindle. And it's hard copy is also available. All right. Thank you so much for watching this video. And if you do have any questions, feel free to, you know, like reach out to me over LinkedIn or Twitter. I'm happy to answer. Thank you again.