为 AI 智能体建立信任：基于 Amazon Bedrock AgentCore 的生产级评估框架 | Amazon Web Services

北极星The North Star

每个竞相部署 AI 智能体的组织都面临同样严峻的考验：一个演示效果出色的智能体，在投入生产后可能悄然退化；等到人类察觉时，信任已经破裂。传统软件出错时往往会明显报错，而智能体的失效却悄无声息。它们给出的答案看起来信心十足、读起来流畅，却是错的。当这个答案影响到真实的业务决策时，代价并不是某个功能无法使用，而是与用户之间的关系遭到破坏。Every organization racing to deploy AI agents faces the same reckoning: an agent that demos beautifully can quietly degrade in production, and by the time a human notices, the trust is already broken. Where traditional software fails loudly, agents fail softly. They return an answer that looks confident, reads fluently, and is wrong. When that answer drives a real business decision, the cost is not a broken feature. It’s a broken relationship with the user.

这正是质量问题的核心。因此，挑战不在于构建能成功运行一次的智能体，而在于证明它们能够持续稳定地运行，并在规模化场景下覆盖每个子智能体、每项工作流和每次发布。在智能体时代胜出的团队，将把质量视为智能体自身的操作系统，而不是最后一道 QA 环节：持续衡量、自动保障，并在每次交互中不断改进。在这篇文章中，我们将展示如何使用 Amazon Bedrock AgentCore，为可投入生产的智能体解决方案构建持续质量闭环，并实现规模化。That’s the quality problem at its core. The challenge is therefore not building agents that work once, but proving they keep working, at scale, across every sub-agent, every workflow, and every release. The teams that win the agent era will be the ones who treat quality not as a final QA step but as the operating system of the agent itself: continuously measured, automatically defended, and improved with every interaction. In this post, we show how we built a continuous quality loop for production-ready agentic solutions at scale using Amazon Bedrock AgentCore.

业务问题Business problem

该解决方案使用 Amazon Bedrock AgentCore 和其他 AWS 服务供内部使用，让销售人员和销售领导者能够用自然语言在 Amazon Quick 中提出业务问题。在这一统一体验的背后，多个专用子代理协同工作：Performance Insights Agent 负责按销售区域划分的营收与绩效洞察，Customer Insights Agent 负责客户情报，Data Insights Agent 负责针对基础数据集进行文本转 SQL 查询，此外还可直接访问仪表板。答案会根据每位用户的销售区域和角色级安全权限进行限定。销售团队会利用这些答案辅助制定客户策略、为客户会议做准备，因此数据必须准确。危害最大的故障不是系统崩溃，而是给出流畅、自信却悄然出错的答案。上线前，在评估 Customer Insights Agent 时，我们就发现了这种情况：The solution uses Amazon Bedrock AgentCore and other AWS services for internal use, so sellers and sales leaders can ask business questions in plain English from Amazon Quick. Behind that single experience, several specialized sub-agents work together: the Performance Insights Agent for territory-scoped revenue and performance, the Customer Insights Agent for customer intelligence, the Data Insights Agent for text-to-SQL over foundational datasets, and direct dashboard access. Answers are scoped to each user’s territory and role-level security. Field teams use these answers to assist in planning account strategy, prepare for customer meetings, so the numbers must be right. The most damaging failure is not a crash. It’s a fluent, confident answer that is quietly wrong. Before launch, when we evaluated the Customer Insights Agent, we found exactly that:

• 它将某客户三个月内在一项服务上产生的 410.08 美元收入报告为“410.08 百万美元”。
  It reported $410.08 of a customer’s three-month revenue on a service as “$410.08 Million.”
• 它报告的销售收入标注为“未计调整项的销售收入”。
  It reported Sales Revenue labeled as Sales Revenue excluding Adjustments.
• 它给出了一个数据中根本不存在的年度经常性收入估算值。
  It offered an Annual Recurring Revenue estimate that appeared nowhere in the data.
• 它把自己的计算结果当作有来源的事实来呈现。
  It presented its own calculations as sourced facts.

在不同的运行过程中，监控始终显示绿色。它告诉我们系统已作出响应，却无法说明答案是否正确。Through the different runs, monitoring stayed green. It told us the system had responded, not whether the answer was right.

如果这些错误未经发现就发布，一线团队就得为此付出代价。销售人员如果反复报出虚高数字，可能会失去客户的信任。这正是我们在上线前发现问题的原因。风险并不会在上线时止步：每增加一个智能体，或对数据、提示词、代码或模型进行任何更改，风险都会随之增大。即使没有发布任何版本，风险也会增大，因为上游流水线每周都会刷新数据。协调发布还会带来另一类风险：当数据变更、工具变更和配套的提示词更新原本应该一同发布，却分开落地时，在所有变更同步之前，系统就会处于不匹配状态。在一次预生产测试中，仅这一不匹配窗口就使通过率从 78% 降至 38%。为了在每次发布前以及生产环境中持续捕捉此类故障，我们需要一个框架，能够判断回答是否有依据且正确，并将每次故障追溯到数据、提示词或智能体代码中的根本原因。Had these errors shipped uncaught, the field would have paid for them. A seller who repeats an inflated figure can lose credibility with the customer. This is exactly why we caught it before launch. The risk doesn’t end at launch: it grows with every new agent and every change to data, prompts, code, or models. It also grows without any release, because upstream pipelines refresh the data every week. Coordinated releases introduce a related risk: when a data change, a tool change, and a paired prompt update are meant to ship together but land apart, the system runs mismatched until all changes catch up. In one pre-production test, that mismatch window by itself dropped the pass rate from 78 percent to 38 percent. To catch failures like these, before every release and continuously in production, we need a framework that can show whether answers are grounded and correct, and trace each failure back to its root cause in the data, the prompt, or the agent’s code.

在 AWS，销售、营销和全球服务（SMGS）运营团队开发了一套面向全球 AWS 一线销售团队的生产级多智能体系统。它服务于两类不同的受众。对于一线销售团队，它以按需提供的对话式智能，取代了数小时的手动数据分析，帮助销售人员了解业务健康状况、发现风险与机遇，并更快采取行动。对于管理层，它为包括每周业务回顾（WBR）在内的业务管理例会提供支持，无需手动准备报告，即可呈现整个业务组合中值得关注的信号。同一套智能体系统既服务于身处一线的销售人员，也服务于大规模审视业绩的领导者。At AWS, the Sales, Marketing, and Global Services (SMGS) Operations team developed a production multi-agent system serving AWS Field Sales worldwide. It serves two distinct audiences. For field sales teams, it replaces hours of manual data analysis with on-demand, conversational intelligence, helping sellers understand the health of their business, surface risks and opportunities, and take action faster. For leadership, it powers business management cadences including Weekly Business Reviews (WBRs), surfacing the signals that matter across the portfolio without requiring manual report preparation. The same agentic system serves both the seller on the front line and the leader reviewing performance at scale.

智能体系统负责协调一个不断扩大的专业子智能体组合，每个子智能体都专注于不同的能力领域。其中包括：绩效洞察智能体根据复杂的销售数据生成自然语言形式的商业智能分析，将原始指标转化为可供销售人员采取行动的洞察。客户洞察智能体综合产品使用情况、支持历史、合同状态和互动数据等信号，为每位客户构建 360 度全景视图，帮助销售人员了解客户目前所处的旅程阶段以及接下来需要什么。数据洞察智能体是一种文本转 SQL 智能体，客户团队可以用通俗的英语查询结构化分析数据，无需编写任何 SQL 代码。本文其余部分提到的就是这三个智能体。The agentic system orchestrates a growing portfolio of specialized sub-agents, each owning a distinct capability domain. Among them: the Performance Insights Agent generates natural-language business intelligence narratives from complex sales data, turning raw metrics into actionable seller insights. The Customer Insights Agent assembles a 360-degree view of each customer by synthesizing signals across product usage, support history, contract status, and engagement data, helping sellers understand where a customer is in their journey and what they need next. The Data Insights Agent is a text-to-SQL agent that account teams use to query structured analytics data in plain English without writing a single line of SQL. These three agents are the ones referenced throughout the rest of this post.

要在大规模场景下可靠地协调这些智能体，有一点变得十分明确：智能体质量并非可以留待日后管理的运营问题，而是开展运营的先决条件。单个性能下降的子智能体，就可能在不知不觉中破坏下游的每个响应，而传统监控无法发现这一问题。正因如此，全面的全生命周期评估框架并非可有可无，而是必不可少。Amazon Bedrock AgentCore 的一项能力 AgentCore Evaluations 是支撑这一框架运行的纽带，此外还包括 Amazon Bedrock Guardrails（安全防护）、Amazon CloudWatch（指标、警报、跟踪）、Amazon Simple Storage Service（Amazon S3）和 Amazon Redshift（评估数据持久化）、Amazon Quick（仪表板）以及 AWS Lambda（自定义评估器）。Coordinating these agents reliably at scale made one thing clear: agentic quality wasn’t an operational concern to manage later. It was the prerequisite for operating. A single degraded sub-agent can silently corrupt every response downstream, and traditional monitoring won’t catch it. That reality made a full lifecycle evaluation framework not optional but necessary. AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, is the connective tissue powering it, alongside Amazon Bedrock Guardrails (safety), Amazon CloudWatch (metrics, alarms, tracing), Amazon Simple Storage Service (Amazon S3) and Amazon Redshift (evaluation persistence), Amazon Quick (dashboards), and AWS Lambda (custom evaluators).

在这篇文章中，我们将分享如何应对以企业级规模运行 AI 智能体时最棘手的运营挑战，以及 Amazon Bedrock AgentCore Evaluations 如何成为保障智能体质量的核心。具体来说，我们将介绍：In this post, we share how we tackled the hardest operational challenges in running AI agents at enterprise scale, and how Amazon Bedrock AgentCore Evaluations became the backbone of agent quality. Specifically, we cover:

• 智能体遵循不同的规则——为什么智能体需要一套不同的运维规范，以及为什么传统监控（延迟、错误率、正常运行时间）会让你看不见最关键的故障。
  Agents play by different rules – Why agents demand a different operational discipline, and why traditional monitoring (latency, error rates, uptime) leaves you blind to the failures that matter most.
• 贯穿全程的路线图——我们基于 AgentCore Evaluations 构建的九阶段评估框架，涵盖智能体开发生命周期的每个阶段，从首次提交到生产环境。
  A map for the whole journey – The nine-stage evaluation framework we built on AgentCore Evaluations, covering every phase of the agent development lifecycle from first commit to production.
• 从理论到实践——具体模式（持续集成和持续交付（CI/CD）回归门禁、金丝雀评分、漂移检测和自动反馈循环），以及 AgentCore Evaluations 如何为每一种模式提供支持。
  From theory to practice – Concrete patterns (continuous integration and continuous delivery (CI/CD) regression gates, canary scoring, drift detection, and automated feedback loops) and exactly how AgentCore Evaluations powers each one.
• 真正重要的指标——衡量其对我们整个智能体组合的部署信心、回归问题预防和迭代速度所产生的影响。
  Metrics that actually matter – The measurable impact on deployment confidence, regression prevention, and iteration speed across our agent portfolio.

为什么智能体需要结构化的评估框架Why agents need a structured evaluation framework

传统软件具有确定性：相同的输入会产生相同的输出。机器学习（ML）模型带来了概率性输出，MLOps 随之兴起，用于管理模型的训练、版本控制、监控和再训练。AI 智能体又增加了第三层运维复杂性：它们会自主进行多步骤推理，此时问题从“是否有东西出故障了？”转变为“智能体在整个推理链中是否做出了合理决策？”Traditional software is deterministic: the same input produces the same output. Machine learning (ML) models introduced probabilistic outputs, and MLOps emerged to manage their training, versioning, monitoring, and re-training. AI agents add a third layer of operational complexity: autonomous, multi-step reasoning, where the question shifts from “did something break?” to “is the agent making sound decisions across its entire reasoning chain?”

智能体继承了前两代技术中的棘手问题，还引入了一些新问题。它们具有非确定性，会针对相同的输入遵循不同的推理路径。它们会在路由、工具选择和综合生成等环节做出一连串决策，其中任何一步都可能悄无声息地失败，并破坏后续所有环节。最关键的是，它们会悄然退化：延迟依然很低，错误率依然平稳，但智能体响应的质量却在下降，因为它仍在响应，只是答错了。Agents inherit the hard problems of both predecessors, and introduce some new ones. They’re non-deterministic, following different reasoning paths for identical inputs. They make cascading decisions across routing, tool selection, and synthesis, where a single step can fail quietly and corrupt everything downstream. Most critically, they degrade silently: latency stays low, error rates stay flat, yet the quality of agent response erodes, because the agent is still responding. It is just responding wrong.

Agent 评估实践正是为了发现这类问题。它涵盖五个维度：安全性（是否在既定边界内运行？）、质量（Agent 是否能生成正确、连贯且有依据的回答？）、性能（速度和可靠性是否足够？）、成本（能否在大规模运行时保持高效？）以及运营控制（能否快速发现、诊断并解决问题？）。成熟的评估实践会在 Agent 的整个生命周期中持续关注这五个维度。Agent evaluation is the practice of catching exactly that. It spans five dimensions: safety (is it behaving within defined boundaries?), quality (is the agent producing correct, coherent, grounded responses?), performance (is it fast and reliable enough?), cost (is it running efficiently at scale?), and operational control (can we detect, diagnose, and remediate issues quickly?). A mature evaluation practice addresses five continuously, throughout the full agent lifecycle.

在这篇文章中，我们将分享如何在生产环境中的多智能体系统里，将五个评估维度落到实际运作中。安全与保障是我们的首要任务。任何智能体若达不到安全门槛，就不会上线，也不会继续运行。本文对质量着墨最多，因为五个维度中，质量最难把握，也最能从专门的框架中受益，但它的优先级仍低于安全。性能、成本和运维控制则贯穿整个生命周期的每个阶段，与安全和质量并行。关键在于，这套框架适用于智能体层级结构中的每一层。在工具层面，每项能力（例如客户消歧工具、图检索工具）都有各自的评估约定。在智能体层面，每个子智能体都要接受端到端评估。在多智能体层面，则要评估编排路径在智能体之间是否连贯、路由是否正确，以及综合结果是否准确。In this post, we share how we operationalize five evaluation dimensions across a production multi-agent system. Safety and security is our first priority. No agent ships, and no agent keeps running, if it can’t meet its safety bar. Quality gets the most space in this post because it’s the hardest of the five to get right and benefits most from a dedicated framework, but it doesn’t outrank safety. Performance, cost, and operational control are woven into every stage of the lifecycle alongside both. Critically, this framework applies at every level of the agent hierarchy. At the tool level, each capability (a customer disambiguation tool, graph retrieval tool) has its own evaluation contract. At the agent level, each sub-agent is evaluated end-to-end. At the multi-agent level, orchestration paths are evaluated for coherence, correct routing, and synthesis accuracy across agent boundaries.

这九个阶段同时适用于三个层级，而且这种层级结构并非特定于本产品。构建或集成新的生产级智能体时，这才是正确的思维模型。大多数团队从智能体层级开始评估，然后就此止步。但智能体层级的质量取决于它所依赖的工具，而多智能体系统还会引入第三个故障面，单看任何一个层级都无法发现。健全的评估实践会覆盖这三个层级，并让每个层级在这九个阶段中各自使用自己的黄金集、评估器和晋级标准。The nine stages apply at three levels simultaneously, and this hierarchy is not specific to this product. It’s the right mental model when we build or integrate new production agents. Most teams start by evaluating at the agent level and stop there. But agent-level quality is only as good as the tools it relies on, and multi-agent systems introduce a third failure surface that neither level can see alone. A robust evaluation practice covers three, with each level running its own golden set, its own evaluators, and its own promotion criteria through the nine stages.

智能体评估框架：九个阶段，一个闭环The agent evaluation framework: nine stages, one closed loop

这九个阶段描绘了智能体从首次提交代码到上线生产环境的完整生命周期，每个阶段都回答了团队在智能体逐渐成熟过程中面临的一个问题。These nine stages trace the agent lifecycle from first commit to live production, and each one answers a question teams face as their agents mature.

智能体生命周期与九个常见问题The agent lifecycle and the nine common questions

每个构建生产级智能体的团队，都会随着智能体逐渐成熟而遇到以下问题：Every team building production agents encounters the following questions as their agent matures:

阶段 1：基础——“优秀是什么样的？”Stage 1: The Foundation: “What does good look like?”

阶段 2：实验：“哪个版本更好？”Stage 2: The Experiment: “Which version is better?”

第 3 阶段：合并（CI）：“我是不是弄坏了什么？”Stage 3: The Merge (CI): “Did I break something?”

阶段 4：把关（CD）：“可以安全地用于生产环境吗？”Stage 4: The Gate (CD): “Is it safe for production?”

阶段 5：金丝雀：“是不是有什么东西悄无声息地坏掉了？”Stage 5: The Canary: “Did something silently break?”

第 6 阶段：对抗者：“如果有人试图破坏它，会怎样？”Stage 6: The Adversary: “What if someone tries to break it?”

阶段 7：漂移：“线上质量是否正在发生漂移？”Stage 7: The Drift: “Is quality drifting in live?”

第 8 阶段：脉搏：“我们怎么知道它此刻正在正常运行？”Stage 8: The Pulse: “How do we know it’s working right now?”

阶段 9：反馈循环：“它如何自动变得更好？”Stage 9: The Feedback Loop: “How does it get better automatically?”

我们使用 AgentCore Evaluations，为智能体生命周期的每个阶段构建统一框架。这九个阶段并非事后拼接起来的独立工具，而是同一连续评估体系上的九个环节。从首次提交代码到生产环境中的响应，整个过程都由 AgentCore Evaluations 提供支持，如图 1 所示。We build one framework for every stage of the agent lifecycle using AgentCore Evaluations. The nine stages are not separate tools stitched together after the fact. They’re nine points on a single, continuous evaluation surface, powered by AgentCore Evaluations from first commit to production response, as shown in Figure 1.

图 1：AgentCore Evaluations 的九阶段流程，从“定义”到“改进”Figure 1: The nine-stage AgentCore Evaluations journey, from Define through Improve

以下是我们应对每个问题的方式：行之有效的模式、为其提供支持的 AgentCore 服务，以及它在生产环境中的运行情况，系统设计如图 2 所示。Here’s how we tackle each one: the pattern that works, the AgentCore services behind it, and what it looks like running in production, shown in system design in Figure 2.

图 2：AgentCore 评估框架的系统设计Figure 2: System design of the AgentCore evaluation framework

阶段 1：基础。定义质量Stage 1: The Foundation. Define quality

评估代理的基础不在于选择合适的工具，而在于提出正确的问题：这个代理的成功是什么样的？我们如何判断它何时失效？答案必须来自客户，而不是模型。这是一个迭代过程。随着我们观察代理在生产环境中的实际表现，我们对成功的定义也变得更加明确。这里定义的所有内容（失败阈值、评估器栈、黄金集、安全规则、延迟预算和成本目标）都是后续每个阶段都必须执行的契约。如果第 1 阶段的定义含糊不清，下游的每个关卡都只能靠猜。The foundation of evaluating an agent isn’t about choosing the right tools. It’s about asking the right question: what does success look like for this agent, and how do we know when it is failing? The answer must come from the customer, not from the model. This is an iterative process. Our definition of success has sharpened as we observe what the agent actually does in production. Everything defined here (failure thresholds, evaluator stack, golden set, safety rules, latency budgets, and cost targets) is the contract every subsequent stage enforces. If Stage 1 is vague, every downstream gate is guessing.

阶段 1.1：首先定义业务指标。先问清楚这个智能体要实现什么目标。两类终端用户角色对成功的定义各不相同：Stage 1.1: Define your business metrics first. Start by asking what the agent exists to deliver. The two end user personas define success differently:

• 对销售人员而言，成功意味着在恰当的时机，以合适的细节层次，掌握准确且最新的情报，与客户展开交流。
  For sellers, success means walking into a customer conversation with accurate, current intelligence at the right detail level and at the right time.
• 对于领导层而言，成功意味着每周及跨业务审查都能获得可靠的输入，且无需耗费大量时间手动准备。这些是业务成果，而非模型属性。
  For leadership, success means reliable weekly and cross-business review inputs without lengthy manual preparation. These are business outcomes, not model properties.

第 1.2 阶段：将这些业务指标转化为评估指标。针对每项结果，确定能够确认智能体已达成该结果的信号。每个信号都应成为一项跟踪指标，并明确其阈值和告警条件。准确的客户情报对应实体解析准确率和依据支撑率。卖家信任对应忠实度和幻觉率。卖家行动对应有用性和范围遵循度。领导层审核的可靠性对应准确率达标率和响应覆盖率。除了质量阈值之外，这项转化还必须涵盖五个维度：定义安全护栏策略；为每种工具和每个智能体设定 p50、p90 和 p99 延迟目标；制定每次查询的成本预算；并定义服务级别协议（SLA），明确何时触发告警以及触发哪个运行手册。Stage 1.2: Translate those business metrics into evaluation metrics. For each outcome, identify the signal that confirms the agent delivered it. Each signal becomes a tracked metric with an explicit threshold and an alarm condition. Accurate account intelligence maps to entity resolution accuracy and grounding rate. Seller trust maps to faithfulness and hallucination rate. Seller action maps to helpfulness and scope adherence. Leadership review reliability maps to accuracy pass rate and response coverage. This translation must cover five dimensions: alongside quality thresholds, define safety guardrail policies, set p50, p90, and p99 latency targets per tool and per agent, establish cost-per-query budgets, and define the service level agreements (SLAs) that determine when an alarm fires and which runbook it triggers.

尤其是在准确性方面，由于该解决方案会直接影响收入决策、续约策略和高管评审，因此标准很高。一个看似笃定却错误的答案，浪费的不只是销售人员的时间，还会导致业务关键时刻做出错误决策。我们根据实际生产环境中的错误定义了六种失败类型，只要出现其中任何一种，就意味着回答不准确：For accuracy specifically, because the solution directly informs revenue decisions, renewal strategies, and executive reviews, the bar is high. A confidently wrong answer does not only waste a seller’s time. It leads to a wrong decision in a real business moment. We define six failure types derived from real production errors, any one of which marks a response as inaccurate:

• 指标置换：客户和期间正确，但返回的指标有误。按每 100 条回复中的发生次数进行跟踪。
  Metric swap: correct customer and period, wrong metric returned. Tracked as occurrences per 100 responses.
• 周期错误：一个时间窗口的数据被标记为另一个时间窗口，例如将同比（YoY）数据作为环比（MoM）数据呈现给卖家，或反之。按引用正确时间窗口的百分比进行跟踪。
  Wrong period: data from one time window is labeled as another, for example a year-over-year (YoY) figure presented to the seller as month-over-month (MoM), or the reverse. Tracked as % citing the correct window.
• 实体错误：将账户名称或子公司混为一谈。以实体解析准确率（%）进行跟踪。
  Wrong entity: conflated account names or subsidiaries. Tracked as entity resolution accuracy %.
• 纯属幻觉：没有可追溯来源、缺乏依据的数据。以依据率百分比进行跟踪。
  Pure hallucination: ungrounded figures with no traceable source. Tracked as grounding rate %.
• 符号错误：指标正确，但货币或单位符号错误。以范围遵循率（%）进行跟踪。
  Wrong symbols: correct metric, wrong currency or unit symbol. Tracked as scope adherence rate %.
• 未经授权的计算：源数据中不存在的、未经批准的派生数据。按每 100 条回复中的发生次数进行跟踪。
  Unauthorized calculation: unsanctioned derived figures not present in the source data. Tracked as occurrences per 100 responses.

总体准确率通过率是核心指标。按维度跟踪才能让指标真正发挥作用。除了准确率，我们还跟踪回复相关性、忠实度、有用性、指令遵循情况和工具选择准确率。答案即使正确，如果埋在三段文字中，或来得太迟，对卖家来说也算失败。The overall accuracy pass rate is the headline metric. Per-dimension tracking is what makes it actionable. Beyond accuracy, we also track Response relevance, Faithfulness, Helpfulness, Instruction Following,Tool Selection Accuracy. A correct answer buried in three paragraphs, or arriving too late, has also failed the seller.

阶段 1.3：构建能够反映这些指标的评估器栈。任何单一类型的评估器都无法覆盖可能出现的各种失败模式。我们将四种类型结合起来：Stage 1.3: Build an evaluator stack that reflects those metrics. No single evaluator type covers the various failure modes possible. We combine four types:

• AgentCore 内置评估器可提供广泛的基线覆盖。相关性、忠实度和有用性可直接对应到 AgentCore Evaluations 原生提供的评估模板。
  AgentCore built-in evaluators for broad baseline coverage. Relevance, faithfulness, and helpfulness map directly to evaluation templates available natively in AgentCore Evaluations.
• 针对我们的分类体系中的每种错误类型，各自定制一个评估器：指标替换评估器、时间段错误评估器、客户错误评估器、纯幻觉评估器、符号错误评估器，以及未经授权的计算评估器。这六种都不在原生模板的覆盖范围内。它们是我们基于 AgentCore 框架构建的领域专用信号。事先了解原生模板已经覆盖哪些指标，以及哪些指标需要定制评估器，本身就是设计决策的一部分。
  Customized evaluators, one per failure type in our taxonomy: a metric swap evaluator, a wrong period evaluator, a wrong customer evaluator, a pure hallucination evaluator, a wrong symbols evaluator, and an unauthorized calculation evaluator. None of the six are covered by native templates. They’re domain-specific signals we built on top of the AgentCore framework. Knowing upfront which metrics a native template already covers, and which require a customized evaluator, is itself part of the design decision.
• 第三方评估器方面，我们部署了来自第三方开源库 DeepEval 和 AutoEval 的托管评估器。
  Third-party evaluators, we deployed the managed evaluators from third-party open source libraries DeepEval and AutoEval.
• 用于核验已知确切正确答案的事实依据的确定性评估器，例如收入、续约日期和合同状态。
  Deterministic evaluators for ground truth checks where the correct answer is known exactly, such as revenue, renewal date, and contract status.

对于推理质量和有用性没有唯一正确答案的情况，我们改用 LLM-as-judge 评分标准，内置了相关性、忠实度和有用性评分标准，并支持完全自定义的评分标准。AgentCore Evaluations 原生提供基于真实标注答案和 LLM-as-judge 的评估，因此我们无需自行构建任何评分引擎，就能搭建这两种评估流程。确定性评估能精准核验事实，LLM-as-judge 则能广泛评估推理。Where reasoning quality and helpfulness have no single correct answer, we use LLM-as-judge rubrics instead, with built-in rubrics for relevance, faithfulness, and helpfulness plus support for fully custom rubrics. AgentCore Evaluations provides both ground truth and LLM-as-judge evaluation natively, which is what let us stand up both evaluation paths without building either scoring engine ourselves. Deterministic gives precision on facts. LLM-as-judge gives breadth on reasoning.

阶段 1.4：构建黄金测试集，用于检验评估指标。在工具层面，每个关键工具都有各自确定性的标准答案对：客户识别必须解析到确切的规范实体，否则就判定失败。在智能体层面，对真实用户查询进行主题分析，生成涵盖四类问题（综合类、报告类、推理类和模糊类）的数据集，每类问题都需要采用不同的评估策略，并每月重新审视。在多智能体层面，端到端编排场景可以捕捉只有在智能体交互时才会出现的错误。黄金测试集不仅是质量数据集，还包括安全测试用例、延迟基准用例和成本参考用例，因此第 3 和第 4 阶段可以基于同一经过整理的数据集检查五个维度。Stage 1.4: Build a golden set to exercise your evaluation metrics. At the tool level, each critical tool has its own deterministic ground truth pairs: customer identification must resolve to the exact canonical entity or fail. At the agent level, topic analysis on real user queries produces a dataset across four question types (comprehensive, reporting, reasoning, and ambiguous), each requiring a different evaluation strategy, revisited monthly. At the multi-agent level, end-to-end orchestration scenarios catch errors that only emerge when agents interact. The golden set is not only a quality dataset. It also includes safety test cases, latency benchmark cases, and cost reference cases, so Stages 3 and 4 can check five dimensions against the same curated dataset.

黄金测试集是一份持续演进的文档，由三个来源构成。源自生产环境的用例从卖家和管理层的真实查询中抽样得出，包括在第 7 阶段提升的低评分线上会话。源自故障的用例涵盖在预发布或生产环境中捕获的所有回归问题，包括在第 3 阶段提升的哨兵用例，以及在第 9 阶段提升的永久回归防护用例。合成边界用例则经过专门设计，旨在确保覆盖我们已知至关重要、但不能等到实际发生才处理的场景，例如上述四种问题类型，以及第 4 阶段中的母子公司和股票代码消歧用例。该测试集采用版本控制，经过审查，并持续从这三个来源扩充。The golden set is a living document, built from three sources. Production-derived cases are sampled from real seller and leadership queries, including the low-scoring live sessions promoted in Stage 7. Failure-derived cases come from every regression caught in staging or production, including the sentinel cases promoted in Stage 3 and the permanent regression guards promoted in Stage 9. Synthetic edge cases are deliberately constructed to guarantee coverage of scenarios we know matter but can’t wait to encounter in the wild, such as the four question types above and the parent-subsidiary and ticker-symbol disambiguation cases in Stage 4. The set is version-controlled, reviewed, and continuously extended across three sources.

阶段 2：实验。了解哪个版本胜出。Stage 2: The Experiment. Know which version wins

每一次变更（更新提示词、增加新工具、升级模型、刷新数据集）都会引出同一个问题：这个版本真的更好吗？对于一个面向全球 AWS 一线团队提供服务的系统来说，凭直觉作答并不可取。Every change (a prompt update, a new tool, a model upgrade, a dataset refresh) raises the same question: is this version actually better? In a system serving the AWS field at global scale, intuition isn’t an acceptable answer.

在变更合并前，我们会运行一次 AgentCore 按需评估，并使用相同的黄金集和评估器套件进行评分。开发团队会利用质量、延迟、成本指标和上下文令牌，整理下游对比结果，并同时从五个维度进行评估，如图 3 所示。准确率提高 5%，但延迟翻倍、成本增至三倍的模型，并不会自动胜出。Before a change merges, we run an AgentCore On-Demand Evaluation, scored against the same golden set with the same evaluator suite. The development team uses the quality, latency, cost metrics, and context token, to curate downstream comparisons which is evaluated across five dimensions simultaneously, as shown side by side in Figure 3. A model that improves accuracy by 5 percent but doubles latency and triples cost doesn’t automatically win.

图 3：在 AgentCore 按需评估中，对两个候选智能体响应在质量、延迟和成本方面进行并列比较。图中的数字和数值仅供示意。Figure 3: Side-by-side comparison of two candidate agent responses in an AgentCore On-Demand Evaluation across quality, latency, and cost. The numbers and values in the figure are illustrative.

我们将质量跟踪为一份全面的评分卡，而不是单一数字。仅看通过率并不能说明足够多的信息。每次评估运行都会根据我们分类体系中的每种失败类别，生成完整的细分结果：评分卡（见图 4）不仅记录质量是否有所提升，还会指出提升发生在哪里、原因是什么。每次评估也都会被完整记录：输入、输出、检索到的上下文、分数和评估理由都存储在 S3 和我们的数据湖中，为我们提供持久的可观测性层，以便追溯失败的根本原因：智能体看到了什么、生成了什么，以及出错的原因。经过三个月有针对性的迭代，这份评分卡推动通过率从 93% 提升至稳定的 96%：纯粹幻觉从每 10,000 条响应中的 138 次降至不足 52 次，未经授权的计算从 59 次降至不足 44 次，这证实了检索过程本身没有问题，取得的进步源于生成环节的修复。如果没有按类别细分的结果以及背后的详细追踪数据，我们的迭代就只能盲目进行。We track quality as a comprehensive scorecard, not one single number. A pass rate alone doesn’t tell you enough. Each eval run produces a full breakdown across every failure category in our taxonomy: not just whether quality improved, but where and why, as captured in the scorecard in Figure 4. Every evaluation is also fully recorded: inputs, outputs, retrieved context, scores and evaluation reasons are stored in S3 and our data lake, giving us a persistent observability layer to trace failure back to its root cause: what the agent saw, what it generated, and why it went wrong. Over three months of targeted iteration, this scorecard drove pass rate from 93 percent to a stable 96 percent: pure hallucinations dropped from 138 to under 52 per 10,000 responses, unauthorized calculations fell from 59 to under 44, confirming that retrieval was sound and the gains came from generation fixes. Without the per-category breakdown and the detailed trace data behind it, we would have been iterating blind.

图 4：评估评分卡，展示各类别的失败情况明细和通过率趋势。图中的数字和数值仅供示意。Figure 4: Evaluation scorecard with per-category failure breakdown and pass-rate trend. The numbers and values in the figure are illustrative.

阶段 3：合并（CI）。在合并时防止回归Stage 3: The Merge (CI). Prevent regression at merge time

阶段 3 是预生产门禁。它在预发布环境中运行，由代码和配置变更触发，目的是在不良变更进入生产环境之前将其拦截。如果阶段 3 检测到回归，任何内容都不会发布。Stage 3 is a pre-production gate. It runs in the staging environment, triggered by code and configuration changes, and its job is to block a bad change before it ever reaches production. If Stage 3 catches a regression, nothing ships.

工程师或科学家会以多种方式改变系统：提示词、底层数据、工具或模型。这些改动在本地看起来很完善，测试也都通过了，但一个能提升科技类账户回复质量的提示词，可能会悄无声息地降低金融服务类账户的回复质量。如果没有自动化门禁，这些改动就会被发布。Engineers or scientists change the system in many ways: the prompt, the underlying data, the tools, or the model. The change looks clean locally, tests pass, but a prompt that improves responses for technology accounts could silently degrade responses for financial services accounts. Without an automated gate, this ships.

我们的 CI 评估会在每次部署到暂存环境时，通过 AgentCore On-Demand Evaluation 自动触发。工程师无需记得手动运行评估。任何部署都无法悄无声息地绕过这道关卡。由于评估的只是哨兵集，而非完整的黄金集，整个流程只需几分钟即可完成。速度是刻意设计的：哨兵集范围足够小，可以在每次部署时运行，同时又具有足够的代表性，能够捕捉最重要的故障。Our CI evaluation is triggered automatically on every staging deployment using AgentCore On-Demand Evaluation. Engineers do not need to remember to run an evaluation. No deployment can silently skip the gate. The run completes in minutes because it evaluates only the sentinel set, not the full golden set. Speed is intentional: the sentinel is narrow enough to run on every deployment but representative enough to catch the failures that matter most.

哨兵测试集是从五个维度中精选出的高风险场景集合。超出容忍范围的质量回退会阻止部署。违反安全边界同样会被阻止。p90 延迟翻倍的变更，未经审慎决策不得上线。令牌成本激增会在规模扩大、影响累积之前被标记出来。每次哨兵测试运行后，都会将其评分作为自定义指标发布到 CloudWatch；质量评分、延迟、每次查询成本和安全通过率也都会在每次运行后上报，并配置 CloudWatch 警报，在任何指标突破其第 1 阶段阈值时触发。每次运行也都会记录并存储，为第 8 阶段和第 9 阶段所依赖的审计轨迹提供数据。我们在工具、智能体和多智能体三个层级分别维护哨兵测试集。即使某项变更提升了智能体层面的质量，却损害了某个关键工具，也会被阻止。哨兵测试集每月审核一次；任何漏过测试并进入生产环境的回归问题，都会立即被纳入永久测试用例，确保不会让用户再次遇到。The sentinel set is a curated selection of the highest-risk scenarios across five dimensions. Quality regressions beyond tolerance block the deployment. A safety boundary violation blocks just as hard. A change that doubles p90 latency does not proceed without a deliberate decision. Token cost spikes are flagged before they compound at scale. Every sentinel run publishes its scores to CloudWatch as custom metrics alongside quality scores, latency, cost per query, and safety pass rate are all emitted after every run, and CloudWatch alarms are configured to fire if any metric breaches its Stage 1 threshold. Every run is also logged and stored, feeding the audit trail Stage 8 and Stage 9 rely on. We maintain a separate sentinel at each level: tool, agent, and multi-agent. A change that improves agent-level quality but degrades a critical tool still blocks. The sentinel is reviewed monthly, and a regression that slips through to production is immediately promoted into a permanent case so it doesn’t reach users twice.

这是它能够捕获的一种情况。一次数据规范化更新后，消歧工具开始对某些账户名称格式返回未解析实体。智能体收到未解析实体后，给出了一个最可能的匹配结果，返回了错误客户的真实数据。我们的工具级哨兵检查恰好涵盖了这类边缘情况：受影响账户名称格式的实体解析基准事实检查未通过，因此部署被自动拦截。这项变更从未进入生产环境。One example of what this catches. After a data normalization update, a disambiguation tool started returning unresolved entities for a subset of account name formats. The agent, receiving an unresolved entity, responded with a best-guess match, returning real data for the wrong customer. Our tool-level sentinel includes exactly this class of edge case: the entity resolution ground truth check failed for the affected account name formats, and the deployment was blocked automatically. The change never reached production.

阶段 4：关卡（CD）。以量化信心交付Stage 4: The Gate (CD). Ship with quantitative confidence

哨兵测试覆盖 15 种情况。生产环境则涵盖数万种查询模式，涉及数百种账户类型、地理区域和卖家细分群体。通过哨兵测试是必要条件，但并非充分条件。The sentinel covers 15 cases. Production covers tens of thousands of query patterns across hundreds of account types, geographies, and seller segments. Clearing the sentinel is necessary. It is not sufficient.

候选版本上线前，我们会针对完整的黄金数据集，在各项评估维度上运行一次 AgentCore 按需评估。我们称之为质量门禁。它以量化的上线决策取代“看起来没问题”这种主观判断。Before a candidate ships, we run an AgentCore On-Demand Evaluation against the full golden set across the evaluator dimensions. We call this the Quality Gate. It replaces “it seems fine” with a quantitative shipping decision.

沙箱化执行和工具模拟。可以把它看作一次回放，而不是实时调用。我们会提前冻结一份已知且带版本的正确工具行为数据集。候选系统在准入评估期间发出的每次工具调用，都会被拦截，并从这份冻结数据集中获得响应，而不是调用真实系统。我们分两步构建这份冻结数据集。首先，单独测试每个工具：从黄金测试集中选取一组留出输入发送给工具，这些输入的正确答案已知，然后将工具的实际返回结果与已知答案进行评分比较。这样，在智能体参与之前，我们就能判断工具本身是否正常工作（实体解析、检索、SQL 生成）。其次，确认工具的响应正确后，我们会将这些经过验证的相同答案冻结到回放数据集中，供智能体在准入评估期间使用。这意味着，测试只考察智能体自身的推理能力，而不受实时工具的速度或可用性影响；并且由于每个候选系统都基于同一份冻结数据集进行回放，直接对比时出现的任何分数差异都反映的是智能体本身，而非工具表现的波动。Sandboxed execution and tool mocking. Think of it as a replay, not a live call. Ahead of time, we freeze a known, versioned dataset of correct tool behavior. Every tool call the candidate makes during the gate is intercepted and answered from that frozen dataset instead of a real system. We build that frozen dataset in two steps. First, we test each tool on its own: we send it a held-out set of inputs from our golden set where the correct answer is already known, and score what it actually returns against that known answer. This tells us whether the tool itself works (entity resolution, retrieval, SQL generation) before the agent is even involved. Second, after a tool’s responses are confirmed correct, we freeze those same validated answers into the replay dataset the agent sees during gate evaluation. That means the agent is tested purely on its own reasoning, not on live tool speed or availability, and because every candidate replays from the same frozen dataset, any score difference head-to-head is about the agent, not tool variance.

端到端聚合评分。该门禁还会进行全面评估：由完整的候选智能体端到端回答 200 道留出问题，并按照涵盖所有评估维度的 LLM-as-judge 评分标准进行评分。出现以下三种情况中的任意一种，门禁都会自动关闭：聚合得分低于候选版本自身上一次运行的得分；未能超过当前生产基线；或低于单题阈值的问题比例超过预设百分比。候选版本必须满足全部三项要求。这样可以捕捉简单平均值可能掩盖的问题：模型即便提高了平均分，却悄然导致更多单题表现不达标，也无法通过。Aggregated end-to-end scoring. The gate also runs a holistic pass: the full candidate agent answers 200 held-out questions end-to-end, scored by an LLM-as-judge rubric across all evaluator dimensions. Any one of three conditions shuts the gate automatically: the aggregate score regresses against the candidate’s own prior run, it fails to beat the current production baseline, or more than a defined percentage of the 200 questions falls below the per-question threshold. A candidate must clear all three, which catches what a simple average can hide: a model that improves the mean while quietly failing a larger share of individual questions doesn’t pass.

下面是这种测试能捕捉到的问题示例。在这里，模拟测试考验的是代理自身的推理能力，而不只是它底层使用的工具。我们的金标集包含一个层级不匹配案例：客户洞察代理和数据洞察代理针对同一客户返回了不同但各自正确的收入数据，一个按母账户统计，另一个按子公司统计。两个数字都没有错。这个案例测试代理能否根据查询的语义和请求者的角色推断出正确的统计范围（销售人员很可能想看子公司的数据，而管理层的查询很可能想看母账户汇总数据），而不是默认采用某个范围、取平均值，或要求销售人员进一步说明。候选方案未通过，对一个限定为子公司的问题返回了母账户层级的数据。这个案例的金标事实明确规定了问题预期的统计范围，因此质量门禁立即标记出了不匹配：数字本身没错，但统计范围错了。One example of what this catches. This is where mocking tests the agent’s own reasoning, not just the tools underneath it. Our golden set includes a hierarchy mismatch case: the Customer Insights Agent and Data Insights Agent return different, individually correct revenue figures for the same customer, one scoped to the parent account, the other to a subsidiary. Neither number is wrong. The case tests whether the agent infers the right scope from the query’s semantics and the requester’s persona (a seller likely wants the subsidiary, whereas a leadership query likely wants the parent rollup), rather than defaulting, averaging, or asking the seller to disambiguate. The candidate failed, returning the parent-level figure for a subsidiary-scoped question. The golden set’s ground truth for this case specifies which scope the question expects, so the Quality Gate flagged the mismatch immediately: not a wrong number, but the right number for the wrong scope.

阶段 5：金丝雀。捕获部署后的静默故障Stage 5: The Canary. Catch silent post-deploy failures

第 5 阶段是一个生产后监控器。它会在生产环境中持续运行，监测实时流量和合成评估用例，旨在发现那些并非由代码变更触发、因而第 3 阶段未能发现的问题。第 3 阶段防范工程师所做的变更；第 5 阶段则防范其他所有情况，例如上游数据发生偏移、模型行为不断演变、第三方依赖项发生变化，或延迟增加、流量模式变化等只有在真实世界负载下才会显现的情况。Stage 5 is a post-production monitor. It runs continuously in the production environment against live traffic and synthetic evaluation cases, and its job is to catch issues that no code change triggered and that Stage 3 therefore didn’t see. Stage 3 guards against what engineers change. Stage 5 guards against everything else like upstream data shifts, evolving model behavior, changes in third-party dependencies, or other conditions such as increased latency or shifting traffic patterns that only manifest under real-world load.

该产品运行于一个由数千个上游架构和数据管道供数的数据体系之上。在某一天，其中数十条管道会刷新，但不会触发部署或 CI 门禁。智能体代码没有变化，模型也没有变化，但它推理所依据的数据变了，因此质量可能在不知不觉中下降。我们为此付出了惨痛代价：一条上游管道在刷新时更改了架构，智能体开始引用错误时间窗口的数据，而我们收到的第一个信号来自卖家问询，而非自动告警。第 5 阶段就是让我们在首个 30 分钟评估窗口内发现这类问题。The product operates against a data estate fed by thousands of upstream schemas and data pipelines. On a given day, dozens of those pipelines refresh without triggering a deployment or CI gate. The agent code is unchanged, the model is unchanged, but the data it reasons over has changed, and quality can degrade silently as a result. We learned this the hard way: an upstream pipeline refreshed with a schema change, the agent started citing data from the wrong time window, and the first signal came through a seller inquiry rather than an automated alert. Stage 5 is how we catch this in the first 30-minute evaluation window instead.

我们以两种互为补充的周期运行 AgentCore Batch Evaluation，周期的选择是经过深思熟虑的。We run AgentCore Batch Evaluation on two complementary schedules, and the choice of cadence was deliberate.

在听取正反两方面的经验后，我们最终确定采用 30 分钟为一个周期：15 分钟会产生太多误报；60 分钟则会让真实性能退化在被检测到前持续两个小时。我们将每个 30 分钟周期称为一个评估窗口。连续三个窗口即可在 90 分钟内捕捉到真实趋势，同时滤除单次查询造成的噪声。每个窗口都会对黄金集中的一组轮换样本进行评分，从而在 24 小时内覆盖全部样本。We settled on 30-minute cycles after learning from both directions: 15 minutes generated too many false alarms; 60 minutes let two hours pass before a real degradation registered. We call each 30-minute cycle an evaluation window. Thirty minutes catches a real trend within 90 minutes (three consecutive windows) while filtering out single-query noise. Each window scores a round-robin selection across the golden set, giving full coverage in 24 hours.

阈值有意设得比 CI/CD 门禁低。单个窗口失败只是运维信号，并不代表发生了事故。只有连续三个窗口都超过阈值时，我们才会通知值班人员。我们是吃过亏才定下这条规则的：第一个月采用单窗口阈值时，值班人员彻底不再信任这套系统。每天运行一次完整的黄金测试集，使用门禁阈值，作为权威的每日评分卡。30 分钟窗口用于呈现趋势；每日运行则负责确认趋势并创建工单。Thresholds are deliberately lower than CI/CD gates. A single window failure is an ops signal, not an incident. We only page oncall after three consecutive windows breach threshold. We learned this rule the hard way: the first month we used a single-window threshold, oncall stopped trusting the system entirely. Once per day, the full golden set runs at gate thresholds as the authoritative daily scorecard. The 30-minute windows surface the trend. The daily run confirms it and cuts a ticket.

每次运行都会将指标发布到 CloudWatch（准确率、幻觉率、事实依据率、延迟和成本），并配置能够区分两种运行频率的告警逻辑。金丝雀测试分为三个层级：工具级评分能在检索工具退化并影响智能体级响应之前及时发现问题。Every run publishes to CloudWatch (accuracy, hallucination rates, grounding rate, latency, and cost), with alarm logic that distinguishes between the two cadences. The canary runs at three levels: tool-level scores surface a degraded retrieval tool before it corrupts agent-level responses.

再举一个它能捕获的问题。上游索引刷新后，检索工具开始返回相关性较弱的片段，而不是回答查询所需的精确上下文。延迟正常，错误率为零。智能体依靠依据薄弱的上下文进行推理，仍不断生成流畅、自信的回答；这些答案看似正确，引用的数据却有误。没有触发任何基础设施告警。金丝雀监控在第一个 30 分钟窗口内就捕获到了问题：工具级哨兵的依据充分率和检索精确率均有所下降，触发了 CloudWatch 告警，早于卖家根据这些质量下降的答案采取行动。评估器的低分标出了导致问题的确切跟踪记录。之后，值班工程师在 AgentCore 中打开该跟踪记录，并在几分钟内追溯到检索工具，找到了上下文质量下降的源头。Another example of what this catches. After an upstream index refresh, a retrieval tool began returning loosely relevant chunks instead of the precise context needed to answer the query. Latency was nominal. Error rate was zero. The agent kept producing fluent, confident responses by reasoning over weakly grounded context, producing answers that looked correct but cited the wrong figures. No infra alarm fired. The canary caught it in the first 30-minute window: grounding rate and retrieval precision dropped across the tool-level sentinel, triggering a CloudWatch alarm before sellers acted on the degraded answers. The low evaluator score flagged the exact trace responsible. From there, the on-call engineer opened that trace in AgentCore and traced the degraded context back to the retrieval tool in minutes.

阶段 6：对抗者。防范滥用。Stage 6: The Adversary. Protect against misuse

并非所有糟糕的输出都源于模型性能退化。旨在窃取未经授权账户数据的查询，以及嵌入在检索文档中的提示注入，都是一个处理数千名用户敏感业务数据的系统中可预见的模式。Not every bad output comes from a degraded model. Queries crafted to extract unauthorized account data, and prompt injections embedded in retrieved documents, are predictable patterns in a system serving thousands of users on sensitive business data.

安全不是一个单独的阶段，而是贯穿全部九个阶段。我们会在每次交互的两个环节应用 Amazon Bedrock Guardrails。在输入环节，我们会验证查询范围并检测提示注入。在输出环节，我们会确认没有向未经授权的用户泄露敏感数据，也确认智能体未被操纵而违反策略。这些检查会对 100% 的流量同步执行，确保响应在到达用户之前完成检查。Safety isn’t a stage. It runs through all nine stages. We apply Amazon Bedrock Guardrails at two points in every interaction. At input, we validate query scope and screen for prompt injection. At output, we verify that no sensitive data is surfaced to unauthorized users and that the agent has not been manipulated into a policy violation. These checks run on 100 percent of traffic, synchronously, before a response reaches the user.

我们配置了两层护栏，分别应对不同类别的威胁：We configure two guardrail layers, each covering a distinct threat class:

内容过滤器会检测并拦截输入和输出中的有害或不当内容。我们会对包括暴力、仇恨、性内容、不当行为和侮辱在内的类别执行过滤。触发这些过滤器的请求或回复会在送达卖家或呈现给用户之前被拦截。Content filters detect and block harmful or inappropriate content in both inputs and outputs. We enforce filters across categories including VIOLENCE, HATE, SEXUAL, MISCONDUCT, and INSULTS. A request or response that triggers these filters is blocked before it reaches the seller or is surfaced to the user.

提示安全过滤器通过一个名为 PROMPT_ATTACK 的防护类别保护智能体免受对抗性操纵，该类别涵盖三种子类型：JAILBREAK（试图覆盖智能体的系统指令或绕过其运行边界）、PROMPT_INJECTION（嵌入在检索文档、工具输出或用户查询中的非预期指令）以及 PROMPT_LEAKAGE（试图提取系统提示或内部配置）。在使用检索增强生成（RAG）的多智能体系统中，通过检索内容进行提示注入是一项真实风险。卖家账户数据中的某份非预期文档可能会试图在智能体推理过程中途改变其行为。Prompt security filters protect the agent from adversarial manipulation under a single PROMPT_ATTACK guardrail category, covering three sub-types: JAILBREAK (attempts to override the agent’s system instructions or bypass its operational boundaries), PROMPT_INJECTION (unintended instructions embedded in retrieved documents, tool outputs, or user queries), and PROMPT_LEAKAGE (attempts to extract the system prompt or internal configuration). In a multi-agent system that uses Retrieval Augmented Generation (RAG), prompt injection through retrieved content is a real risk. An unintended document in a seller’s account data could attempt to redirect the agent’s behavior mid-reasoning.

安全规则在阶段 1 中定义，在阶段 3 和阶段 4 中检查是否出现回归，并在生产环境中持续执行。Safety rules are defined at Stage 1, checked for regressions at Stages 3 and 4, and enforced continuously in production.

阶段 7：漂移。借助真实流量维持质量并进行验证Stage 7: The Drift. Sustain quality and validate with live traffic

质量会从多个方面开始下降：上游数据发生变化却未触发部署，模型逐渐过时，用户提出超出任何黄金测试集预期范围的问题。第 7 阶段会在卖家察觉之前检测到这些问题，并在全面上线前，使用真实用户流量验证候选版本。Quality degrades from multiple directions: upstream data changes without triggering a deployment, models grow stale, and users push boundaries with queries no golden set fully anticipates. Stage 7 detects it before sellers do, and validates candidates against real user traffic before full rollout.

漂移检测。使用 AgentCore Online Evaluation，我们持续对 10% 的实时流量进行抽样，并通过第 1 阶段定义的不同评估器进行评估，因此一旦出现偏离离线建立的基线，就能立即发现漂移。评分会输入质量状态计算器，在同一时间窗口内监控质量、延迟和成本。Drift detection. Using AgentCore Online Evaluation, we sample 10 percent of live traffic continuously through the different evaluators defined at Stage 1, so the drift is immediately visible as a deviation from the baselines established offline. Scores feed into a Quality-State Calculator monitoring quality, latency, and cost in the same window.

一个生产环境示例。我们的客户识别金标准集基于合成边缘案例构建，涵盖姓名变体、子公司查询和缩写。它在线下表现良好。但在实际流量中，卖家会使用股票代码（“AMZN 怎么了？”）、简称和合成案例未涵盖的拼写错误进行查询。质量状态计算器在连续三个 30 分钟窗口中发现了性能漂移：凌晨 1:30 为 98%，凌晨 2:00 为 81%，凌晨 2:30 为 80%。滚动阈值被突破时，CloudWatch 向值班人员发送了告警。值班工程师向智能体询问：“凌晨 1 点到 3 点之间，客户识别准确率发生了什么变化？”智能体给出趋势摘要，指出股票代码查询和简称是导致故障的原因。智能体的追踪记录让根本原因一目了然。解决办法是：将低分的线上会话纳入金标准集，并更新消歧工具，在进行匹配前先规范化输入格式。经过下一个评估周期，准确率便恢复到了基线水平。One production example. Our golden set for customer identification was built from synthetic edge cases: name variations, subsidiary lookups, abbreviations. It performed well offline. But in live traffic, sellers queried with stock ticker symbols (“what’s going on with AMZN?”), shorthand names, and typos the synthetic cases did not exercise. The Quality-State Calculator surfaced the drift across three consecutive 30-minute windows: 98 percent at 1:30am, 81 percent at 2:00am, 80 percent at 2:30am. CloudWatch paged oncall when the rolling threshold was crossed. The on-call engineer queried the agent: “What happened to customer identification accuracy between 1am and 3am?” and received a trend summary pinpointing ticker symbol lookups and shorthand names as the failure driver. The agent’s trace made the root cause immediately visible. The fix: expand the golden set with the low-scoring live sessions, and update the disambiguation tool to normalize input formats before resolution. Accuracy recovered to baseline within the next evaluation cycle.

实时流量为两个持续改进闭环提供输入：低分会话被纳入黄金集，作为新的边缘案例；低分轨迹的聚类则为提示词修订、工具更新和路由调整提供依据。Live traffic feeds two continuous improvement loops: low-scoring sessions are promoted into the golden set as new edge cases, and clusters of low-scoring traces inform prompt revisions, tool updates, and routing adjustments.

A/B 测试：候选版本通过质量门槛后，AgentCore Gateway（Amazon Bedrock AgentCore 的一项功能）会将一部分真实流量分配给当前智能体和候选智能体。按需评估可确认候选版本在已知案例中没有出现性能回退。在线评估则可确认它能够应对真实用户才会产生的长尾输入和新颖表述。在全面 rollout 之前，候选版本必须在所有评估维度上胜出或打平。A/B testing: After a candidate clears a quality gate, AgentCore Gateway, a capability of Amazon Bedrock AgentCore, routes a slice of real traffic between current and candidate agents. On-demand confirms no regression on known cases. Online confirms it holds up against the long-tail inputs and novel phrasings only real users produce. A candidate must win or draw across all evaluator dimensions before full rollout.

阶段 8：脉动。实时质量可见性Stage 8: The Pulse. Real-time quality visibility

对代理的一次调用可能会涉及三个子代理、五种工具和两条检索路径。质量警报触发时，“代理流水线出了问题”并不能作为操作手册。第 8 阶段将第 5、6、7 阶段的信号汇聚到可观测性层，提供足够精细的监控粒度，以便采取行动。A single call to the agent might touch three sub-agents, five tools, and two retrieval paths. When a quality alarm fires, “something in the agent pipeline” is not a runbook. Stage 8 is where signals from Stages 5, 6, and 7 converge into an observability layer with enough resolution to act.

AgentCore 可观测性是 Amazon Bedrock AgentCore 的一项功能，也是基础。对于托管在 AgentCore 运行时（Amazon Bedrock AgentCore 的一项功能）上的智能体，每次执行都会通过 OpenTelemetry 自动进行监测，为每个推理步骤生成结构化遥测数据：调用的工具、调用顺序、输入和输出、延迟以及令牌消耗量；在启用 CloudWatch Transaction Search（一项一次性设置步骤）后，这些数据会直接作为自定义指标和日志发布到 CloudWatch。评估器评分则属于独立的管道：它们来自评估配置，而非执行监测，并写入各自的 CloudWatch 日志组。每次评估运行也会将完整输出保存到 S3 和我们的数据湖，为两个控制面板提供数据：AgentCore Observability, a capability of Amazon Bedrock AgentCore, is the foundation. For agents hosted on the AgentCore runtime, a capability of Amazon Bedrock AgentCore, every execution is instrumented automatically through OpenTelemetry, emitting structured telemetry for every reasoning step: tools called, order, inputs and outputs, latency, and token consumption, published directly to CloudWatch as custom metrics and logs, after CloudWatch Transaction Search is enabled (a one-time setup step). Evaluator scores are a separate pipeline: they come from evaluation configs, not from execution instrumentation, and land in their own CloudWatch log group. Every evaluation run also persists its full outputs to S3 and our data lake, feeding two dashboards:

• 用于查询评估和可观测性结果的对话式仪表板。
  A conversational dashboard to query evaluation and observability results.
• 一个用于趋势分析、记分卡跟踪以及跨智能体比较的 Amazon Quick 仪表板，涵盖完整历史记录。
  An Amazon Quick dashboard for trend analysis, scorecard tracking, and cross-agent comparisons across the full history.

上层是与角色相匹配的视图。值班工程师可以使用以 trace 为先的视图：只需点击一次，就能从 CloudWatch 告警跳转到触发该告警的 AgentCore Observability trace。trace 会显示调用的每个工具、调用顺序、每个步骤的延迟、token 消耗、成本以及完整的推理上下文：从“告警触发”到“找到根本原因”所需的一切信息，无需打开第二个工具，如图 5 所示。Role-appropriate views sit on top. Oncall engineers get a trace-first view: one click from a CloudWatch alarm to the AgentCore Observability trace that triggered it. The trace shows every tool called, in what order, with latency per step, token consumption, cost, and the full reasoning context: everything needed to go from “alarm fired” to “root cause identified” without opening a second tool, as shown in Figure 5.

图 5：Customer Insights Agent 的 AgentCore 执行跟踪，展示了跨度树、轨迹和跨度级指标。图中的数字和数值仅供示例参考。Figure 5: AgentCore execution trace for the Customer Insights Agent, showing the span tree, trajectory, and span-level metrics. The numbers and values in the figure are illustrative.

产品经理可以查看 Amazon Quick 趋势仪表板，按子代理和功能了解质量评分，并进行版本间对比。领导层则可获得一页式健康状况摘要，涵盖全部五个维度，并与每周及跨业务评审保持一致；只需一分钟快速查看，即可回答“我们的智能体表现如何？”Product managers get Amazon Quick trend dashboards: quality scores by sub-agent and feature, release-over-release. Leadership gets a single-page health summary aligned to weekly and cross-business reviews across all five dimensions, a one-minute spot check that answers “how are our agents performing?”

图 6 展示了每周评估仪表板的概览：本周运行的综合通过率（由自定义评估器和 AgentCore 内置评估器共同得出），以及延迟。图中的数字和数值仅供示意。Figure 6 shows the top-line view of our weekly evaluation dashboard: one combined pass rate drawn from both custom evaluators and AgentCore built-in evaluators, alongside latency, for the week’s run. The numbers and values in the figure are illustrative.

图 6：展示综合通过率和延迟的每周评估总览仪表板。图中数字和数值仅为示意。Figure 6: Top-line weekly evaluation dashboard with combined pass rate and latency. The numbers and values in the figure are illustrative.

图 7 从自定义评估器和失败类别两个层面深入分析了同一次运行，准确拆解出哪些错误类型（指标互换、幻觉、客户 ID 遗漏等）造成了差距，以及每种错误影响了多少个问题。图中的数字和值仅供示例说明。Figure 7 drills into that same run at the custom-evaluator, failure-category level, breaking down exactly which error types (metric swaps, hallucinations, customer ID misses, and others) are driving the gap and how many questions each one affects. The numbers and values in the figure are illustrative.

图 7：按类别细分故障情况的每周错误报告仪表板。图中的数字和数值仅供示例。Figure 7: Weekly error-reporting dashboard breaking down failures by category. The numbers and values in the figure are illustrative.

在第 8 阶段之前，平均检测时间（MTTD）为数天至数周。第 8 阶段之后，这一时间缩短至四小时以内，且最先出现的信号是一个附带 AgentCore 跟踪信息的 CloudWatch 警报。Before Stage 8, mean time to detect (MTTD) was days to weeks. After Stage 8, it is under four hours, and the first signal is a CloudWatch alarm with an AgentCore trace attached.

第 9 阶段：反馈循环。持续改进Stage 9: The Feedback Loop. Continuous improvement

这一阶段将观察结果转化为行动。系统会随着时间不断提升质量。我们使用 Amazon Bedrock AgentCore 的一项功能 AgentCore optimization 来形成闭环。This stage turns observations into actions. The system compounds quality over time. We use AgentCore optimization, a capability of Amazon Bedrock AgentCore, to close the loop.

• 系统提示词建议：分析智能体轨迹，并生成数据驱动的优化系统提示词，以提升其在目标评估器上的表现。该方案利用失败模式，并添加具体的行为指令。
  System prompt recommendation: Analyzes agent traces and generates a data driven optimized system prompt that improves performance on the target evaluator. The solution uses failure patterns and adds specific behavioral instructions.
• 工具描述建议：分析智能体执行轨迹，并生成更清晰精准的工具描述，以减少工具选择上的混淆。尤其适用于智能体针对模糊请求选错工具的情况。
  Tool description recommendation: Analyzes agent traces and generates sharpened tool descriptions that reduce tool selection confusion. Particularly when an agent selects the wrong tool for ambiguous requests.

我们来看一个例子：Let’s look at an example:

AgentCore 优化实践 改进方向：目标评估器：幻觉：模型在无关上下文中逐字复现少样本示例里的硬编码值。 原因：少样本示例中包含真实金额和客户名称（例如“$4.2M”“Acme Corp”），模型会记住这些内容，并在回答无关查询时泄露出来。 解决方法：将少样本示例中的硬编码值替换为通用占位符（例如 $[AMOUNT]、[CUSTOMER_NAME]），这样模型就能学会回答结构，而不会记住具体数据。 影响与结果：已实施修复。纯幻觉发生次数从每 10,000 条回答 138 次降至 52 次以下，整体依据充分率从 93% 提高至 96%。

AgentCore optimization in action

What to improve on:
Target evaluator: hallucination: model reproducing hardcoded values from few-shot examples verbatim in unrelated contexts.

Why this happened: Few-shot examples contain real dollar amounts and customer names (for example, “$4.2M”, “Acme Corp”) that the model memorizes and leaks into unrelated queries.

How to fix: Replace hardcoded values in few-shot examples with generic placeholders (for example, $[AMOUNT], [CUSTOMER_NAME]) so the model learns response structure without memorizing specific data points.

Impact and result: Fix applied. Pure hallucination occurrences dropped from 138 to under 52 per 10,000 responses, and overall grounding rate improved from 93 percent to 96 percent.

AgentCore 优化会根据建议，基于真实的智能体执行轨迹和目标评估器，生成数据驱动的系统提示词和工具描述变更。AgentCore 优化基于 AgentCore Evaluations 构建。低分轨迹会被纳入金标准集，作为永久回归防护，确保同一失败不会再次影响用户。AgentCore optimization uses recommendations to produce data driven system prompt and tool descriptions changes based on real agent traces and a target evaluator. AgentCore optimization builds on AgentCore Evaluations . Low-scoring traces are promoted into the golden set as permanent regression guards so each failure does not reach users twice.

智能体评估关键指标一览Agent evaluation key metrics at a glance

评估指标在第 1 阶段定义，在九个阶段中贯彻执行，并在生产环境中持续监控。这套指标并非一成不变。随着我们从生产环境中的智能体获取经验，新的失败模式不断显现，新的维度也变得可衡量，指标随之演进。以下介绍的是我们最初设定的指标，以及当前追踪的指标。The evaluation metrics are defined at Stage 1, enforced across the nine stages, and monitored continuously in production. This set is not static. As we learn from the agent in production, new failure patterns surface, new dimensions become measurable, and the metrics evolve accordingly. What follows is where we started and what we track today.

结论Conclusion

九阶段智能体评估框架的影响具体且可衡量。过去需要数天甚至数周才会暴露、而且通常要等卖家投诉后才会发现的质量回退，现在通过自动化评估可在四小时内捕获。过去依赖主观判断的部署决策，如今要求每位评估器都达到量化门槛。提示词迭代周期从数天缩短到数分钟。安全保障从定期审计转变为对每个请求实现 100% 的护栏覆盖。延迟回退会在 CI 门禁处被捕获，避免进入生产环境。过去要到数周后的账单中才会显现的成本意外，现在会在第二阶段进行评估，并在变更发布前由质量门禁拦截。过去只能含糊地说“智能体流水线里出了点问题”，如今只需点击一次，就能从 CloudWatch 告警跳转到完整的智能体执行追踪，查看每一步推理过程。The impact of the nine-stage agent evaluation framework was concrete and measurable. Quality regressions that previously took days or weeks to surface, and usually arrived through a seller complaint, are now caught in under four hours by automated evaluation. Deployment decisions that once relied on subjective judgment now require every evaluator to pass a quantitative threshold. Prompt iteration cycles fell from days to minutes. Safety moved from periodic audits to 100 percent guardrail coverage on every request. Latency regressions are caught at the CI gate before they reach production. Cost surprises that once appeared weeks later in billing are now evaluated at Stage two and enforced by the quality gate before a change ships. And “something in the agent pipeline” became one click from a CloudWatch alarm to a full agent execution trace with every reasoning step visible.

更广泛的教训，业界早已领会过。DevOps 告诉我们，不进行自动化测试就发布代码是鲁莽的。MLOps 告诉我们，不进行监控就部署模型是危险的。AgentOps 对 AI 智能体也传达了同样的道理：在没有持续、多层级评估的情况下运行它们，就如同盲目飞行。The broader lesson is one the industry has learned before. DevOps taught us that shipping code without automated testing is reckless. MLOps taught us that deploying models without monitoring is dangerous. AgentOps teaches us the same thing about AI agents: operating them without continuous, multi-level evaluation is flying blind.

这里描述的九阶段框架并非销售情报代理所独有。部署多智能体系统的团队——这类系统会进行推理、调用子代理和工具，并综合生成响应——在整个生命周期中都会面临同样的问题。你的失败分类体系会有所不同；你的黄金评测集会体现你的领域特点；你的评估器栈会针对你的用户进行调优。始终不变的是这一原则：评估不是最后一道质量保证工序，而是代理本身的操作系统；它会持续接受衡量、自动防护，并在每次交互中不断改进。The nine-stage framework described here is not specific to sales intelligence agents. Team deploying multi-agent systems that reason, call sub-agents, tools, and synthesize responses will face these same questions across the same lifecycle. Your failure taxonomy will look different. Your golden set will reflect your domain. Your evaluator stack will be tuned to your users. What stays constant is the principle: evaluation is not a final QA step. It is the operating system of the agent itself, continuously measured, automatically defended, and improved with every interaction.

如果你正在构建生产级智能体，并希望将这种评估规范引入系统，Amazon Bedrock AgentCore Evaluations 可为你提供实现这一目标的平台：按需评估实验、用于持续监控的批量评估、针对实时流量的在线评估、内置评估器和自定义评估器、数据集评估（公开预览版）、CloudWatch 集成，以及 AgentCore 优化功能——建议和 A/B 测试，帮助形成反馈闭环。从第一阶段开始，定义智能体的优秀标准，其余工作交给该框架完成。If you are building production agents and want to bring this kind of evaluation discipline to your system, Amazon Bedrock AgentCore Evaluations gives you the platform to do it: on-demand evaluation for experiments, batch evaluation for continuous monitoring, online evaluation for live traffic, built-in and customized evaluators, dataset evaluation(in public preview), CloudWatch integration, and the AgentCore optimization features recommendations and A/B tests to close the feedback loop. Start with Stage one, define what good looks like for your agent, and let the framework do the rest.

准备好开始了吗？请参阅 Amazon Bedrock AgentCore 文档中的“评估代理性能”。如果你正在构建生产级代理，并希望探讨如何将此框架应用于你的系统，我们很乐意提供帮助。请直接联系我们，或通过你的 AWS 客户团队联系。Ready to start? See Evaluate agent performance in the Amazon Bedrock AgentCore documentation. And if you’re building production agents and want to talk through how to apply this framework to your system, we’re happy to help. Reach out to us directly or through your AWS account team.

要进一步了解 Amazon Bedrock AgentCore Evaluations，请参阅 AWS 文档中的《评估代理性能》。如需技术介绍，请阅读《使用 AgentCore Evaluations 构建可靠的 AI 代理》。如需了解我们如何构建 Performance Insights Agent，请阅读《AWS SMGS 如何利用 AI 驱动的对话助手，通过 Amazon Bedrock AgentCore 转变业务管理方式》。如需了解 AgentCore 的推荐与优化，请阅读《AgentCore 优化：通过推荐和 A/B 测试改进代理质量闭环》To learn more about Amazon Bedrock AgentCore Evaluations, see Evaluate agent performance in the AWS documentation. For a technical introduction, read Build reliable AI agents with AgentCore Evaluations. To see how we built the Performance Insights Agent, read How AWS SMGS uses an AI-powered conversational assistant to transform business management with Amazon Bedrock AgentCore. To learn about AgentCore recommendation and optimization, read AgentCore optimization: Improve agent quality loop with recommendations and A/B tests​

致谢Acknowledgement

我们向以下高管赞助人和导师致以诚挚谢意，正是他们的愿景与指导让这一计划得以实现：Damien Forthomme，应用科学部三级经理；Akhand Singh，数据工程负责人；Sujit Narapareddy，AWS Insights 总监；Monica Jain，技术高级产品经理；Norman Braddock，AWS AI 产品管理高级经理；以及 Muhammad Umar Javed，三级应用科学家，L6。We extend our sincere gratitude to our executive sponsors and mentors whose vision and guidance made this initiative possible: Damien Forthomme, Manager III, Applied Science; Akhand Singh, Head of Data Engineering; Sujit Narapareddy, Director AWS Insights; Monica Jain, Sr. Product Manager – Tech and Norman Braddock, Senior Manager of AI Product Management at AWS; Muhammad Umar Javed, Applied Scientist III, L6.

关于作者About the authors

李睿Rui Li

Rui 是 AWS 销售、营销和全球服务部门（SMGS）的一名应用科学家。她负责推动相关科学研究，让 AI 智能体值得信赖，使 AWS 销售人员、领导者和客户能够放心依赖它们。她的工作比较不同的智能体评估方法和优化方法，并将其转化为可扩展、可复用的评估框架，用于生产环境中的负责任多智能体 AI。AWS 各团队都可以采用这一框架，从头到尾检查智能体的质量，放心发布更新，并在客户发现问题之前及时发现问题。通过将评估与持续改进相结合，她的工作让 AWS AI 智能体随着时间推移变得更准确、更可靠，并增强客户对 AWS 作为生产级智能体式 AI 平台的信任。Rui is an Applied Scientist in AWS Sales, Marketing, and Global Services (SMGS). She leads the science behind making AI agents trustworthy, so AWS sellers, leaders and customers can rely on them with confidence. Her work compares agent evaluation approaches and optimization methods, and turns them into a scalable, reusable evaluation framework for responsible multi-agent AI in production. Teams across AWS can adopt it to check agent quality from start to finish, ship updates with confidence, and catch problems before customers notice. By linking evaluation to continuous improvement, her work makes AWS AI agents more accurate and dependable over time, and builds customer trust in AWS as the platform for production-grade agentic AI.

钱达娜·斯里尼瓦斯Chandana Srinivas

Chandana 是 AWS 销售、市场营销和全球服务（SMGS）部门的高级产品经理。她负责制定并推动智能体 AI 产品的战略与交付，为 AWS 销售人员和领导者大规模提供值得信赖的客户洞察。她负责的产品组合包括基于知识图谱和自然语言查询系统构建的客户洞察智能体。她还负责制定这些产品的智能体评估策略。Chandana is a Senior Product Manager in AWS Sales, Marketing, and Global Services (SMGS). She leads the strategy and delivery of agentic AI products that give AWS sellers and leaders trusted customer intelligence at scale. Her portfolio includes customer intelligence agents built on knowledge graphs and natural-language query systems. She also drives the agent evaluation strategy for these products.

米希尔·加德吉尔Mihir Gadgil

Mihir 是 AWS 销售、营销和全球服务部门（SMGS）的高级数据工程师，专注于企业级数据解决方案和生成式 AI 应用。他拥有超过 9 年的经验，并获得信息技术与管理硕士学位，主要致力于构建稳健的数据管道、复杂的数据建模以及 ETL/ELT 流程。他凭借专业知识，通过创新的数据工程解决方案和先进的分析能力推动业务转型。Mihir is a Senior Data Engineer in AWS Sales, Marketing, and Global Services (SMGS), specializing in enterprise-scale data solutions and generative AI applications. With over 9 years of experience and a Master’s in Information Technology & Management, he focuses on building robust data pipelines, complex data modeling, and ETL/ELT processes. His expertise drives business transformation through innovative data engineering solutions and advanced analytics capabilities.

阿洛克·辛格Alok Singh

Alok 是 AWS 的高级软件开发工程师，专注于构建可扩展的基础设施、Web 服务和 AI/ML 解决方案。他拥有超过 12 年的经验，设计并构建了多套生产级系统，包括基于 Amazon SageMaker 的端到端 ML 平台，使数据科学家能够在一周内将模型部署到生产环境；面向 AWS 销售团队的智能体 AI 解决方案；以及一个大规模计费系统，为全球 AWS 客户处理数十亿条事件。他的专长涵盖分布式系统、云架构和生成式 AI，致力于通过创新的工程解决方案实现可衡量的业务影响。Alok is a Senior Software Development Engineer at AWS, specializing in building scalable infrastructure, web services, and AI/ML solutions. With over 12+ years of experience, he has architected production-grade systems including an end-to-end ML platform on Amazon SageMaker that enables data scientists to deploy models to production within a week, agentic AI solutions for AWS sales teams, and a high-scale billing system processing billions of events for AWS customers worldwide. His expertise spans distributed systems, cloud architecture, and generative AI, with a focus on delivering measurable business impact through innovative engineering solutions.

马杜·萨米塔·万加拉Madhu Samhitha Vangara

Madhu 是 AWS 的全球生成式 AI 专家解决方案架构师，专注于 Amazon Bedrock AgentCore 和 Strands Agents 的智能体 AI 市场推广。她深入了解企业业务价值，曾在 Juniper Networks、VMware、Barclays 和 IGCAR 积累行业经验。她致力于将新兴 AI 能力和研究转化为客户可衡量的成果。她曾在 AWS re:Invent、NVIDIA GTC 和 AI Summit 等 AI 会议上发表演讲，专长领域包括智能体优化、智能体工作负载、合作伙伴网络以及生产级智能体 AI。她拥有马萨诸塞大学阿默斯特分校的计算机科学硕士学位。工作之余，她接受过印度古典舞训练，也是一位艺术爱好者。Madhu is a Worldwide Generative AI Specialist Solution Architect at AWS, focusing on agentic AI go-to-market for Amazon Bedrock AgentCore and Strands Agents. She brings a deep understanding of enterprise business value, with previous industry experience at Juniper Networks, VMware, Barclays, and IGCAR. She translates emerging AI capabilities and research into measurable outcomes for customers. She is a speaker at AI conferences such as AWS re:Invent, NVIDIA GTC, and AI Summit, where she specializes in agent optimization, agentic workloads , partner network, and production-grade agentic AI. She holds a master’s in computer science from UMass Amherst. Outside work, she’s a trained Indian classical dancer and an art enthusiast.

巴拉蒂·斯里尼瓦桑Bharathi Srinivasan

Bharathi 是 AWS 的高级生成式 AI 数据科学家。Bharathi 与企业客户合作，解决大规模生成式 AI 挑战，包括非确定性系统的鲁棒性与验证、生成式 AI 和智能体 AI 平台的治理，以及动态智能体 AI 系统的质量。Bharathi is a Senior Generative AI Data Scientist at AWS. Bharathi works with enterprise customers on large‑scale generative AI challenges, including robustness and verification of non‑deterministic systems, governance of GenAI and agentic AI platforms, and the quality of dynamic agentic AI systems.