构建可靠的数据分析代理：KDD Cup 的经验

NVIDIA KGMON 团队在 KDD Cup 2026 Data Agents 竞赛中获得第二名，他们的系统围绕一个简单理念构建：让智能体的工具框架更精简、更清晰，也更易于验证。The NVIDIA KGMON team placed second in the KDD Cup 2026 Data Agents competition with a system built around a simple idea of making an agent’s harness smaller, clearer, and easier to verify.

这项竞赛要求智能体针对异构数据源回答自然语言问题，数据源包括数据库、CSV 和 JSON 文件、散文文档、PDF 以及简报视频。每项任务都不只是检索：智能体还需要检查可用数据、选择合适的工具、跨来源推理、生成最终答案文件，并处理分析工作流中出现的陷阱。The competition asked agents to answer natural-language questions over heterogeneous data sources, including databases, CSV and JSON files, prose documents, PDFs, and briefing videos. Every task required more than retrieval, with the agent inspecting available data, choosing the right tools, reasoning across sources, producing a final answer file, and handling traps that appear in analytical workflows.

KDD 还要求各团队使用一个小型、固定的 LLM 为智能体提供支持，因此 harness 成了主要的优化空间。团队旨在通过预处理、受限工具、持久化状态和评估，让现有模型更容易完成任务。在围绕较小型开放模型构建可靠系统时，这些技术尤其有用。KDD also required teams to work with a small, fixed LLM to power the agent, making the harness the main optimization surface. The team aimed to make the task easier for the available model through preprocessing, constrained tools, persistent state, and evaluation. These techniques are especially useful when building reliable systems around smaller open models.

这篇文章分享了他们取得这一成果背后的实战方法。它并不是适用于所有数据科学智能体的通用方案，但其中更广泛的经验适用于许多智能体系统：可靠性往往并非来自让模型拥有更大的自由度，而是来自围绕模型搭建恰当的支撑框架。This post shares the playbook behind their result. It isn’t a general recipe for every data science agent, but the broader lesson applies to many agent systems. Reliability often comes less from making the model more open-ended and more from building the right harness around it.

该系统背后的两项原则Two principles behind the system

这套系统由两项原则塑造而成。Two principles shaped the system.

首先，限制操作空间。检查数据、调用工具、写入文件或从错误中恢复的方式过多，可能导致失败。KGMON 统一了数据访问方式，提供了一小组工具，并要求最终答案遵循单一输出路径。First, constrain the action space. Too many ways to inspect data, call tools, write files, or recover from mistakes can lead to failures. KGMON unified data access, exposed a small tool set, and required final answers to follow a single output path.

其次，确保每次尝试都可供检查。一次运行可能因工具调用错误、联接不正确、遗漏文档规则或答案格式问题而失败。执行轨迹、反复尝试和轨迹检查帮助团队识别失败原因并改进测试框架。图 1 展示了整体工作流程。Second, make every attempt inspectable. A run can fail because of a bad tool call, an incorrect join, a missed document rule, or an answer-format issue. Execution traces, repeated attempts, and trajectory inspection helped the team identify failures and refine the harness. Figure 1 shows the overall workflow.

以下技术为构建可靠的数据分析代理提供了实用指导。The following techniques offer practical guidance for building reliable data analytics agents.

1. 将结构化来源转换为统一的查询界面1. Convert structured sources into one query surface

每个 KDD 任务都可能涉及 SQL 数据库、CSV 和 JSON 文件、文档及视频。KGMON 会将 CSV 和 JSON 文件转换为现有 SQLite 数据库中的表，使智能体能够通过统一的 SQL 接口访问所有结构化数据。Each KDD task could combine SQL databases, CSV and JSON files, documents, and videos. KGMON converted CSV and JSON files into tables in the existing SQLite database, giving the agent one SQL interface for all structured data.

KGMON 的自定义持久化 Python 环境提供了两个内置函数，用于检查和查询该统一数据面：KGMON’s custom, persistent Python environment exposed two built-in functions for inspecting and querying that unified data surface:

schema()
sql(query)

这些自定义函数为智能体提供了统一的方式来发现和查询结构化数据。它们内置于 KGMON 的环境中，而非由 Python 或 SQLite 提供。These custom functions gave the agent a single way to discover and query structured data. They were built into KGMON’s environment rather than provided by Python or SQLite.

统一的 SQL 接口减少了路由失败和无效交互，让固定模型有更多空间对数据进行推理。The single SQL interface reduced routing failures and wasted turns, leaving the fixed model more room to reason over the data.

采用这种方法：在智能体启动之前，统一结构化数据的访问方式。临时 SQLite 数据库、虚拟化查询层或受治理的数据仓库接口，都可以提供稳定、精简且文档完善的查询界面。Apply this approach: Normalize access to structured data before the agent starts. A temporary SQLite database, a virtualized query layer, or a governed warehouse interface can provide a stable, narrow, and well-documented query surface.

2. 一开始就向智能体提供 schema 上下文2. Seed the agent with schema context up front

KGMON 在主要推理循环之前增加了一个模式探查步骤。系统会检查表、列、可能的连接键、重复名称、相似字段、单位、空值模式以及行粒度问题。KGMON added a schema-scouting step before the main reasoning loop. The system inspected tables, columns, possible join keys, duplicate names, look-alike fields, units, null patterns, and row-grain issues.

智能体在每项任务开始时都会收到此架构上下文，从而省去了前期的一轮探索。The agent received this schema context at the start of each task, saving an early discovery turn.

模式探查减少了因列选错、遗漏连接或混淆每个答案行应表示什么而导致的错误。图 2 展示了统一的 SQL 接口以及提供给代理的模式上下文。Schema scouting reduced errors caused by wrong columns, missed joins, or confusion about what each answer row should represent. Figure 2 shows the unified SQL interface and the schema context provided to the agent.

采用这种方法：添加一个只读的预检步骤，向代理简要介绍可用的数据表、可能的连接关系、关键列、单位、可疑字段以及已知的歧义。这可能比再增加一个检索工具更有用。Apply this approach: Add a read-only preflight step that briefs the agent on available tables, likely joins, key columns, units, suspicious fields, and known ambiguities. This can be more useful than another retrieval tool.

3. 构建一个小巧且有明确设计取向的工具运行框架3. Build a small, opinionated tool harness

KGMON 将代理限制为只能使用用于检查模式、执行 SQL 查询、查找文档、提取文本和撰写答案的辅助工具。KGMON limited the agent to helpers for schema inspection, SQL queries, document lookup, prose extraction, and answer writing.

该环境公开了以下函数：The environment exposed the following functions:

schema()       	# inspect tables and columns
sql(query)     	# query context.db
write_answer(df)   # write the final answer atomically
prose_helper() 	# answer from prose or extract prose into SQL

中间件会修复格式错误的工具调用，避免一次错误调用导致尝试中止。Middleware repaired malformed tool calls so one bad call did not end an attempt.

有状态的 Python 环境会在工具调用之间保留变量，让智能体能够复用中间结果。预定义函数（如 schema()、sql(query) 和 write_answer(df)）减少了样板代码、语法错误和文件处理失误。这节省了交互轮次，也让小型 LLM 能专注于分析。图 3 展示了该环境的函数、持久状态和错误处理机制。A stateful Python environment retained variables between tool calls, letting the agent reuse intermediate results. Predefined functions such as schema(), sql(query), and write_answer(df) reduced boilerplate, syntax errors, and file-handling mistakes. This saved turns and kept the small LLM focused on analysis. Figure 3 shows the environment’s functions, persistent state, and error handling.

简短且有效的尝试能腾出更多轮次，用于额外运行、评估和集成（结合多次尝试的结果）。Short, valid attempts left more turns for additional runs, evaluation, and ensembling—combining results from multiple attempts.

采用这种方法：围绕工作流需求设计工具，并移除重复路径。统一采用一种经过批准的 SQL 执行或答案编写方式，可以减少破坏状态或生成无效输出的机会。Apply this approach: Design tools around workflow requirements and remove duplicate paths. A single approved way to run SQL or write an answer reduces opportunities to corrupt state or produce invalid outputs.

4. 将散文视为一等输入，但要与结构化数据分开处理4. Treat prose as a first-class input, but keep it separate from structured data

分析工作流通常包含 PDF、Markdown 文件、文档、政策文本、说明和报告。在某些 KDD 任务中，这些来源包含回答问题所需的阈值、规则、定义和类似表格的记录。Analytics workflows often include PDFs, Markdown files, documents, policy text, instructions, and reports. In some KDD tasks, these sources contained thresholds, rules, definitions, and table-like records needed to answer questions.

大型文档可能会占用 LLM 的上下文窗口。KGMON 禁止通过 Python 的 open() 函数或 .read() 方法直接读取整个文件，而是提供了按字符数或正则表达式（regex）匹配来限制预览和搜索范围的工具。Large documents can consume an LLM’s context window. KGMON blocked direct reads of entire files through Python’s open() function or .read() method, providing tools that limited previews and searches by character count or regular expression (regex) matches.

找到相关章节后，智能体可以调用 prose_helper，这是一个自定义工具，会将文档片段传递给另一个温度设为 0 且禁用推理的 LLM 调用。它会返回答案或提取出的表格，避免原始文档内容进入主智能体的上下文。After finding a relevant section, the agent could invoke prose_helper, a custom tool that passed document chunks to a separate LLM call with temperature set to 0 and reasoning disabled. It returned answers or extracted tables, keeping raw document contents out of the main agent’s context.

Figure 4 shows the document-inspection workflow. KGMON used prose_helper in two modes:
mode="answer"  # extract a rule, threshold, or short answer
mode="table"   # extract repeating records into a SQL table

从散文中提取规则，让智能体在 SQL 分析中应用这些规则，同时保持工作上下文聚焦。Extracting rules from prose let the agent apply them in SQL analysis while keeping its working context focused.

重要提示：表格提取适用于将结构化信息嵌入文档中的竞赛任务。生产系统可能只需要针对性地查找正文内容；表格提取可以作为可选功能。Important caveat: Table extraction suited competition tasks that embedded structured information in documents. Production systems may need only targeted prose lookup; table extraction can remain optional.

采用这种方法：提供一个文档检查工具，用于回答有针对性的问题并标注出处，而无需将整份文档加载到主代理的上下文中。当文档包含需要进行连接或筛选的重复记录时，添加表格提取功能。Apply this approach: Provide a document-inspection tool that answers targeted questions and cites sources without loading entire documents into the main agent’s context. Add table extraction when documents contain repeated records that need joins or filters.

5. 当视频属于任务的一部分时，预处理视频5. Preprocess video when it is part of the task

一些 KDD 任务包含简报视频。为避免在智能体循环中处理视频所带来的计算开销，KGMON 提取了关键帧、转录了音频、将转录片段与画面对齐，并将生成的证据提供给智能体。Some KDD tasks included briefing videos. To avoid the compute cost of processing video inside the agent loop, KGMON extracted keyframes, transcribed audio, aligned transcript segments with frames, and supplied the resulting evidence to the agent.

每项任务最多包含一段视频，其中通常包含基于幻灯片的限制条件或干扰值。与转录内容对齐的关键帧将口述上下文与正确的视觉证据联系起来。图 5 展示了预处理步骤。Each task included at most one video, often containing slide-based constraints or distractor values. Transcript-aligned keyframes connected spoken context to the correct visual evidence. Figure 5 shows the preprocessing steps.

重要注意事项：预处理适合这次竞赛的配置。对于包含大量视频的应用，按需调用辅助函数可能更合适，类似于 prose_helper。Important caveat: Preprocessing suited the competition setup. Applications with many videos may benefit from an on-demand helper, similar to prose_helper.

采用这种方法：在智能体循环开始前，预处理数量有限的视频。对于规模更大的视频集合，提供检索或检查工具，以便按需查询视频证据。Apply this approach: Preprocess a limited set of videos before the agent loop. For larger collections, provide a retrieval or inspection tool that queries video evidence on demand.

6. 记录日志跟踪信息，以便其他智能体或人工人员检查故障。6. Log traces so another agent or a human can inspect failures

每次尝试都会记录提示词、工具调用、SQL 查询、中间结果、错误、修复操作、文档查找和最终答案。Each attempt logged prompts, tool calls, SQL queries, intermediate results, errors, repairs, document lookups, and final answers.

一个专用的检查代理可以审查失败的执行轨迹，对错误进行分类，并找出反复出现的故障，帮助团队确定改进工作的优先级。A specialized inspector agent could review failed trajectories, categorize mistakes, and surface recurring failures to help the team prioritize improvements.

追踪记录显示，错误答案究竟是源于对架构的混淆、错误的联接、遗漏了正文证据、输出格式问题，还是脆弱的提示规则。Traces showed whether a wrong answer came from schema confusion, an incorrect join, missed prose evidence, output formatting, or a brittle prompt rule.

采用这种方法：将轨迹检查纳入开发工作流。子代理或评估脚本可以对近期运行中的失败进行分类，并建议调整测试框架。图 6 展示了轨迹如何揭示第一个错误决策。Apply this approach: Build trace inspection into the development workflow. A subagent or evaluation script can categorize failures from recent runs and recommend harness changes. Figure 6 shows how a trace reveals the first wrong decision.

7. 评估多次尝试，但要注意成本7. Evaluate multiple attempts, but watch the cost

KGMON 通过多次尝试和答案选择提升了覆盖率和可靠性。它按答案值而非列名对尝试进行分组，并可为存在争议的任务增加运行次数。KGMON improved coverage and reliability through repeated attempts and answer selection. It grouped attempts by answer values rather than column names and could give contested tasks additional runs.

多次尝试有助于在排行榜精确到数值的评分规则下，区分稳定答案和偶然失误。Multiple attempts helped distinguish stable answers from one-off mistakes under the leaderboard’s exact value-level scoring.

重要注意事项：反复尝试会增加令牌用量、延迟和计算成本。在生产环境中，应仅在任务的价值、不确定性或风险足以证明其成本合理时，才采用集成方法。Important caveat: Repeated attempts increase token usage, latency, and compute cost. In production, reserve ensembling for tasks whose value, uncertainty, or risk justifies the cost.

采用这种方法：先进行单次运行评估并检查轨迹，再根据置信度、意见分歧或验证失败情况，判断何时值得为额外尝试付出成本。Apply this approach: Start with single-run evaluation and trace inspection. Use confidence, disagreement, or validation failures to decide when additional attempts justify their cost.

8. 谨慎使用改进循环8. Use improvement loops carefully

在自主改进循环中，智能体会利用评估反馈来优化提示词、工具、后处理流程和评估逻辑。这些改动可能提升性能，但也可能导致对基准测试过拟合。In autonomous improvement loops, an agent uses evaluation feedback to refine prompts, tools, postprocessing, and evaluation logic. These changes can improve performance but also overfit to the benchmark.

KGMON 发现了若干风险：KGMON saw several risks:

• 将训练示例硬编码到提示词中
  Hardcoding training examples into prompts
• 不断累积相互矛盾的指令
  Accumulating contradictory instructions
• 添加脆弱的后处理规则
  Adding brittle postprocessing rules
• 改进一个基准数据集划分，同时损害泛化能力
  Improving one benchmark split while hurting generalization

团队需要快速改进，但不能构建一个只会记住基准测试的测试框架。The team needed rapid improvements without building a harness that memorized the benchmark.

采用这种方法：在推广变更之前，要求使用留出任务、进行提示词审计和轨迹审查，并获得人工批准。在改进闭环中设置审核关卡。Apply this approach: Require held-out tasks, prompt audits, trace review, and human approval before promoting changes. Build a review gate into the improvement loop.

9. 让人类参与到流程中恰当的环节9. Keep humans in the right part of the loop

团队将人类指导与智能体规模的实验相结合。The team combined human guidance with experimentation at agent scale.

人们定义了任务要求、审查了轨迹、指导了智能体早期的行为、拒绝了脆弱的改动，并选出要保留在测试框架中的改进。People defined the task requirements, audited traces, guided early agent behavior, rejected brittle changes, and selected improvements to retain in the harness.

有针对性的人为干预可以改善未来的运行效果，而无需监督每一次工具调用。Targeted human interventions could improve future runs without requiring oversight of every tool call.

采用这种方法：审查任务设计、评估标准、失败分析以及拟议的可复用技能。让智能体执行并探索，由人来决定保留哪些改进。图 7 展示了人工审查在这一循环中的位置。Apply this approach: Review task design, evaluation criteria, failure analysis, and proposed reusable skills. Let agents execute and explore, while people decide which improvements to retain. Figure 7 shows where human review fits into the loop.

构建数据科学智能体的核心经验教训Top lessons for building a data science agent

KDD 的具体解决方案是由基准测试条件塑造的：模型固定、无法访问互联网、任务包异构、按值评分，以及长期运行预算。并非每项设计选择都应直接照搬到生产环境中。The exact KDD solution was shaped by the benchmark: a fixed model, no internet access, heterogeneous task bundles, value-level scoring, and a long-run budget. Not every design choice should be copied directly into production.

你可以将这些实践应用于其他代理系统：You can apply these practices to other agent systems:

• 在代理启动前规范化数据访问。
  Normalize data access before the agent starts.
• 为代理提供一组精简且可靠的工具。
  Give the agent a small, reliable tool surface.
• 确保每次尝试都可追溯。
  Make every attempt traceable.
• 评估两个答案及其轨迹。
  Evaluate both answers and trajectories.
• 有选择地进行重复尝试。
  Use repeated attempts selectively.
• 保留可复用的知识。
  Preserve reusable knowledge.
• 仅在验证后推广改进。
  Promote improvements only after validation.

这些做法有助于代理可靠地使用数据。These practices help an agent use data reliably.

开始使用Getting started

从一个可重复的分析工作流开始，构建能够完成该工作流的最小测试框架。按以下步骤操作：Start with one repeatable analytics workflow and build the smallest harness that can complete it. Follow these steps:

• 定义问题和所需的答案格式。
  Define the question and required answer format.
• 为代理提供一组范围有限的工具，用于检查和查询数据。
  Give the agent a narrow set of tools for inspecting and querying data.
• 记录每次工具调用和中间结果。
  Record every tool call and intermediate result.
• 创建一个小型评估集，其中既包含成功案例，也包含可能的失败模式。
  Create a small evaluation set that includes both successful cases and likely failure modes.
• 审查代理的执行轨迹，然后改进其工具、提示词和验证检查。
  Review the agent’s trajectories, then refine its tools, prompts, and validation checks.

基础可靠后，添加文档检查、共享知识、经过审核的改进，以及对多次尝试进行有选择的评估。Once the foundation is reliable, add document inspection, shared knowledge, reviewed improvements, and selective evaluation of multiple attempts.

如需查看更多架构示例和实现思路，请浏览 KDD Cup Data Agents 演示文稿档案，其中收录了八支入选团队的演示录像和幻灯片。这些材料旨在作为设计参考，而非逐步复现 KGMON 解决方案的指南。For additional architecture examples and implementation ideas, explore the KDD Cup Data Agents presentation archive, which includes the recorded presentation session and slides from eight featured teams. These materials are intended as design references rather than a step-by-step reimplementation of KGMON’s solution.

智能体的可靠性取决于模型及其运行框架。两者必须共同支持可检查、可重复且有用的分析。Agent reliability depends on the model and its harness. Together, they must support analysis that is inspectable, repeatable, and useful.