使用 NVIDIA NeMo Gym 优化芯片设计的代理可靠性和工具调用效率
原题Optimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo Gym
读前速览
NVIDIA NeMo Gym 支持对芯片设计 AI 代理进行系统化评估，通过验证最终的 Liberty 工件并优化 MCP 工具描述，将 Siemens Fuse EDA 代理的可靠性从 55% 提升至 95%。
NeMo Gym 通过验证最终工件（而非中间步骤），支持对多阶段芯片设计工作流程中的 AI 代理进行系统化评估。
该环境定义了任务、工具、状态，并包含一个领域专用验证器，用于将最终的 Liberty 文件与参考工件进行比较。这种端到端评估可以发现中间工具调用检查所遗漏的微妙错误，例如丢失的单元、错误的斜率阈值以及超出容差范围的时序值。
一个带有硬栅门（覆盖范围、有效性、相对验证、完整性）和连续准确性度量的综合奖励函数，为代理可靠性提供了严格且可重复的基准。
奖励在任何栅门失败时为零，阻止代理通过复用或截断结果来“作弊”。准确性组件衡量时序和功率值在给定容差范围内的比例，从而可以对回滚结果进行精细比较。
基于轨迹分析，对代理技能和 MCP 工具描述进行微调，显著提高可靠性和效率。
通过简化重叠的文件/shell 接口，并使用显式的 JSON 模式定义工具输入，团队减少了歧义指令和错误的工具选择。这导致综合分数从 ~55% 上升到 ~95%，回合数减少 66%，输出令牌减少 64%。
同样的方法适用于 EDA、CAE 等领域中，用于可靠生成工程工件的代理。
核心原则包括评估最终工件、编写可以检测丢失或变形结果的领域专用验证器，以及通过反复迭代优化技能和工具接口，以收集大量的回滚数据。
完整摘要
本文详细介绍了 NVIDIA NeMo Gym 如何用于评估和改进 Siemens Fuse EDA AI 代理，用于标准单元表征。该代理协调 229 个单元和 36 个 PVT 角色的 SPICE 模拟，生成 Liberty 时序模型。NeMo Gym 提供了一个带有领域专用验证器的受控评估循环，该验证器检查最终工件的覆盖范围、有效性、相对验证、准确性和完整性。通过分析回滚轨迹，工程师们完善了代理的技能并改进了 MCP 工具描述，简化了重叠的接口并添加了显式的 JSON 模式。这种优化将综合验证分数从 ~55% 提升到 ~95%，同时将代理回合数减少 66%（从 155 到 53），并减少输出令牌 64%（从 ~81K 到 ~29K）。该方法强调评估最终工程工件而不是单个工具调用，并适用于 EDA、CAE 和其他工程领域。
目录 · 6 节
AI agents offer remarkable productivity gains in complex chip design workflows, but their tendency to generate plausible yet incorrect artifacts makes reliability a major barrier to production adoption. In multistage engineering workflows, an undetected error early in setup can silently propagate through downstream stages and final artifact generation. Evaluating and refining these agents against verified final outputs, rather than isolated intermediate steps, is essential for building trustworthy automation.
NVIDIA NeMo Gym, an open library for evaluating and improving models and agents using environments, addresses this challenge. The NeMo Gym agentic evaluation environment is designed to diagnose failure modes and track error propagation across complex, multistage workflows. By establishing a domain-specific verifier to inspect final outputs against verified reference artifacts, NeMo Gym enables engineers to trace complete rollout trajectories and identify root causes behind subtle errors.
This post walks through how NeMo Gym was used to evaluate and improve the reliability of the Siemens Fuse EDA AI Agent when calling characterization tools. The Siemens Fuse team used NeMo Gym to scale up evaluations and verify the resulting Liberty (.lib) files against known references and tracing—both common and rare failures through each rollout. Those findings guided changes to the agent’s skill and Model Context Protocol (MCP) tool descriptions, increasing the measured benchmark composite score from approximately 55% to ~95% across a benchmark of representative standard-cell characterization tasks, while reducing turns and output tokens.
Siemens’ work provides a practical example for chip design teams and EDA developers on how to leverage the NVIDIA AI software stack and open libraries to improve agent reliability and chip design workflows.
Automating standard-cell characterization
Standard-cell characterization is the process of generating accurate timing, power, and noise models for logic cells across operating conditions. It is foundational to chip design and required whenever a semiconductor foundry issues a new process design kit (PDK). Each new PDK carries SPICE transistor models; design rules; and process, voltage, and temperature (PVT) definitions that connect the manufacturing process to design tools. When a foundry updates a PDK, existing characterization datasets may no longer accurately describe the updated devices. Timing and power models must then be regenerated and validated for every affected cell across combinations of input slew, output load, and PVT conditions.
The Solido Characterization Suite agent automates this process. The agent orchestrates tools from the broader Solido Characterization Suite through Siemens Fuse EDA AI Agent, which exposes MCP servers and skills to common coding harnesses including Claude Code, Codex, and Cursor.
MCP interfaces make these accelerated tools accessible to engineering agents, while skills guide their use in characterization workflows. Faster simulation helps agents complete engineering iterations sooner, while NVIDIA NeMo Gym helps developers measure and improve how reliably and efficiently agents orchestrate those tools.
The Solido Characterization Suite agent must perform tool setup, large-scale SPICE simulation, distributed compute, Liberty file generation, debugging, and final validation (Figure 1). Errors often produce plausible outputs instead of clean failures, leaving incorrect thresholds, missing corners, or incomplete cells hidden. Because these stages depended on one another, an early setup error could propagate into every step that followed. Defining a rigorous evaluation protocol was a key step in preparing the Solido Characterization Suite agent for production workflows.
Measuring and improving agent reliability with NeMo Gym
Part of the NVIDIA NeMo collection of open libraries for agent specialization and optimization, NeMo Gym integrates with the standard-cell characterization flow inside a controlled evaluation loop (Figure 2). It defines the task, tools, state, and verifier for each rollout so that attempts started from a clean state and produced comparable results. It also records thinking traces, input and output messages, and tool actions.
This enabled engineers to run the following optimization loop:
- Run baseline rollouts with a versioned task and verifier in a custom NeMo Gym environment
- Inspect trajectories, failed tool calls, and verifier diagnostics
- Identify ambiguous instructions, unproductive actions, and incorrect tool selection
- Refine the agent’s skills and MCP tool descriptions
- Repeat the same tasks under the same evaluation conditions
- Compare verified outcomes and efficiency per successful result
Reward design and artifact verification
As the team analyzed agent trajectories, end-to-end evaluation uncovered failures that intermediate checks missed. For example, final Liberty verification exposed missing cells and operating conditions, incorrect slew thresholds, and timing or power values outside tolerance. These findings show why engineering agents must be evaluated against final artifacts rather than individual tool calls.
To make the work repeatable, the team implemented verifiers that provide a final composite reward score based on the following metrics:
- Coverage: Whether the agent produced every expected cell and PVT corner.
- Validity: Whether the delivered Liberty files are well-formed and usable by the validation tools.
- Relative validation: Whether the output introduced failures beyond those already present in the reference.
- Accuracy: The fraction of expected timing and power values that falls within the defined tolerance.
- Integrity: Whether the characterization workflow produced the complete result rather than reusing, truncating, or reformatting existing values.
The reward score is defined as:
\(R=\mathbf{1}\left[\text{Coverage}\right]\times\mathbf{1}\left[\text{Validity}\right]\times\mathbf{1}\left[\text{Relative Validation}\right]\times\mathbf{1}\left[\text{Integrity}\right]\times\frac{X_{\text{tol}}}{X_{\text{total}}}\)
Here, \(X_{\mathrm{tol}}\) is the number of values in the output that are within tolerance, and \(X_{\mathrm{total}}\) is the total number of output values. Their ratio represents the fraction of output values within tolerance.
Each indicator \(\mathbf{1}[\cdot]\) equals \(1\) when its corresponding gate passes and \(0\) when it fails. The reward is zero if any gate fails.
Using this verifier, the team systematically evaluated and tuned the agent’s performance across multiple iterations.
The team used the open source GF180MCU PDK to benchmark the Solido Characterization Suite agent on Liberty-file generation. The dataset includes 229 standard cells across 36 PVT corners, using Solido AFS for SPICE simulation.
A custom agentic environment was built using NeMo Gym. For each run, the characterization flow was restricted to two cells across 36 PVT corners, as this smaller subset was sufficient for tuning the agentic workflow. Each run took approximately 1.5 to 2 hours, and the team conducted approximately 25 rollouts using Claude Opus 4.8 running in Claude Code as the fixed model and agent harness.
Across successive evaluation rounds, the team refined the agent’s characterization skills and MCP tool descriptions. To improve tool selection, they simplified overlapping file and shell interfaces and defined tool inputs with explicit JSON schemas, making characterization tools easier for the agent to call.
Following these refinements, the mean composite verification score across evaluated rollouts increased from approximately 55% to 95%. The improved workflow also required fewer agent turns and output tokens. Agent turns fell from 155 to 53, a reduction of approximately 66%, while output tokens fell from approximately 81K to 29K, a reduction of approximately 64%.
| Model | Benchmark composite score | Turns | Avg. output of tokens |
| Claude Opus 4.8 | ~55% → ~95% | 155 → 53 | ~81K → ~29K |
Developing reliable agents in other domains using the same method
The same evaluation method applies to engineering agents in EDA, CAE, and other engineering domains:
- Evaluate the final engineering artifacts and not just whether individual tool calls succeeded.
- Write a domain-specific grader/verifier that detects missing, truncated, reformatted, or reused results to prevent agent cheating.
- Collect large numbers of rollouts to target improved reliability through optimized skills and MCP tool interfaces.
Get started with NeMo Gym
Use the NeMo Gym Quickstart guide to run an initial evaluation. Then apply the approach outlined in this post to one representative task: define a valid final artifact, build a domain-specific verifier, inspect rollout failures, refine skills and tool descriptions, and rerun under consistent evaluation conditions.
For more details about how Solido Fuse brings generative and agentic AI to Solido Characterization Suite agent, watch the Self-Verifying, Long-Running Agentic Characterization Workflow Demo.
Learn more about the collaboration between NVIDIA and Siemens.
Acknowledgments
We would like to acknowledge Gunjan Shah of NVIDIA, along with Tim Wheler, Lakshminarayanan Ramesh, and Carl Hofmeister of Siemens for their key contributions to developing the agent and establishing the evaluation methodology.