别盲信 AI 智能体：像科学家一样测试它们｜Omni Analytics 的 Stephanie Jarmak

Brief Description

Stephanie Jarmak of Omni Analytics discusses how to evaluate AI agents with a scientist's skepticism: test realistic workloads and complete configurations, trace the whole pipeline, choose measurements that support decisions, and enforce quality gates while balancing human review and operational risk.

Table of Contents

• Intro
• When the agent ignores your tool
• The harness changes the results
• Public benchmarks vs. your own evals
• Designing evals like a scientist
• Trace the whole pipeline
• Choose what you measure
• Quality gates that actually stop work
• The human review bottleneck
• When the agent's record can't be trusted
• Production is truth
• When to escalate to a human

Intro

My name is Stephanie Jarmak. I became an AI engineer a couple of weeks ago at Omni Analytics, where we have a platform with a governance semantic layer to provide a shared understanding of business data, using our beloved agent Bloopy.

Before that, I spent a year at Sourcegraph. I was a solutions engineer for Amp, a coding agent, and then moved into a more amorphous role largely working on their evaluation platform. I built a very large public benchmark called Code Scale Bench to understand how agents work across very complex code bases. Before that, I was an astronomer.

I approach this from a research scientist's point of view, relentlessly questioning everything. Hence the title, “Lies, Damned Lies, and Evals.” It says how to trust your agents, but this talk is mainly about how not to trust your agents.

When the agent ignores your tool

Here is an example of a run I did at Sourcegraph while testing their code search MCP tool. It shows the different numbers you can get after running an evaluation. What I was trying to figure out was: why isn't this coding agent using the tool I want it to use?

I had all the traces and looked at the primary metrics for evaluating the overall effectiveness of a tool. I had changed the initial prompt given to the agent and run it because I wanted it to use the tool. What I saw was that it was 28% cheaper. That is interesting, but it should not change my decisions about what I am actually interested in measuring and understanding about changes to the harness and setup, because it still did not use the tool.

You need to understand what you are setting up to measure. The 28% may be an interesting avenue to explore: we have some experimental space to investigate the variance and what is happening with cost measurements. But we still were not seeing it use the tool it was supposed to use.

We had an indication that it was receiving certain information and then trusting that the information was not there. That was purely because it did not use that tool.

The harness changes the results

I also want to talk about the impact of the harness itself. Here is an example from a third-party source with 63 or so tasks on Turbo Bench 4, a popular, widely used benchmark for coding agents. Suppose you are trying to pick which model to put into your harness.

They compared Opus 5 and Astra. With Open Hands, one of the open harnesses, Opus 5 beat it out. But when they swapped just the harness, Astra did a much better job. There was a 30-plus percent difference in the effectiveness of these configurations.

The takeaway is to evaluate the closest configuration you can to the work and environment you will be measuring, because there are so many variables involved. You might see benchmarks with fancy results, but the system might not do what you expect within your actual environment.

Public benchmarks vs. your own evals

That leads into distinguishing public benchmarks from evaluations and how to interpret them. An evaluation means making measurements to make a decision about your product or process and changes to it. Public benchmarks are primarily helpful for understanding the broader landscape and creating a shortlist of what to explore within your organization or tooling.

You do not want to over-index on those results because of the spiky nature of the training of different models. If you are a law firm, you do not necessarily want to look at terminal bench results, unless you are looking to incorporate coding agents with your developers. Make sure the configurations reflect your actual environments.

Think of benchmarks as a standardized reference set for comparison. Alongside public benchmarks, you can set up benchmarks within your organization that reflect actual workloads as closely as possible. Have an iterative flywheel process in which you increasingly hill-climb to optimize your configurations.

Designing evals like a scientist

Approach this as a scientist to the best of your ability. Once you have shortlisted your possible configurations, consider the variables you might change so that the results reflect your system.

Think about workload coverage. Pick tasks that are reflective of the work, and cluster them into different families. When I made the coding benchmark, the reason I built a whole different one was that, at the time, STB bench was largely just bug fixing. There is a whole other swath of tasks relevant to developers across the software development life cycle.

How you integrate tooling into those workloads may differ. A model might be more effective at code understanding than at fixing a bug, for example. You also need to consider how many runs to do. These systems are non-deterministic, so you may see variation from run to run.

As I showed with the model and harness configurations, think about how combinations of potential changes create interaction effects. How might those influence downstream effects?

Trace the whole pipeline

Gather as much observability data as possible. Retain all your traces. This goes back to the first example: what is happening, and why is the agent not calling the tool?

First, break down the entire context available to the agent. Was the tool even available? If it was not, stop there and fix that scenario. Then ask whether it was called and what values it returned. Keep going along the pipeline of available information.

Suppose I make an improvement and the agent finally uses the much better tool, but the outcome I care about at the end of the task still does not change. You can have a beautiful tool that you are really optimizing, but, for some reason involving the agent, configuration, or something else, the agent is not using that information. It is then your job to investigate and fix the problem.

Keep non-use, failed attempts, and the rest of this information in mind as you act like a detective to understand what is actually happening in your system. You also need to control what measurement you are making.

Choose what you measure

When I compared different search tools, I was interested in optimizing for something like an F1 score, balancing precision as well as recall. In an example involving grep, Claude was pulling in the entire kitchen sink: it returned 295 files when it only needed six.

If all I cared about was pulling in every possibly relevant piece of information, perhaps a recall of one would be sufficient. But I was interested in optimizing for a completely different type of value.

This matters when you look at single-number scores in public benchmarks. What was the validator actually measuring, and why was that considered important? What you want depends entirely on your business needs.

Quality gates that actually stop work

There is an interesting overlap between evals, quality gates, and reviews. I want to touch on that in terms of trust and safety.

I have been experimenting with a software factory with different approval queues. I want to automate as much of the work as possible, but I do not trust these systems at all, so I have regular audits.

For example, I put through a request for work to get done, with approval and required checks. The agent noticed that things were going wrong, but there was no mechanism to enforce a stop.

You need to decouple an agent or another system flagging something from enforcing that gate in code. Then consider how that is eventually escalated to a human. You also need to balance the human review load.

The human review bottleneck

This example comes from auditing and repairing LLM-as-a-judge failures in a separate system. They were seeing a 1% false alarm rate. If all the alarms raised go to a human, that would result in 10,000 reviews. Does your organization have the capacity for domain experts to do that?

Think about where that bottleneck might be and how to change the alarm rates. How consequential are those alarms, and how will they affect downstream outcomes?

We also need to consider that the agent trace itself can sometimes be unreliable or untrustworthy.

When the agent's record can't be trusted

In this example, researchers seeded different security issues into information provided to the agent. They had a hash check in place to detect tampering with the system. With that setup, everything passed: it did not notice any of the injected security issues.

But an independent record comparison that looked at the traffic history and model changes detected everything. Depending on your system's security needs and how you measure and observe what is happening, make sure you have sufficient trust in the record you retain of what happened.

Production is truth

This brings us back to evaluating realistic work associated with your particular task loads. Identify a target work comparison.

I am doing a lot of this at Omni right now, working on their evaluation harness. We look at internal production data to see how users interact with the agents and classify the different types of things they do. In our case, that includes Q&A ranging from easy to harder questions, dashboard building, and app building. There are different ways to classify the actual tests.

Then identify whether changes to the system are adding value, or where the downstream failures are. We are trying to come up with representative proxies for actual workloads, but ultimately production is truth.

After deploying changes, you need measurement systems that collect relevant data in the actual environment where the system runs. You can also do this in shadow to gain information when making a change would be less trustworthy or have a higher blast radius.

When to escalate to a human

Ultimately, optimize for observability and realistic environments that reflect where your system will be deployed, so you can trust the values you measure. Identify the level of risk you are willing to accept.

Within my own systems, such as my software factory, I give the agent some autonomy in lower-risk decisions. It identifies the next action, why human input is needed, and the operating scope. It also considers potential human checks given the available options.

Establish boundaries for when it is safe to deploy autonomously and when to stop and escalate to a human.

I have a GitHub repository with different skills I have worked on. A few are directly relevant to evaluation design, harness traceability, and reliability procedures. I also have a monstrous 300-page archive tome that you can point your agents at. You can point your agents at all of this to bring some of those best practices into your systems. Thank you.