创始人如何基于 Claude Managed Agents 构建产品

Brief Description

This roundtable explores how three companies use Claude Managed Agents in meeting preparation, sales intelligence, and product analytics. The discussion focuses on the practical design choices behind outcomes and independent verification, organization-wide and account-level memory, sandboxed code access, rapid iteration, evaluation, model migration, and the trade-offs among speed, control, cost, and product experience.

Table of Contents

• Managed Agents across meetings, sales, and analytics
• Outcomes, rubrics, and independent verification
• Memory and cross-account work in Watchtower
• Sandboxing source code and operational analytics
• When to build a harness and when to buy one
• Rapid iteration, prompts, tools, and evaluations
• Stateful systems, external tools, and model migrations
• Cost, attribution, and product control

Managed Agents across meetings, sales, and analytics

Speaker: The conversation opens with three examples of Managed Agents in production. One team treats a meeting as a complete journey: preparation before the meeting, support during it, and follow-through afterward. Most products focus only on the meeting itself, but the more useful opportunity is helping a person prepare for a conversation, ensuring the intended outcome happens in that conversation, and carrying out the work that follows. The team uses Managed Agents both for briefs that prepare people and for task automation after a meeting.

Speaker: Another company builds agents for sales teams. Its basic product assigns an always-working agent to each account; that agent builds memory and determines what should happen to move the account forward. Sales work also includes questions that span accounts, such as which five accounts a representative should pursue today or which deals a manager should forecast next quarter. The company created Watchtower for cross-account reporting, aggregation, and rollups. Unlike the per-account agent, Watchtower needs a different set of tools and capabilities. The team assembled it with Managed Agents in two weeks, and it became one of the most-used features while also powering other parts of the product.

Speaker: A third company provides product analytics embedded in applications. As teams ship faster, it becomes more difficult to instrument applications and to understand where analytics diverges from what the software is actually doing. The company built an agent in front of its platform that connects a snapshot of a codebase with observed application behavior. Managed Agents let it compare code with production reality, send proactive insights, and sometimes propose fixes.

Outcomes, rubrics, and independent verification

Speaker: The meeting-product team explains why outcomes are central to its use of Managed Agents. A brief should identify the right person, explain why that person matters, recover prior context, and clarify what should be achieved in the meeting. A brief is valuable only when it is correct. If it confuses one person named Tom with another, it can actively damage the experience by making the user enter a meeting with false context.

Speaker: The team would rather show nothing than show a false positive. Outcomes allow an independent assessor to determine whether the agent produced the right result and whether source attribution is correct. This makes displayed information more reliable and also improves feedback loops: the more frequently a result can be assessed, the more that learning can be built into the initial agent loop.

Speaker: An outcome is described as a rubric supplied to an agent, with the agent iterating until that rubric is satisfied. For a meeting pre-read, the rubric includes end-user needs such as identifying the correct LinkedIn profile, which is objectively verifiable, as well as presentation quality. Information must be easy to scan and read; a five-page pre-read or one arranged in the wrong order will not be used. The team turns desired user experience into rubric criteria and hill-climbs against them.

Speaker: These checks happen at runtime. A meeting brief may be generated before a meeting and assessed within that available window. In one example, work begins twenty-four hours before the conversation, giving the system time to assess results. Verification can use independent evidence such as LinkedIn, email, and calendar data. The central value is a separate agent with an independent context window assessing work independently from the trace that created it.

Speaker: Full iteration to perfection is not always required. In proactive products, it can be better to withhold uncertain information: help when possible, but do not cause harm or annoyance. An independent clean context window makes quality assessment part of deployment rather than merely another internal step in the original agent loop.

Memory and cross-account work in Watchtower

Speaker: The sales team says that company-specific concepts are difficult because every organization defines ownership, forecasting, and deal processes differently, and the source data is often messy. A Salesforce instance may have many deprecated fields that mean nearly the same thing. Questions such as what should be forecast next quarter are important and require a correct answer.

Speaker: As people use the product, the system learns organizational concepts and stores them in organization-wide memory. That improves results for everyone and avoids forcing a large configuration exercise at the beginning. When a user asks a question, the product can ask clarifying questions and save the resulting information. Some queries also require deep work across many accounts. For hundreds of deals, the system can write code, make programmatic tool calls, narrow a candidate set, fan work out to independent agents for each account, and roll the results back up.

Speaker: The product distinguishes between memory attached to a particular account and memory about cross-account processes such as forecasting. Account agents run throughout the full account lifecycle, building context and learning concepts that can later be rolled up. Cross-account work produces another class of memory that may be less relevant to a single account but is important to Watchtower. The company uses account, user, and organization-level memory together to improve different parts of the product.

Speaker: In a typical setup, an account executive may have an agent for every account. The newer Watchtower product uses the collected memory across that fleet to answer broader questions and suggest actions. It can use account memory as an input, write code to query it, and fan work out when necessary. It also uses managed memory about the individual, the company, its priorities, and how it operates. This supports both individual contributors and leaders: a sales leader managing five account executives may effectively need visibility into five hundred accounts.

Sandboxing source code and operational analytics

Speaker: The analytics team describes sandboxing as essential when an agent snapshots customer source code. Source code may contain secrets, so the team must carefully control the tools available to an agent, the data sent to it, and the actions it can take. A successful interaction may end with a pull request pushed to a repository. Building all the required safety and security boundaries internally would not be core to the product's value, but they must work well enough for small businesses and large enterprises to trust the system.

Speaker: Managed Agents helped the team reach that baseline quickly. It uses several types of sessions: overnight batch systems that look for issues at scale, scheduled analysis of funnel changes, and faster sessions that inspect source code to answer a user's question more accurately. The roundtable characterizes these as three examples of Managed Agents: agents with sandboxing, agents with outcomes, and agents with memory.

Speaker: The analytics integration has practical uses beyond a single review. When a pull request is submitted, the company can launch a managed session to review the code, understand the change, and apply a lightweight UX review intended to prevent poor user experience from being committed. Once a change is merged, it can identify whether analytics must be updated or instrumentation added, and it may open a follow-up pull request. Nightly processes examine whether conversion dropped overnight or degraded over a longer period, look for changes that might explain the pattern, and suggest fixes. The team also examines agentic conversations, identifies unusually high error rates, and can propose prompt changes. The larger goal is software that increasingly understands real user behavior and suggests fixes while teams keep shipping quickly.

When to build a harness and when to buy one

Speaker: The decision to build a custom harness or use a managed solution depends on whether the harness is a core competency of the feature. For meeting briefs, the harness is not the core value; what matters is whether users receive the information they need. In contrast, a fully voice-controlled Jarvis-like system might make the harness itself a core competency because it could unlock new capabilities. The practical trade-off is control against cost and time: if the infrastructure does not determine the quality of the product, buy it; if it does, build it.

Speaker: The early stage of a product is especially sensitive to speed. Teams are validating product-market fit and the special features that make their solution dramatically better than traditional alternatives. The participants are not primarily infrastructure companies, and they do not need to build their own infrastructure to solve their core product problems. There may later be a need for finer-grained control or different operating costs, but early on the priority is getting features in front of customers quickly and changing them quickly.

Speaker: One participant had a homegrown solution but began encountering enough issues to conclude it would need a substantial rebuild. Moving to Managed Agents was fast and covered edge cases the team had expected to handle themselves. They had working infrastructure within days and moved the product over within weeks. Another team could build a first version of a meeting feature in a day, then scale from roughly one hundred to one thousand times more users over the following weeks with a few more days of iteration. The remaining hard work was learning exactly what users wanted through building, dogfooding, testing with others, and iterating over two or three weeks.

Rapid iteration, prompts, tools, and evaluations

Speaker: During iteration, the teams change prompts, skills, tools, and the orchestration loop. For a meeting assistant, the key questions include when to search, retrieve, rank relevant information, decide what to surface, and coordinate those steps across sources. When managed infrastructure handles concerns such as compaction, the team can focus on selecting and structuring the most important information for the particular meeting. Outcomes then help the team assess that information and iterate toward the desired result.

Speaker: A useful distinction emerges between the phase where a product is still becoming compelling and the phase where it must be made consistently excellent. In the early phase, teams can iterate on intuition and discover whether a feature is sticky. Once it is sticky, systematic evaluations are necessary. Evaluation cannot make an unappealing product compelling; the team must first find what makes it valuable to users. Managed infrastructure is especially useful during rapid iteration because it solves shared infrastructure concerns such as prompt caching, which also matters for cost.

Speaker: Evaluation begins after a product feels sufficiently good. Internal feedback is easy to gather, but it risks overfitting to employees rather than customers. Teams need external data points and feedback such as positive and negative responses to a feature outcome. Over time, that produces cohorts of evaluations for regressions, hard scenarios, different data partitions, and different task types. The roundtable recommends aligning the rubrics used for offline evaluations with the rubrics used in runtime outcomes whenever possible.

Stateful systems, external tools, and model migrations

Speaker: The sales team found it difficult to create good evaluations at first because internal dogfooding was not representative of real customers. A more useful approach was to inspect real customer queries, speak with the user who made the query, understand what they wanted, and build evaluations around those cases within the customer's data. That reduced the risk of evaluating an abstraction rather than the actual job to be done.

Speaker: Evaluation becomes harder for stateful systems with live memory. Teams can evaluate trajectories, such as how well an agent interacts with memory, but offline result evaluations become difficult when external state is changing. The same problem appears with MCP tools that call third-party services and mutate state over time. A Slack-backed evaluation may drift as Slack data changes, while mocking an entire customer environment is often impractical.

Speaker: On model migrations, the advice is to use a strong evaluation suite and avoid over-optimizing prompts for one model family. New models can introduce distinct failure modes, such as undesirable writing patterns or awkward sentence structures. It is more valuable to identify and defend against those new failure modes than to chase the last marginal improvement from a particular model generation, because the latter will change as models improve.

Cost, attribution, and product control

Speaker: Cost is a concern for nearly everyone building agents. A product that ignores cost is unlikely to have a sustainable long-term structure. The core question is when the cost, value, and control trade-off favors moving work in-house. The participants see the infrastructure markup of Managed Agents as relatively modest, but note limitations such as the difficulty of running some batch workloads. If a job can run well in advance, a team might save substantially through a cheaper batch-oriented approach.

Speaker: In the short and medium term, teams may have more user value to create than cost optimization to pursue. The recommendation is to be mature and mindful about spending, use the managed system while it enables rapid learning, and optimize details later. By then, the managed platform may offer more of the capabilities needed for that optimization.

Speaker: Fine-grained attribution also matters. Teams want to know which parts of an agent harness are accumulating cost. Large evening batch processes that handle significant data can become expensive, but teams can often make inexpensive engineering changes, such as running less often or sending less data, before over-optimizing the entire system. At sufficient scale and maturity, however, they will want greater control and better visibility.

Speaker: Model choice is one important cost lever. A coordinating agent that performs difficult search may need frontier intelligence, but fanning work out across five hundred accounts may require cheaper models. The current managed structure can make that optimization feel constrained or all-or-nothing. Participants would welcome more visible cost controls, including effort settings, multi-agent controls, more model options, and flexible or batch modes.

Speaker: Managed Agents work particularly well for hard tasks that need frontier intelligence, are relatively price-insensitive, and can run for minutes without making every second of latency critical. They let teams launch those capabilities quickly. For more latency-sensitive or lower-cost workloads, teams want additional levers, such as pre-warmed sandboxes to reduce cold starts. The final view is not that every workload should use a custom harness. If reducing cost or increasing control is necessary to make a specific feature viable, building more internally makes sense; otherwise, teams can combine managed components with the parts of their systems they already build themselves. The cost of an experiment is often the time it takes to learn, which is another situation where a managed solution can be especially valuable.