自主计算机：面向计算机操作智能体的基础设施｜Simular 创始人 Ang Li

简要描述Brief Description

自主计算机：计算机使用智能体的基础设施——Ang Li，SimularThe Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li， Simular

目录Table of Contents

• Ang 与自主计算机目标
  Ang and the autonomous computer goal
• 计算机使用代理的作用
  What a computer use agent does
• 多个智能体与可见的桌面工作
  Multiple agents and visible desktop work
• 早期视觉定位的局限性
  Early visual grounding limitations
• 自主性与计算机
  Autonomy and the computer
• 屏幕截图和无障碍信号
  Screenshots and accessibility signals
• Agent S 的研究历程
  The Agent S research trajectory
• 并行尝试与基准测试进展
  Parallel attempts and benchmark progress
• 一个 FaceTime 电脑控制演示
  A FaceTime computer control demo
• 多次运行的可靠性
  Reliability across repeated runs
• 澄清含糊不清的指示
  Clarifying ambiguous instructions
• 符号自动化层
  The symbolic automation layer
• 编码代理设置与现场演示
  Coding agent setup and live demo
• 编写并修复工作流
  Write and repair a workflow
• 读取无障碍快照
  Read an accessibility snapshot
• 稳定性与适应性
  Stability and adaptation
• 重放代码并从变更中恢复
  Replay code and recover from changes
• 启用实验性回放模式
  Enable the experimental replay mode
• 旧版表单自动化示例
  Legacy form automation example
• 寻找会议演讲嘉宾
  Find a conference speaker
• 将工作流保存为技能
  Save the workflow as a skill
• 重放另一天的内容
  Replay for a different day
• 复用已有工作并减少模型调用
  Reusing work and reducing model calls
• 为什么智能体需要独立的计算机
  Why agents need separate computers
• 云桌面与开发者访问权限
  Cloud desktops and developer access
• 多个平台和 Android 演示程序
  Multiple platforms and Android demos
• 开发者工具与工作坊总结
  Developer tools and workshop wrap up
• 屏幕理解与记忆问题
  Screen understanding and memory questions
• 状态与多种输入模态
  State and multiple input modalities
• 准确性与验证循环
  Accuracy and verification loops
• 不可逆操作与操作系统限制
  Irreversible actions and OS restrictions
• 遗留工作流和流程
  Legacy workflows and procedures
• 适应应用程序的变化
  Adapting when applications change
• 训练数据与代码复杂度
  Training data and code complexity

Ang 与自主计算机目标Ang and the autonomous computer goal

简单介绍一下我的背景。嗯，我是 Similar 公司的联合创始人兼 CEO。今年差不多是我和计算机打交道的第 30 年，而现在我正试着打造一台完全自主的计算机，这样我就不必再和它打交道了。回顾我的经历，最初的 10 年里，我很小的时候就开始接触计算机。那 10 年，我专注于竞赛编程，基本上每天都在写代码。接下来的 10 年，我花了很多时间训练神经网络；读博士期间以及后来在 DeepMind 工作时，我发表了很多 AI 研究论文。第三个十年——嗯，第三个十年是从 2017 年我加入 DeepMind 开始的。今年是 2026 年，差不多快满 10 年了。我刚加入 DeepMind 时，我们真的在努力解决一个叫作 AGI（通用人工智能）的问题。当时大家都还很困惑。a little bit about my background. Um, I'm the co-founder and CEO of a similar and this year is pretty much the like the 30th years that I've interacted with computers and now I'm trying to build a computers that's fully autonomous so I don't have to interact with that. And if I look look back to my history, the first 10 years I'm I I I basically get to know computer when I was pretty young. The first 10 years I focus on comparative programming like basically writing code every day. And the the second decade I spent a lot of time uh training neuron nets basically published a lot of AI research papers uh during my PhD and later on in deep mind and the third um uh the third decade so I joined deep mind in 2017 that's the beginning of the third decade so this year is 20 uh 26 almost 10 years so so when when I first joined deep mind we really try to solve this problem called AGI artificial general intelligence And back then everyone is still confused.

即使到了现在，大家还是都搞不清楚 AGI 的定义是什么。不过一般来说，它有点像一种通用系统，有点像人类——像一台机器，能做各种各样的事情。希望明年我们就能解决这个问题。所以我可以想想该做些什么、该解决什么问题，作为第一个十年的目标。好，我现在想开始演讲了。嗯，我之所以想介绍这些，基本上是因为我花了 10 年编程、10 年研究神经网络、10 年解决 AGI 问题。实际上，这让我形成了一种偏见，而我们也试着把这种想法带到现在的公司里：我们秉持这样一个原则，通用 AI 系统需要采用神经符号架构，基本上就是把代码和神经网络结合起来。所以，我很快会向大家说明，我们为什么会有这样的直觉。Even now it's everyone's still confused about what's the definition of AGI but in general it's kind of like a general system kind of like like human uh humans like like a machine can do all different kinds of things in there and uh hopefully next year we can solve the problem. So I I can think about what to do what kind of problem to solve uh for the for the first decade. Okay. So I want to u start my talk now. Um basically the reason I want to introduce that is because I spent 10 years on coding 10 years on neuronets 10 10 years on solving AGI. So actually that gave me this this kind of bias that we we try to inject to our current company that we have this uh this kind of principle in mind the general AI system needs to be neurosyolic basically code combined with neuronets. So, so which I will I will show you quickly why uh like we have this kind of intuition in there.

所以，我知道这个工作坊时间有点紧，不过我确实想给大家带来三样东西。第一是关于计算机使用的基础知识——我知道你们当中有 50% 的人对这个领域还不熟悉。第二是我们一直在基础设施建设中开发的工具。我们认为，有四类不同的组件会对你们今后开发计算机应用有所帮助。我还会展示其他人，也就是其他开发者正在开发的一些应用。第三件事也很重要：我们有一款名为 Sigh 的产品，目前仍处于封闭测试阶段，所以我想在工作坊结束时给大家发邀请码，让大家也有机会体验这款产品。另外，欢迎在 Twitter 上关注我，也欢迎加入 Discord。So, the goal for this workshop I know we we're kind of short on time and I do want to give uh three things back to you. First is basic knowledge about computer use u I understand 50% of you are still new to this domain and second is the tools we we have been working on as part of the infrastructure. There are four different type type of components that we believe would be helpful to your future development of computers uh applications and I'll also show some applications that other people has been working on or other developers and third which is important uh since uh what we have a product called sigh and this product is still closed beta right now so I want to give everyone invitation code at the end of the the workshop so you have a you know opportunity to play with the product also do let me know uh Feel free to follow me on uh Twitter and join Discord.

如果你需要更多额度，告诉我。所以，所以我肯定会多给一些额度。所以，首先】【。Let me know if you need more credits. So, so I definitely like give more credits in there. So, the first

计算机使用代理的功能What a computer use agent does

问题是，什么是计算机操作智能体，对吧？所以，呃，通常我演讲时会和大家在这里自拍一张。我拍一半。好。所以，这就是我现在每天做的事。其实我是 CEO，所以我不会整天都用电脑。我在手机上有自己的智能体。我可以给智能体发消息，把照片发过去，然后说：“把这张照片发到 LinkedIn 上。”然后智能体就会开始处理。我们来看看。我想看看智能体有没有回复。我想我收到了一些。好，我们看看。我这边确实有消息。好，智能体正在回复。它已经收到我的照片了。你们可以看到，智能体开始问我问题了。呃，我现在先不和智能体互动，不过我想给大家看看基本界面。所谓计算机操作智能体，指的就是有一台计算机的智能体；在这个场景中，这是一台完整的 Windows 桌面电脑。如果我继续与它互动，它就会通过鼠标和键盘在这台电脑上操作，之后打开 LinkedIn。接下来我会回到演示文稿。question is what is a computer use agents, right? So, uh normally what I do with the with my talk is I can take a selfie here with everyone. I'll take half. Yeah. So, so this is what I do every day now. So actually I'm I'm the CEO, so I don't use computer all day long. I have my agents on my phone. So I can text my agent, send a photo there and say post the photo to LinkedIn. Yeah. And then the agent will will be working on it. So let's see. So I do want to see if the agent is responding there. I think I got some. Okay. So, let's see. I do have my message there. Okay. The agent is responding. So, he has my photo there. And you can see the agent is starting asking me questions there. So uh I uh I won't interact with the agent right now but I want to show you the basic interface. So computer is agents means mean means the agent that has a computer which is a full Windows desktop in this scenario and if I continue interact with the computer the mouse and keyboard is know it's going to work on this desktop and open a LinkedIn later and I'll go back to the presentation.

我刚才只是给你展示了一个只有一个智能体的例子。所以我可以和负责应用程序的智能体交谈，但实际上你可以有多个智能体。I just show you an example of having one agent. So I can talk to the agent agent working on applications but actually you can have multiple

多智能体与可见桌面操作工作Multiple agents and visible desktop work

这就是我们实际上正在做的事情。你可以看到，这里有四个智能体，它们的名字各不相同。QA 智能体基本上就是负责对产品进行 QA，比如测试产品就是其中一个主要用例。它们还会研究客户支持，进入 score 频道查看那里的所有信息。实际运行中的智能体速度比较慢，所以这段录制实际上有好几个小时。我把它压缩成了 30 秒，让你大致了解一下计算机使用智能体是如何工作的。总体来说，这和我们现在讨论的大多数智能体非常不同。大多数智能体都基于 API，你实际上看不到智能体在做什么。计算机使用智能体是另一类智能体，我们认为它会成为一个更大的类别，能够做更多事情：即使没有 API，你仍然可以让智能体来处理。然后我想稍微回顾一下，因为我们是在 2023 年创办公司的，那是在 GT 3.5 发布之后不久。that's what we are actually working on. So uh the these are four agents you can see they have different names right so QA agents is basically like uh doing QA on the products testing product is one big use case in there they also research customer support go to the score channel and looking at all of the information there. So the the actual agents are slower. So it's actually this is recording for like hours. So I just compressed that into 30 seconds for you to basically grab the you know the basic understanding of of how computer us agents is working in there. So the general idea this is very different from um most of the agents that we talk about right now like most of the agent they're based on APIs and you don't actually see visually what the agent is doing. Computer use agents is another category and we believe is a is a is another bigger category which can do a lot more things that you you don't have APIs you can still let agents to work on it and then I want to do a little bit ritual because we started a company in 2023 right and that was uh right after GT 3.5

早期视觉接地能力的局限性Early visual grounding limitations

还有，呃，GT4V。我想给你们看一个例子：两年前，我们在这个问题上可以说是束手无策。我当时还把它发到了 Twitter 上。这是我的推文，我稍微停一下。这是我两年前发的推文。问题是：我们能不能用 GP4V 理解屏幕，然后点击屏幕上的 iMessage 图标？这是个非常非常简单、微不足道的问题。我基本上是用模型对定位结果进行迭代优化。你们可以看到，第一次的结果不正确，第二次点到了邮箱图标，第三次才正确地点到了 iMessage 图标。这是我做过的唯一一个成功演示。所以，我让智能体点击 Zoom，但它完全不起作用。and uh GT4V so I want to show you one example that two years ago we kind of we're kind of hopeless two years ago on this problem in there. I actually post it on on Twitter. So this is is my tweet and let me pause a little bit. This is my tweet two years ago. So the the problems is like can we actually use GP4V to understand the screen and then click on the iMessage icon on the screen. Very very simple trivial trivial problem. And I basically use the model build up a iterative refinement of the the grounding. You see the the most the first time is is not correct. The second time is go to the mailbox icon and the third time is correct on the iMessage. So this is this is the only one successful demo I did. So so I asked I asked the agent to to click on Zoom. It doesn't work at all.

所以两年前这个问题基本上是无解的，但我们多少有种直觉，呃，金融模型会随着时间推移变得越来越好，最终这个问题会得到解决，但目前就是做不到。So two years ago this problem is basically hopeless but we kind of have the intuition that uh financial model is going to be better and better over time eventually this problem will be solvable but it's just like it's not possible

自主性与计算机Autonomy and the computer

回顾过去两年，这实际上促使我们思考：解决计算机使用智能体问题的一般策略是什么？我们给它起了个名字，叫“自主计算机”，因为你知道，如果计算机能做各种各样的事情，那它就是自主的，对吧？那么，问题是我们该如何解决自主计算机的问题？我谈到这个概念时，只会用两个词：自主和计算机。within two years back our um that actually motivates us to think about what's the general strategy for solving this computer use agents uh problem and we do have a name for that it's called autonous computer because you know like if the computer can do all different kind of things it's autonomous right and then The question is how do we solve autonomous computer? So when I when I talk about it there are only two words autonomous and computer.

所以这两个方向自然而然地就成了你会思考的问题：一个是自主性，另一个是计算机。自主性指的是，我们有没有一个系统，能够真正理解计算机的上下文和运行环境，完成所有推理和规划，并且能从自己的错误中恢复过来，不需要人类一直在旁边照看这个智能体。第二部分是计算机：智能体可以使用什么样的计算机？这是个核心问题。不过两年前，我们觉得计算机其实是个简单问题，因为每个人本来就有计算机，对吧？所以我们把很多精力放在了自主性上。 谈到自主性，就要回到我们最初的底层原则，也就是所谓的神经符号方法。“神经”这一部分也会出现在自动驾驶汽车中；基本上，各种不同类型的自主系统都遵循同一个概念：其中有一个神经组件，在这里就是视觉语言模型（VLM）智能体。第二部分是符号组件，我们当时称之为编码智能体。要实现自主性，我们基本上就得解决这两项技术。 我刚才给你们看过一个两年前的演示。当时的 VLM 根本行不通，对吧？所以两年前，我们确实花了很多时间研究编码层，确保我们能以更可靠的方式控制计算机。但第二个问题是，两年前我们组建了一个研究团队，思考如何提升这个 Craft App 的 VLM，让它更擅长使用计算机。So it's become so natural two directions you think about one is the autonomy the other is computer right autonomy meaning like do we have a system that can actually understand the context the environment of the of the computer and do all the reasoning planning can recover from its own mistake um like doesn't require human to babysitting the the agent and the second part is computer like what kind of computer uh the agent can use so this is a core question but back then two years ago we feel computer is actually easy problem because everyone already have computers right so that's why we focus a lot on the autonomy part so when we talk about autonomy this goes back to the the underlying principle we initially have is called neuro symbolic neuro means this also happens to self-driving car it's basically the the same concept for all different kinds of autonomous system they all have the same concept you you you have a component that's neuro here is a VM vision language model VM agent and the Second part is symbolic and here is we call it coding agent back then and basically we need to solve these two technology in order for us to make autonomy works and I just show you a demo two years ago VLM doesn't work at all right so that's why we actually spent a lot of time two years ago working on the coding layer to ensure okay like we can actually control the computer in a more reliable way but the second problem is we we set up a research team two years ago to think about okay how can we make this craft app VM better in terms of using the uh using the computers.

呃，在我开始谈论我们的……轨迹之前♀♀♀♀♀♀Um before I start talk about the the trajectory our

屏幕截图和无障碍信号Screenshots and accessibility signals

在 VM agents 这段旅程中，我想简单谈谈这些信号。你知道，就像自动驾驶汽车在道路上行驶时，会用到信号灯、雷达，还有摄像头，对吧？计算机也有同样的三种信号。所以我经常告诉别人：“好吧，我们其实是在数字世界里研究自动驾驶汽车，只不过是把方向盘换成了鼠标和键盘。”因此，我们仍然有三种不同类型的信号。一种是屏幕截图，另一种是无障碍树。无障碍树很特别，在我们的日常生活中比较少见。DOM 树则很常见，因为我们会使用浏览器。无障碍树有点像 DOM 树，但它与各个操作系统紧密相关。journey on the VM agents I want to uh briefly talk about the the signals like you know like self-driving cars uh like we running on the road they have signals lighter radar and uh and like camera right three different kind of signal computers had the same so I often tell people like okay we are we actually working on self-driving cars in the in computer in a digital world basically changing the steering wheel to your mouse and keyboard so we still three different kinds of signals. One is screenshot, the other is accessibility tree. So accessibility tree is very unique. It's it's kind of rare, you know, in in our in our daily life. DOM tree is pretty popular because we use browsers. Accessibility tree is a kind of a uh still kind of like a dome tree, but this is this tied to each of the operating system.

Mac 有它自己的无障碍树格式，而 Windows 则有一套独立的无障碍树协议。但这就是操作系统用来在屏幕上渲染所有窗口、所有图标的基础层。所以这一部分非常重要，因为，嗯……Mac has it its own accessibility tree format and Windows has it like a separate protocol for accessibility tree. But this is the basic the the the layer that operating system use to render all of the windows all of the icons on on your screen in there. So this this part is very important. It's because um

Agent S 的研究历程The Agent S research trajectory

对，我们一直在做的这项工作叫作……整个想法就是：我们能不能搭建一个 harness？两年前我们还不叫它 harness，现在大家都叫它 harness。两年前，我们着手完成一个叫作 Agents 的项目，尝试搭建一个 harness，基于所有基础模型，而不只是某一个模型，这样你就可以轻松切换模型。这是一个与模型无关的框架，能让你在其中实现最佳的计算机操作能力。第一代在 2024 年末完成，叫作 Agents。我们就叫它 Agents，因为那只是个开始。这个智能体在 OSWorld 上的成功率达到了 20%。这在当时，也就是 2024 年，是最先进的水平，非常有意思。这篇论文还获得了 ICLR 2025 Agentic AI Workshop 的最佳论文奖。你知道，过去一年整个行业发展得飞快，简直不可思议。我们之所以选择这个 harness，首先是因为我们知道视觉语言模型实在太差了，根本行不通。yeah the these are the the effort we've been working on is called a so the whole idea is can we build out a harness. So two years ago we don't call it harness. Now day everyone call it harness. So two years ago we we build out we try to finish this project called agents build out a harness b uh build upon all of the foundation models not just one so you can easily switch models it's model agnostic framework that allows you to produce the best uh computers capability in there and the first generation was done in late 2024 and it's called agents uh we just call it agents because that that's the beginning and This agent achieved 20% success rate on OS word 20%. And that was the state of art state-of-the-art back then in 2024 which is very interesting. And this paper this paper also won the best paper in I clear 2025 agenda AI workshop you know that the the whole industry was moving so fast over the past year and which is crazy and this the the choice for this harness is because first we know vision language model is so bad it's not going to work.

这就是我们选择在第一代中使用无障碍树的原因。基本上，我们会查看树结构底层的所有协议，用它们来描述当前环境，并将其传递给大型语言模型，而不是视觉语言模型；然后使用基于 ID 的定位方式，把所有按钮线性化，为它们分配 ID，再问语言模型：“我现在需要点击的那个按钮，ID 是什么？”这种方法有点效果，成功率达到了 20%，这非常令人兴奋，因为在 2024 年初，这个基准测试的成功率只有 5%。到了 2025 年初，我们开始发现视觉语言模型正在变得更好。因此，在第二代智能体中，我们实际上完全舍弃了无障碍树，采用纯视觉方案。有趣的是，这种架构有多个层级，比如管理器和工作器。That's why we chose to use accessibility tree in the first generation. Basically, we look at all of the the underlying protocols of the tree structure to describe the current environment and pass it to a large language model instead of vision language model and use ID ID based grounding basically linearize all of the all of the buttons put them IDs and ask LM like which button what's the ID of the button where I need to click on right now. It kind of works. There's 20%. which is super exciting because in the beginning this benchmark is like 5% success rate in the early 2024 in there and then early 2025 we started to see okay vision language model is getting better so the second generation of agents we actually uh completely dis discard the accessibility tree so it's a vision only approach and the interesting part is this architecture we have multiple layers like managers and workers.

所以这有点像人类组织，我们展示了它在某种程度上确实可行。具体来说，这项工作其实有很多朋友跟我聊过，他们对这项工作相当兴奋。原因是，这项工作证明了：有两个模型比只有一个模型更好。在同一个运行框架中使用两个基础模型，也比只用一个基础模型更好。而第二个基础模型是一个 70 亿参数的开源模型。所以，是的，所有创业圈的朋友都对此相当兴奋，因为这意味着我们有了这样一条路：我可以训练一些小模型，让它们比 Claude 或 OpenAI 的计算机代理强 20%。So it's kind of like human organization and we showed it kind of works and this work specifically was um a a lot of friends in summary actually talked to me about this work. They they feel pretty excited about this work. The reason is this master demonstrate that if you have two models you are better than one models. If you have two foundation models within the same harness it's better than one uh foundation models. And the second foundation model is a seven billion open source model. So yeah, so all of the startup friends, they are pretty excited about it because okay, there's a there's our journey like I can train some small models and we can be 20% better than uh like cloud or openi uh computer agents there

并行尝试与基准测试进展Parallel attempts and benchmark progress

到 2025 年底，我们有了第三代智能体。这部分的整体思路是：我们能否在测试时进行扩展？这叫作推理时扩展，也就是在有一个智能体的情况下，能否让多个智能体尝试解决同一个任务，最终获得更好的表现？在演示中，这种方法确实有效。这里的图展示了总资源预算，也就是工作者总数乘以总步数，用于进行公平比较。你可以看到，如果有 10 个智能体，成功率的提升效果确实更好。and late 2025 we have our third generation agents and the whole idea of this part is can we actually do test time scaling it's called inference time scaling is when you have a agent can you have multiple agents to try to attack uh solve the same task and eventually you have better performance and with demonstrator it works. So this is uh the figure for the total resource budget meaning the total number of workers versus uh times the total number of steps and this for like fair comparison and you can see if you have 10 agents you actually scale better in terms of success rate.

于是到了去年 12 月，我们在 OSWorld 基准测试中打破了人类表现纪录。这里展示的是各个智能体之间的所有轨迹对比。Anthropic 和 OpenAI 这两家大型实验室都在今年 3 月达到了同等的人类表现。去年 12 月也是一个令人兴奋的时刻，我们觉得，电脑使用智能体终于开始变得有些实用了。不过你知道，这个基准测试毕竟是研究基准，并不意味着它在现实生活中已经达到 100% 的水平，或者已经比人类更好了。几天前，OSWorld 发布了第二代基准，名为 OSWorld 2。我对这个新基准特别兴奋。在新基准上，表现最好的模型得分也只有 20%。所以，我们正在进入电脑智能体发展的第二阶段。接下来我想给你们看一个视频。So then in December we break the human performance uh record in OS world benchmark and these are comparison of all the trajectory between among uh agents and anthra and openi uh both big labs they achieve the same human performance uh in March this year and that also uh it's uh last December was an exciting moment like we feel okay like computer user agents is finally start to be somewhat useful but you know this benchmark is kind of research benchmark it doesn't mean like in real life it's already 100% it's already better than human um a few days ago OS word has uh has released their second generation called OS world 2 I'm specifically excited about this new benchmark so um in the new benchmark the best performing model is only 20% in that benchmark so so we are we are entering the second phase of computers uh agents in there I want to show show you one video.

呃，有一个非常有趣的演示。Uh there was a very interesting um demo

一个 FaceTime 电脑控制演示A FaceTime computer control demo

那个开发者用的那个。所以这个项目实际上去年赢得了 Cow Hacks。它是伯克利举办的规模最大的黑客松之一，他们基于开源智能体构建了自己的应用。that one of the developer used. So this this project actually won the cow hacks last year. So it's it's a it's a one of the biggest hackathon in uh hosted in in Berkeley and they they built their applications based on agents the open source one.

主持人：我想，呃，你可能不会……我会听到那个声音。Host: I guess uh you probably don't I'll hear the voice.

嘉宾：如果你只靠它，就能从世界任何地方控制自己的电脑呢？我们打造了一个能够全面控制电脑的多模态智能体。所以，与其打电话给妈妈，让她帮你排查那两个内存不足错误，不如直接通过 FaceTime 联系 Mac。让我来演示一下它是怎么工作的。Guest: What if you can control your computer from anywhere in the world just by it? We built a multimodal agent with full control of the computer. So instead of calling your mom to debug your two out of memory errors, you can just FaceTime the Mac instead. Let me show you how it works.

主持人：好，假设你在家训练一个 A 模型，但你已经出门了，这时才意识到发生了 CUDA 显存不足错误。那最简单的解决办法是什么？其实，你不用一路开车回家，只要给电脑打个电话，让它自己处理就行。我们可以给它打电话，然后发短信让它通过 FaceTime 联系我。这样就会在这里启动电脑使用代理，开始 FaceTime 通话，这样我就能告诉它我想做什么。Host: Okay, so suppose that you're training an A model at home, but you already left and you just realized that you had a CUDA out of memory error. So what's the easiest way to fix this? Well, instead of driving all the way back home, you can just call your computer and tell it to do itself. So, we can call it and text it to FaceTime me and then this will initiate the computer use agent right here to uh start a FaceTime call so that I can tell it what I want to do.

嘉宾：是的，他们他们做了一件非常有趣的事。让我……Guest: Yeah, they they did a very interesting thing. Let me

主持人：现在我能看到它显示的内容了。我们能解决 VS Code 中的 CUDA 内存不足错误吗？好，任务是解决 VS Code 中的 CUDA 内存不足错误。我按下 Command+Space 打开 VS Code 应用程序。Host: So now I can see what the thing says. Can we fix my CUDA out of memory error in VS Code? Okay, the task is to fix the CUDA out of memory error in VS Code. I will hit command space VS Code to open the application.

嘉宾：所以这些都是自动的。Guest: So these are all automatic.

主持人：好的，我现在在 VS Code 里。我在查找 CUDA 显存不足错误。Host: Okay, I'm in VS Code. I'm looking for the CUDA out of memory error.

嘉宾：看起来这个文件出现了 CUDA 显存不足错误。根据注释来看，张量的尺寸似乎太大了。所以我应该试着把它调小到一个合理的大小。是的，我想你大概明白这个意思了。这个想法就是构建一个应用程序，让你可以通过 FaceTime 控制电脑，并用 FaceTime 在 VS Code 里调试。那是去年十月发生的事。那是在……之前。Guest: It looks like this file has a CUDA out of memory error. Based on the comments, it looks like the tensor size is too large. So I should try to decrease this to something reasonable. Yeah, I think I think you kind of get the idea. So the idea is build a application that you can control your computer using FaceTime and use FaceTime to debug in VS code. That was happening in October last year. That's before like

多次运行的可靠性Reliability across repeated runs

甚至 OpenClaw 也是在那里作为一个项目启动的。后来我们不得不思考，计算机使用智能体的第二阶段会是什么。我们已经在基准测试中超越了人类表现，接下来呢？然后我们意识到，其实有个很有意思的概念叫 pass@k。过去，大家衡量的都是 pass@1，这在氛围编程中很常见：你有一项任务，就让智能体生成代码；如果代码跑不通，你就再输入一些提示，让它再生成一份代码。只要其中一份代码成功了，就算成功，因为你不在乎之前发生了什么，只在乎能不能得到一份可用的代码，对吧？但在计算机使用场景中，情况很不一样。两年前我们开始有客户时，他们都问过我同一个问题：我能不能把你们的智能体部署在服务器上，让它每天运行这个工作流，而我不用一直盯着工作流，也不用在那儿调试？even open claw was started as a project in there. Then we had to think about what's the second uh you know second phase of computer use agents in there. So we already surpass human performance in the benchmark like what's next? Then we realized okay there's actually a very interesting concept to called pass to the power of K. And in the past, everyone was measuring pass ad meaning is is pretty popular in in VIP coding like you you have a task you add you ask the agent to uh produce the code right if the code doesn't work and then you you put some prompts in there and produce another another uh code in there one of the code becomes successful is successful because you don't care about what's happening in the past you only care can I get a code that works right in computer use is very different because Like we we started to get customers two years ago and all of the customer talk to me about this question like can I actually put your agents hosted in a server and run this workflow every day and I don't have to you know keep looking at looking at looking at what workflows I don't have to debug in there.

所以基本上，大家都在跟我讲这个指标——pass 的 k 次方。这意味着每次要成功，我都得运行这个智能体 K 次，而这个指标并没有被这里的任何基准测试所涵盖。于是我们开始研究这个问题。我们选取了 OSWorld 中表现最好的智能体，它们在三个不同类别中的表现都相当不错，然后让这些智能体运行三次。我们测量了 pass 的表现，结果大约是三分之一。从这张图中你可以看到，所有智能体的表现都下降了，成功率几乎都降低了 10%，甚至连其中表现最好的智能体也不例外。第二部分是在环境受到扰动的情况下进行的，也就是说，我的查询可能会略有不同，环境每次也可能略有不同，结果就更糟了。所以，这仍然是当今计算机智能体面临的一个问题。因为你可以看到，Claude、Codex 都有智能体，Claw 也有智能体。So basically everyone was telling me about this metric pass to the power k. So this means I run this agent K times every time it needs to be successful and this metric is not captured by any benchmark in there and then we started to research this problem. So we took the best performing agents in OS word and their performance is pretty good in three different categories and then we run the stem agents three times. We measure the performance of pass to about to about three right from this figure you can see all the agents has degraded performance in there they almost dropped 10% success rate even even the best performing agent in there and the second part is on perturb environment meaning my querics could be a little bit different and environment could be a little bit different each time and it get even worse so this this is still a problem in the computers agents nowadays Because you can see uh Claude has Codex has uh agents, claw has agents.

这些代理实际上都无法生成可靠的None of the agents actually produce reliable

澄清含糊不清的指令Clarifying ambiguous instructions

这是一个可重复使用的流程。这是我们最近发表的一篇论文，大约是两个月前发表的。论文主要讨论了这个问题，并提出了一个简单的小技巧。如果你希望自己的智能体以后更可靠，任何人都可以借鉴这个思路。这个技巧基本上就是：你给智能体下达指令时，让它先向你提出澄清问题。论文的意思是，假设智能体总会就你提示词中含糊不清的部分向你提问，那么它的可靠性就会更高。论文里有一段提示词，大家可以看看。它其实很简单，大意是：“你的目标是尽量减少澄清问题，同时仍保留原本意图。”这一点非常重要。把这段提示词给智能体，它的表现就会更好。repeatable process for you. So this is a recent paper we published uh like two months ago. So basically discussing this problem and there's a little simple trick that we proposed in this paper that everyone can you know can borrow the ideas in the future if you want your agents to be more reliable. The trick is basically ask the agent to to ask you clarification clarification questions when you give them the instruction. So, so basically this paper was saying like suppose the agent always ask you some questions about ambi ambiguous part in your in your prompt the reliability of the agent will be better and there's a prompt in the in a paper that everyone can look at it like basically it's a it's it's pretty simple it's just like okay like your goal is to minimize u uh have minimum uh clarification that still preserving the nature this line is very important You put this prompt to the agent and the agent can perform better.

在你把这个澄清提示发给智能体之后，可以看到智能体的整体可靠性有所提升：虚线表示没有澄清时的原始表现，实线表示加入澄清后的表现。对吧？可以看到，Claude 的表现有所提升；有意思的是，使用 GPT-5 的 Agent S 提升幅度最大。紫色线代表与模型无关的框架，绿色线代表大型实验室开发的计算机智能体。好，基本上，这就是我们解决 VM 智能体问题的历程。它似乎运行得相当不错。直到现在，人们仍在使用这个智能体框架，因为它与模型无关，可以接入任意开源基础模型。我也很期待能看到它在最新开源大型模型（比如 GLM 5.2 Tool）上的测试结果，看看表现如何。After you put this clarification prompt to the agent, you can see like the overall um reliability of the agent is getting better like dash line is the original performance with no clarification and the solid line is the one with clarification. Right? You can see cloud is getting better and agent S with GPT5 which is interesting has the biggest improvements in there. So, so the the purple line is model agnostic framework and the green line is is the you know like big labs computer agents in there. Okay. So, so basically this is our journey for solving the VM agent which seems to be work work pretty well and still like nowadays still people using the agents framework because model agnostic you can plug in any open source foundation models. I'm also like looking forward to getting like results with the latest open source big model like GLM uh 5.2 tool and see how the performance looks like in there.

然后第二个问题是，好吧，这并没有解决可靠性问题，对吧？这些智能体似乎并没有……只要你用的是神经网络，只要你用的是虚拟机，说到底，它就是一个概率模型。它每次都会给出一个概率。所以，如果你每天 24/7 地运行它，它的表现也不会始终一样。然后……And then the second question is okay, it doesn't solve the reliability problem, right? It like these these agent doesn't seem to be have as long as you're using a neuronet, as long as you use a VM, it's in the end at the end of the day, it's a probabilistic model. It produce a probability each time. So it's not going to give you the same performance uh if you run it for like 247 every day. Then

符号自动化层The symbolic automation layer

我们不得不思考接下来要考虑什么，基本上就是符号层。我们称之为编码智能体，并将这种语言开源了，我们称它为 Simulan。嗯，它是一种用于类似场景的语言，所以我们就叫它 Simulan。这种语言有点像 RPA 行业过去的做法：他们会让工程师编写代码。嗯，RPA 常见的问题是，整个社区没有一个大家都能使用的统一协议。我们尝试在浏览器自动化领域创建这样的协议；我们有 Playwright，它是一种标准化方案，但在计算机操作领域，实际上并没有行业标准。所以我们开源了它，希望能对所有开发者有所帮助，这样你们就可以为不同的工作流构建自己的脚本层，使其每次都可靠且具有确定性。we had to think about what's the next thing to think about is basically symbolic layer. We call it coding agent and we open source this uh language we call it simulan. Uh it's a language for similar. So we call it simulan there. So this language is kind of like if you're familiar with RPA industry it's kind of like what RPA industry used to do like they ask engineers to write a code. uh but the common problem for uh RPA is there's no single protocol that's shared across the whole community that everyone can use and we try to create this uh uh in browser automation we have playright which is in a standard way but in computer use there's no really no uh industry standard there so that's why we we open source this we hope it's helpful to all the developers so you can u actually like build your own square pin layer for different workflows that's reliable deterministic each time.

对。所以，基本上，目标是让你能够运行并为 Simulan 创建呃源代码，然后运行这段代码，你知道，在终端里运行，而且你也可以在 Claude Code 里使用它。所以我确实想给你一个……Yeah. So, basically the goal is to allow you to run to create uh source code for Simulan and run this code you know you know terminal and you can also use it in claw code. So I I do want to give you a

编码代理设置与现场演示Coding agent setup and live demo

先快速演示一下。好，那我现在在 Claude Code 里。要是你安装了……嗯，也许我应该给你们演示一下 Simulan。对，这些是它的落地页；打开之后，你会看到一个简单的单行安装命令，用 npm 安装。安装完成后，你就可以在自己的终端里使用它了。我是 Claude Code 用户，所以我想看看这个，因为现在已经没人写代码了，对吧？使用氛围编程来完成这类自动化任务很重要。安装后，你可以使用一个与 Simulan 配套的技能。所以你只要……嗯，使用这个技能就行。比如我可以说：“访问 AI Engineer 网站，总结一下活动信息。”好，我们来看看它是怎么运行的。现在它正在检查版本。这些是一些日志信息。不过它确实明白了，接下来会使用 Simulan 打开 AI Engineer World's Fair 网站。quick demo there. So let me Okay. So now I'm in a cloud code. So if you install um maybe I should show you simul Yeah. So these are the the landing page and if you go there there's a simple oneline installer npm installer. Once we install, you should be able to use it on your own terminal. And I'm a claw code user. So I like to see because no one is writing code nowadays, right? It's important for you to use V coding to to do this kind of automation. So once you install there's a skill that you can is paired with Simulan. So you can just uh you know use the skill. So I can say go to AIE website and summarize the events. Okay. So let's see how it works. So now it's checking checking version. So these are the logistic stuff. But it does understand it's going to use simulan to open AI engineer words fail website continue.

所以你可以注意到，它在尝试使用 Simulant JS，这是一种 JavaScript 语言。现在克劳德要尝试使用 Simul 协议编写一些代码。是的，我觉得这是因为克劳德是新的，所以它需要处理很多后勤事务。我允许它读取所有内容。So you can notice is it's try to use a simulant JS is a JavaScript uh language. And now clock is gonna try to write some code using the simul protocol. Yeah, I think it's because the the clock is new. So u it needs to do lots of uh logistics stuff. I allow reading for everything.

编写并修复工作流Write and repair a workflow

所以你仍然可以这样想……这是一种不同的计算机处理方式。它基本上是把计算机问题简化成一个编码问题。我有需要的东西。让我写个脚本来打开网站并提取其中的内容，这样你就能看到智能体正在编写代码。之后，它会自动在你的电脑上打开网站，也就是说，类似的流程是在你自己的电脑上运行的。如果你有虚拟机，也可以把它部署到虚拟机上。然后你知道，Claude 运行代码后，会识别出一些错误，发现错误后就会尝试修改代码。所以，如果你启用了自动批准之类的功能，这基本上已经是一个具有某种程度自我改进能力的计算机智能体了。我确实想看看那个……它还不在那里。So you can still think about this this uh this is approach. It's a different approach for computers. It's basically reduce computers problem into a uh coding problem in there. I have what I need. Let me write a script that opens the website extract the content in there and then you can see the the agent is writing the code. After that it opens the website automatically on your so so similar line is running on your own computer. If you have a virtual machine you can also deploy virtual machine there. So there's a and then you know after after clock run the code it recognize some errors and after it gets the errors it try to revise the code. So this is pretty much if you enable you know like auto approval this is pretty much already um a self-improving uh computer agents there some level of self self-improving there I do want to see the it's not there

读取无障碍快照Read an accessibility snapshot

你可以看到，claw 代码决定使用无障碍快照，而不是屏幕截图，因为你知道，这是一个摘要任务。好了，结果出来了。它确实做了摘要。没错，它在里面总结了面向 AI 工程师的所有事件。好，那我先讲到这里。我确实想给你们看看完整代码，看看应该会挺有意思的。是的，这就是完整代码。看起来是 JavaScript，和其他任何 JavaScript 都一样，而且它确实会打开一个窗口、获取快照，然后写入文件。好了，我回到演示文稿。and you can see claw code is decided to use accessibility snapshot instead of the screen screenshot because you know it's a summarization task. Okay, it's coming. It gets uh it actually summarize it did. Yeah, it summarized all of the events for AI engineer in there. Okay. So, I'll pause here and I do want to show you the full code which could be interesting to look at. Yeah. So, these are the full code. It looks like JavaScript uh just like any any other JavaScript and it does open a window take the snapshot and write to the file. Okay. So, let me go back to my presentation.

稳定性与适应性Stability and adaptation

好。所以，我一直在谈论两种截然不同的方法。它们彼此正交，并不是同一种方法。但问题是，我们真的需要两个框架吗？对吧？所以，它们试图相互竞争。这又回到了深度学习领域一个非常有趣且重要的话题。因为我在 DeepMind 的时候，专门研究一个叫作持续学习的问题。所以我们在那里对 AI 系统做了很多研究，但最后我们意识到，没有什么万能良方。无论采用什么技术，仍然必须权衡解决这两个问题：任何学习系统都需要学会新东西，同时也需要确保能够反复完成已经成功完成过的任务。Okay. So, so I I've been talking about like two completely different approach. They're orthogonal. They're not the same same approach. But the question is do we actually need two frameworks? Right? So, so they they try to compete with each other. And this goes back to like a like a very uh interesting and important topic in in the whole deep learning because uh when I was in deep mind I specifically focusing on this problem called continual learning. So that's why we did a lot of study on like uh AI systems in there but in the end we realized okay there's no silver bullet like yeah the the when whenever technology you have uh like you have still have to solve the two problems as a trade-off like the the learning any learning system needs to learn to new things and any of the learning learning system needs to you know make sure you can accomplish the the task that you you you're already successful uh repeatedly.

所以这就成了所谓的稳定性与可塑性困境。你希望学习系统保持稳定，但同时又希望它具备适应新环境的能力。你不可能两方面都做得很好，这就是结论。是的，这始终是一种权衡。那么，我们可以思考的问题是：该如何在两者之间取得平衡？So this become it's called stability and plasticity dilemma. So you you want the learning system to be stable but at the same time you want the learning system to to have this capability to adapt uh to adapt to new environments in there. It's never be the case that you you are both you you're good at both. So that's the conclusion. Yeah, it's always trade-off. Then then the problem we can think about is how do we balance the two in it?

如果你看看我刚才演示的这两个系统：它们分别是编码智能体和 VM 智能体，对吧？它们基本上代表了这类学习系统的两个极端。编码引擎会生成代码，而且始终是确定性的，但它有点固执，对吧？如果你每天都运行一段代码，一旦环境发生变化，代码就会出问题。这就是问题所在。VM 引擎则有一定的适应性，但问题是，你不能指望 VM 智能体每次都能重复同一个流程。所以，核心问题是：我们能不能在两者之间找到一个平衡点？因为我们的终极目标基本上是兼顾最佳适应性和最佳可靠性，对吧？我们也可以把这条曲线称为帕累托前沿。我们能不能真的找到其中的最佳平衡点？ 这也就成了我们公司目前专注的方向：尝试找到一个统一框架，把这两者结合成一个单一的原则，我们称之为神经符号。也就是说，能不能把这两种技术融合进一个统一框架，让它能够持续自我改进，在需要可靠性时足够稳健，在环境发生变化时又能适应？我不会说我们已经解决了这个问题，但我会说，我们已经迈出了朝这个方向前进的第一步，因为现在已经有一条明确可行的路径：我们有一个 VM 智能体，在一个小型基准测试中表现优于人类，也有一个编码协议。And if you look at the two system I just demonstrated these two systems are coding agent and VM agents right they're basically representing the two extreme of this learning system the coding engine produce the code and it's determinance all the time but it's kind of stubborn right if you run a code every day if the environment changes then the code breaks so that that's the problem and the V engine is kind of adaptive but the problem is you cannot rely on the V agents to uh you know like repeat the process all the So the core question is okay can we find a middle ground between the two because the north star is basically the best adaptivity and the best reliability right can we also call this this curve we call this curve parental front and can can we actually find a the sweet spot there so this becomes the current direction that our company is specifically focusing on like basically try to find a unified framework that we can combine these two into one single single principle which which we call neurosyic right because there's a neuro then symbolic is there a way that we can actually like fuse these two technology into one single framework that it becomes self-improving all the time and becomes robust when when reliable when it's needed becomes uh adaptive when environment changes so I wouldn't say we solved the problem I wouldn't say we solved the problem but uh I would say like we we are already at the initial phase of moving towards this direction because there's a clear paradigm that we can do right now because we have we have a VM agent that's better than human in a small benchmark and we have a coding protocol

重放代码并从变更中恢复Replay code and recover from changes

这样我们就可以使用编码代理实际编写代码，并让协议在其中生成确定性的工作流。这是我们进行这项新实验的一种简单方式。基本上，我们让 VM 代理在环境中运行，代理不断把代码写入文件。一旦这个工作流成功，你就有了一段成功运行的代码。下次运行代理时，你可以先运行这段代码，而不是运行 VM 代理，对吧？如果它不起作用，比如环境发生变化导致它失效，就像我刚才演示的那样，你实际上可以通过编码代理构建一个修复循环。这就变得很有意思了。嗯，我加了这一页幻灯片。that we can actually write a code using coding agent and the protocol can produce deterministic workflows in there. This is a simple way for us to do this new experiment. So basically we we let the VM agent to run in the environment and the agent keep writing the code to a file and once this workflow is successful you have a you have a you have code that's that's that's successful and the next time you run the agent you can run a code first instead of run a VM agent right if it doesn't work if the environment changes like when it breaks uh as as I just demonstrated there's actually a healing loop that you can do with the coding agent. So this this become very interesting and uh yeah I added this slide.

所以，另一个挺有意思的故事是，今年的 Cow Hacks，也就是 Sam 黑客马拉松，第一名也使用了我们的智能体。不过这次他们不是用智能体来开发产品，而是用它来测试产品。这是另一个大家都可以关注的有趣方向，因为我们都是开发者，对吧？我们写代码，也知道如今 PR 越来越多，代码也越来越多，那么谁来做测试呢？计算机操作智能体就是解决方案。简单来说，这种智能体可以帮你移动鼠标、点击仪表盘上的内容并进行测试。我不确定自己有没有时间。如果有时间的话，我真的很想展示一个关于测试环节的演示，不过我可以试试。So basically another interesting uh story is this year's cow hacks the the Sam hackathon the first place also use our uh our agent but this time they they're not using the agent to build a product they actually use the agent to test the product which is another interesting category that everyone can look at because all of us are developers right we write code and we know like nowadays there are more and more PRs more and more code then who's going to do the testing then computer use agency is a solution for that So basically this is the agent that can just move in the mouse click on the uh on the dashboard and doing a testing for you. I'm not sure if I have time. If I have time I really want to uh show a demo about like the the testing part but I can I can try

启用实验性回放模式Enable the experimental replay mode

让我试试，嗯，让我试着给你们看看我们产品里的这个循环。这是一次初步尝试。在很多情况下都能用，但有时还是会出现一些小问题。所以，嗯，我很期待大家测试这款产品并给我们反馈。好，那我来……不确定这个是不是……哦，是这个。好，这是 Value VM。这是 Agent，不是……你看，我这里有很多 VM，而且我给它们起了不同的名字。嗯，那我试试这个。所以，要启用神经符号模式，你需要进入这个控制面板，点击 Agent。我们这里做了很多特殊算法，比如拟人化控制（轨迹会有所不同）、临床集成等。所以你需要查看 Sim Flow Code Replay。这是我们目前正在做的一个具体实验。整个思路基本上就是我刚才展示的那个示意图；如果你启用它，我们就可以试试某些东西。let me try um let me try to show you the loop that we have in the product. It's a it's an initial effort. It works in a lot of cases, but sometimes there are still some hiccups in there. So, um I do look I do look forward to everyone like testing the product and give us feedback in there. Okay. So, let me I'm not sure if this one is doing Oh, this one. Okay. So this is the value VM. This is the agent is not you see I have I have a lot of VMs here and I I do put different names to to them. Um so let me try this one though. So in order for you to enable the neuros symbolic mode you need to come to this dashboard click on agent there are a lot of special there are a lot of special algorithm we do uh here like humanized control like the trajectory is different uh clical integration so you need to look at the sim flow code replay so this is a specific experiment we're doing right now so the whole idea is uh basically the diagram I just showed uh if you enable that we can try something.

让我确认一下里面的桌面。好。那么，我们可以试试这个。对，我们还是可以试试简单的方法。只是想让你大致了解一下它是怎么运作的。打开 AIE 网页，嗯，找到星期二的第一位演讲者。现场演示时总是会出问题，嗯，让我重启 agent。Let me ensure the desktop in there. Okay. So, we can try this. Yeah, we can still try the the simple thing. Just want to give you a rough idea of how this works. Go to AIE web page and um find the first speaker on Tuesday. There's a always has problem during live demo in let me restart agent.

旧版表单自动化示例Legacy form automation example

我看看我的 VPM 有没有醒着。嗯，我们等着的时候，我再给你们看另一个演示，因为我这里有个备用演示，以防现场演示耗时太久。这样我们可以展示另一个。这个演示是和微软合作的。微软也内置在这套计算机基础设施中，因为他们是 Windows 的创建者。这些是微软的虚拟机，叫作 Microsoft Windows 365 for agents。你们可以从左侧的图表看到它的构想：在微软虚拟机上使用我们的智能体，来填写传统 ERP 系统中的表单。Let me see if my VPM is awake. Yeah, while we're waiting. So, let me show you another demo because I al I have a backup in there if live demo needs no longer time. So, we can show another one. So, this one is a collaboration with Microsoft. Uh so Microsoft has uh Microsoft is also built in this computer infrastructure uh because because they are the creator of Windows. So these are the VMs from Microsoft. It's called Microsoft Windows 365 for agents and the idea is from this you can see like on the left side the chart. So the idea is okay like use uh use our agents on Microsoft virtual machines to to fill out the form on a traditional ERP.

所以这算是计算机使用代理最大的应用领域之一，因为仍然有很多传统行业没有现代 API。你知道，他们还在 Windows 电脑上运行各种老旧软件，对吧？所以，如果你开发出能服务于某些垂直行业的应用——因为我们不是垂直行业公司，而是提供横向基础设施——如果你有兴趣针对不同的垂直行业进行开发，这些都是很好的机会。比如，我们的一些客户与保险网站合作，专门从事保险业务。他们基本上会让代理访问保险网站并获取数据。还有一些客户来自医疗行业，他们与医生合作；医生使用装有 ERP 系统的 Windows 电脑，而这些系统都非常老旧。你可以用计算机使用代理来自动完成所有数据录入工作。So this is one of the biggest area for computer computer use agents because uh there's still so many traditional industries and they don't have modern APIs they don't have like you know they still running all the old um softwares on their Windows machines right so if you have if you build out any like applications that allow you to go to certain verticals because we are not vertical company and we are horizontal infrastructure if you're interested in building like towards different verticals these are good opportunities Um like for example some of our customers they uh they work with insurance website they specifically working on insurance. So basically they use the agents go to insurance website and uh fetch the data and some of them are healthcare u they work with doctors they have uh like doctors has windows uh machines with ERP system very old and you can use computeriz agents to automate all the data entry.

所以，人们一直在这个方向上探索很多想法。嗯，褪So there's lots of ideas that uh people has been uh exploring in this direction. So, um,

寻找会议演讲嘉宾Find a conference speaker

是的。所以这些内容希望能给你一些启发。哦，能用了。对。所以，我想我已经有一些现成的技能了。这里有一个技能，我把它移除。打开 AIE 网站，看看星期二谁是第一个发言的。我已经有网站了。那我来创建一个挑战，创建一个新的。你可以在这里看到所有日志。有意思的是，你可以看到这个代理几乎是逐行编写 Selenium 代码。之后，它会把所有代码组合到一个大文件里。导航到日程页面。好，我们拿到了信息。如果你注意到了，这个代理实际上并没有截图。所以，每当你试图查找信息时，代理都会决定这次是应该截图，还是应该获取无障碍树；这次它实际上选择获取无障碍树。yeah. So, those are hopefully that's that that'll give you some insights there. Oh, it works. Yeah. So, so let me because I guess I have some skills that's already there. So, there's a skill here. Let me remove it. go to AIE website and see who's speaking first on Tuesday. I already have website. So, let me make a challenge. I create a new one. And you can see all of the logs here. And the interesting part you can see this agent is writing uh almost like line by line simulan code. And after that it's going to com compose all of the code into one big file. Navigate into the schedule page. Okay, so we get the get the information and if you notice the agent is actually it's not a screenshot. So whenever you try to find information the agent actually decide should I take a screenshot or should I take accessibility tree this time actually take access accessibility tree.

这样你就可以在那里一眼看到下面的所有信息。我能做的是……So you can see all the information that beneath in one shot in there. And what I can do

将工作流保存为一项技能Save the workflow as a skill

现在，因为我想给你展示如何保存到一个技能里。所以，如果我启用那个 Neurosy 模式，然后说“保存到技能”，这个技能就会包含你刚才在上一个会话中运行的所有代码。然后我们可以看看 AI speaker on day。我来找找有没有这个技能。不过我没找到。now because I want to show you that uh save to a skill. So if I enable that neurosy mode and I say save to skill, this skill will have all of the code that you just run in a in a previous session. And then we can look at AI speaker on day. Let me find a skill there. I didn't find it though.

重放另一天的内容Replay for a different day

好的。所以这个技能已经保存了。我来试试另一个查询，换一天试试。我们试试星期三。你可以看到，重放模式出现了。界面有点不一样，因为它确实包含代码。它有代码，而且还更改了代码中的参数。如果你点击活动日志，就能看到代理在第一步实际上生成了更多代码。这些就是代码，里面有不同的参数。U 默认是星期二，对吧？默认值是星期二。如果你把它设为星期三，它就会更改那里的参数，然后出现一个小故障。你可以看到，代理需要在里面进行筛选时出了点小故障。它确实找到了信息，并成功进行了重放。中间出现了一点小故障，然后代理在故障之后将其转换成了神经模式，也就是 VM 代理。Okay. So this skill is saved. So I'll try a different query. on a different day. Let's try on Wednesday. And you can see there's a replay mode showing up. The interface a little bit different because it it does have the code. It has the code and it changed the parameter of the code. And if you click on the activity log, you can see the agent is actually producing a lot more code in the first step. So these are the the code. It has like different parameters in there. U by default is Tuesday, right? The default is Tuesday. If you set it Wednesday and it's going to uh change the parameter there and there's a hiccup. You can see there's a hiccup for the the agent need to filter in there. It does find the information and replay successful in there. there's a little hiccup and then after the hiccup the agent uh convert it to the neural mode which is a VM agent.

所以一开始它总是会尽可能多地生成所有代码，因为如果命中率相当高，就能很稳定地成功完成任务，而且智能体还会在中间进行验证。所以每次运行代码时，都会进行验证。如果有问题，就停下来。这有点像我以前上学的时候，我们会把老师说的所有内容都记在备忘录里，当作一本操作手册，对吧？下次我想做什么的时候，就会翻看备忘录，试着一步一步完成所有步骤。如果环境不一样，我就会发现，哦，我试着执行这个操作，但环境不允许我这么做。然后我就得多想一想，花更多时间去探索新环境，并试着弄清楚什么才是最好的策略。So the first time it always try to produce all of the code as much as possible because if you have a pretty good hit rate it's going to have a very determin successful overflow and then the agent also have verification in the middle. So every time it runs the code, you have verification. If there's something wrong, stop the it's kind of like it's kind of when back when I was uh when I was in school, we we write everything that teacher says in the in the memo as a playbook, right? And next time when I want to do something, I look at my memo and try to do all the, you know, the steps step by step. And if the environment is different, I notice, okay, I I try to do this action, but the environment doesn't allow me to do that. And then I have to like think more spend more time on you know like exploring the new environment and try to figure out like what's the best strategy to do.

好，那我们继续。我认为这是一个很好的方向，不仅我们，整个行业也都可以思考：如何在灵活性和可靠性之间取得平衡，打造更好的智能体。我们确实在这类实验中取得了相当不错的成果。就是，好……Okay. So let's move on. Uh I believe that's a good direction for us to for even for the whole industry to think about like what's the what's the better um agents that we can produce with this kind of balance of flexibility and reliability. And we do see pretty good uh success in this this type of experiments. It's like okay

复用工作并减少模型调用Reusing work and reducing model calls

我不……我觉得，重点是，借助这类技术，你不必每次都花费同样数量的 token，因为，呃，第二次运行工作流时，可能有 50% 的工作已经由代码完成了，不需要发起任何 LM 请求，对吧？而且，呃，你可以大幅降低成本。所以我们也在这个产品中做过实验，也尝试过向用户推出比如 500 美元的无限任务套餐。因此，价格和套餐也可能会改变，不会永远都按用量计费。呃，因为你想想，作为一个人，我是人类。当我尝试做某项任务 100 次后，就会对它非常熟悉，基本上闭着眼睛也能把所有事情做好，对吧？但如今的智能体还不具备这种能力。每次做同样的任务，即便你做了 1,000 次或 10,000 次，哪怕是同一个查询，你仍然要向 Anthropic 和 OpenAI 支付同样的钱，这似乎不太合理，呃，也不经济。I don't I think the I think the main point is you don't have to spend the same amount of tokens each time anymore with this kind of technology because uh because the second time you run the workflow maybe 50% of the effort is done by code without any LM request in there right and uh you can you can dramatically reduce your cost so that's why we also experimented in this product we also experiment with people on like $500 unlimited task plan. So the price and plan may also change. It's not going to be like uh usage based forever. Uh because you think about like as a human, I'm I'm a human. When I try to do a task a 100 times, I become so familiar with that. I can pretty much do all the things with my eyes closed, right? But today's agents, they don't have this capability. every time you do the same task, even you do a thousand times or 10,000 times for the same task, even the same query, you still pay the same amount of money to anthrop and open, which doesn't seems to be like the the reasonable uh is it's not economical as well.

是的，所以，呃，希望这能让大家比较清楚地了解我们接下来要朝哪个方向发展。呃，我也鼓励大家都试一试，因为我们正在积极、快速地改进这些神经符号智能体，呃。And yeah, so uh hopefully that that gave you a pretty good sense about what direction we're we're going forward. And uh I do encourage everyone to try try it out because we we actively rapidly improving that neuros symbolic agents uh

为什么智能体需要独立的计算机Why agents need separate computers

在接下来的几个月里，呃，希望我们能在这个方向上取得一些突破。我们要考虑的第二个问题是：我们该怎么处理电脑？两年前，我们在研究电脑上的自主运行技术。基本上，我们曾经有一款产品，可以在你自己的电脑上运行智能体，类似于叫作 co-working 的产品。但我们收到了很多用户投诉，因为每个人只有一台电脑，而智能体运行时会占据前台，你就什么也做不了。是的。所以这实际上促使我们思考：假设我们有一个达到人类水平的智能体能够运行。假设我们有了一个可用的东西。它还没达到那个水平，但已经非常接近了。等到那一天，我们有了能正常工作的智能体，就得考虑该怎么处理电脑：你还需要一台电脑吗？还是说你可能需要十台？in in a in the coming months and uh hopefully we can achieve some like a breakthrough in that direction. And the second second problem we have to think about is okay like what's the what do we do with with the computer? So two years ago we we work on this autonomy uh technology on your computer. So basically we have a we we used to have a product that runs the agent kind of like called co-working on your own computer but we receive a lot of complaint from users because everyone has only one computer and then when the agent is working on it it's in foreground you cannot you cannot do anything. Yeah. So so that actually motivates us to think about okay suppose we have a human level agent uh that works. Suppose we have something that works. It's not it's not there yet. We we're very close but when that day comes we have agent that works then we have to think about what do we do with the computer like do you still need one computer or maybe you need 10.

所以我们目前观察到的情况是，只要有东西能用，用户总会问我们：“你们能不能、能不能真的把这个智能体部署在一台单独的机器上？能部署在服务器上吗？”所以大多数用户最后得出的结论总是一样的。然后他们就会开始要求更多机器。这是一个发展轨迹。这是一个发展轨迹。起初，我们处于一个努力让智能体真正运行起来的阶段。一旦它能运行，你就需要 10 台机器、100 台机器。这就是促使我们开发云端计算机的原因。嗯，我、我……So the current observation is whenever you have something that works, users always ask us to can you can you actually host this agent on a separate machine? Can you host on a server? So this is always always the same conclusion from most of the users. And then they will start to ask for more machines. So there's a trajectory. There's a trajectory. First we are in the in a in a phase that we really try to make agents works. Once it works, you need 10 machines, 100 machines. That's why it motivates us to work on the cloud computers. Um I I

云桌面与开发者访问权限Cloud desktops and developer access

简单说一下，我之前已经简单谈过这个了。我觉得关键在于，如果我们有云端托管桌面，它就会变得非常容易扩展。最终，我希望每个人都能拥有 100 台电脑，这样你就会成为自己 AI 组织的 CEO，对吧？里面有 100 名 AI 员工，每个人的生产力都能提升 100 倍。 所以，基本上，今年我们在 3 月推出了一个产品。它其实是个相当新的产品，叫做 Sidebox。它里面有一台云端虚拟电脑。从那个演示中你们已经能看到，我个人有 20 台机器在后台运行。这是一张总体示意图：S 实际上是界面，而 Sausent 负责自主运行部分，进行规划和决策，然后在 Sidebox（也就是云端电脑）上执行操作。好，然后我们还有很多开发者朋友。briefly talk I already briefly talked about it. I think that the whole point is if we have the cloud managed desktop uh it becomes so scalable and everyone in the end I hope everyone can have a 100 computers so you become the CEO of your own AI organization, right? So you have 100 AI employees in there and everyone become 100x productive. So yeah, basically this year uh we we launched a product side in March. Uh it's actually a pretty new product and together with that is called side box. is the virtual computer cloud in there and from the from from that demo you can already see I have personally I have 20 machines uh running in the background in there and uh this is a general diagram s is actually the interface and sausent which uh do which does is auton autonomy part does the planning and decision and then doing operations on the sidebox which is a cloud computer and okay and then we have a lot of friends who developers.

所以他们一直在问我们：“你们能不能开放 API？”今天我们确实发布了用于访问云电脑的 API。我想我没时间演示了，不过它其实非常简单。它叫 SAPI，S API，SAPI。基本上，你可以通过命令行工具，在终端里完成所有我刚才在 Web 门户里做的操作；里面也有 API 工具。我们刚刚发布了这个网站，欢迎大家去看看，里面有很多 API 和 CLI 工具。注册后，你就会拥有一台远程环境中的机器，可以使用命令行和 API 来控制这台电脑，并将其集成到你自己的产品中。[吸鼻子]好吧，我就不做现场演示了。那么，我们在这个方向上的整体愿景是——So they keep asking us so can you actually expose the API. So today we actually uh releasing the API for uh accessing the uh the cloud computer. I don't think I have time to demo but it's actually pretty simple. It's called SAPI S API SAPI and basically you can do everything that I did in the web portal through your terminal using the CLI tools and you can also use there's also API tool in there and uh we just released this website so feel free to check it out there's a lots of APIs CLI and once you sign up you have a machine in the remote uh remote setting you can use this command line and API to uh control that computer and integrate to your own product. [snorts] Yeah, I'll skip the live demo. So the the general vision for this direction is

多个平台和 Android 演示程序Multiple platforms and Android demos

我们希望每个人都拥有一个统一的智能体，这个智能体可以访问多个平台，而不仅仅是 Windows。我们也支持 macOS。Linux 即将推出。Linux 非常经济实惠，定价相当便宜，这是根据我们的内部评估得出的。有趣的是，我们也支持 Android，所以 Android 也即将推出。我希望将来你能在云端拥有几部 Android 手机和几台 Windows 设备，让智能体能够在所有设备之间进行编排，完成各种不同的任务。我想展示一些有趣的 Android 演示。这些都是内部评估内容。这个演示是在玩《愤怒的小鸟》。有趣的是，它实际上会拖动。让我看看。we want we want everyone to have one single agent and this agent can access multiple platforms not just Windows. So we also have uh uh support on Mac OS. The NUX is coming soon. Lux is very uh economically friendly. is it's pretty cheap uh in terms of our pricing uh like based on our internal evaluation and the interesting part is we also have Android. So Android is also coming soon. So I I do hope like in the future you have a couple Android phones in the cloud you have a couple windows and the agent actually can orchestrate among all of the device to doing all different task uh in there and I want to show some interesting demos for the Android. So these are all internal uh like evaluation. So this one is a play in the Angry Bird game. And the interesting part is actually drag. Let me see.

是的。是的，你确实可以拖动，你知道，它支持拖动操作。他在屏幕上拖来拖去，并在里面执行任务。另一个我觉得很酷的演示是身份验证器。所以你可以打开身份验证器，呃，里面的界面是一样的。基本上，我们把 Windows 虚拟机换成了 Android 虚拟机。让我加快一点。好，它会打开这个 MFA 身份验证挑战网站，在那里填写表单，然后获取令牌。对，然后就成功了。所以，这是人们一直问我们的一个问题：我们能不能进行双重身份验证？答案是仍然可以。我确实觉得这有一定价值，但使用时必须谨慎。这是一个示例，我相信是手机上的社交媒体应用，你可以在上面执行各种不同的任务。是的，这挺令人兴奋的。我觉得 Android 是一个很有意思的领域，目前业界还没有怎么探索过。你知道，我们应该可以通过 Android 虚拟机实现很多应用场景。Yeah. Yeah, you actually drag, you know, it has a drag actions. He drag drag the screen and doing a task in there. And another demo I feel pretty cool is authenticator. So you actually go to authenticator and uh it's the same interface inside. Basically we swap Windows VM to Android one. So let me make it faster. Yeah, it goes to this M MFA authentication challenge website and fill out the forms there and then get the token. Yeah. And then success. So, so this is this is one of the problem that people keep asking us like okay like can we do 2FA? still possible and I do I do see there's a there's some value but we have to use it in with like uh c caution there and this is a example for I believe the social media on the phone u like you can actually do different kind of task there yeah it's pretty exciting I I feel the Android is a interesting area that like the current industry hasn't really like explored much you know there there should be a lot of applications that we can do with Android virtual machines.

所以基本上现在所有东西都汇聚到一起了。我一开始从自主计算机这个概念出发，把所有东西拆分成自主性和计算机两部分，然后自主性又分成两部分：神经部分和符号部分。计算机则分为本地机器和云端机器。所以，S 实际上两边都适用。你可以把自己的机器整合进来，成为其中机群的一部分；也可以购买我们的云端计算机。然后所有这些就变成了 Site，它于今年 3 月发布。我对这款新产品非常兴奋，希望它能带来很多新的机会，因为这是我两年前完全没想到的事情。我们一直在开发这项技术，当时并不太清楚它能应用于哪些垂直领域。但后来我们意识到，比如有一家 YC 初创公司在某场活动上见到我，他们告诉我，他们实际上正在把智能体用于自己的垂直领域应用。我确实看到了巨大的机会。接下来一年，或者说明年，这项技术可能会相当成熟。按照之前的估计，明年会很有意思，值得看看会发生什么。So basically everything is converged right now. So I started with autonomous computer and I decompose everything into autonomy and computer and then autonomy becomes two parts the neural part and symbolic part. Computer become local machines and cloud machines. So s actually serve for both. So you can actually like integrate your own machines to be part of the you know the fleet in there and you can also buy our cloud computers and then everything become site which is released uh in March this year. I'm pretty excited about this new product and I hope uh that will bring a lot of new opportunities uh u because this is something like I didn't imagine like two years ago like we work on this technology we don't know much about like what the vertical application we can do but we actually realized okay like for example there's a YC startup uh saw me in one of the event and they tell me they actually use agents for their own vertical applications so I do see there's a huge opportunity that within like in the Next year could be in the next year the tech technology will be uh relatively based on the the past estimation the next year will be very interesting to to see what's what's

开发者工具与研讨会总结Developer tools and workshop wrap up

即将发生什么，之后我可以想象，在一年之内，这个领域会涌现出很多新的应用。总之，简单总结一下，我们为开发者提供了四项服务。首先是 Sidework，基本上就是完整的界面。你可以进行探索，在开发者模式下查看不同的算法，试用一下，然后告诉我们效果如何。我们也在积极开发新技术。Agents Simulan 是开源的，任何人都可以使用，而且基本上是免费的。Sidebox 则是面向虚拟桌面的 API，我们今天刚刚发布，所以可能会有一些漏洞，请告诉我们。今天我会给大家邀请码：AIE EWF 2026，欢迎试用，并告诉我你的反馈。我们确实有一个积分计划，也很乐意以不同的方式、不同的合作形式展开合作，看看未来如何帮助你们改进开发工作。going to happen and then what I can imagine there are a lot of uh new applications like blossom in in in this this domain in within a year yeah basically to summary we have four uh offering for all the developers side work is the basically the whole interface you can explore you go to the different algorithms in the developer mode give it a try let us know how it works and we actively developing new technology in there agents simulan which is open sourced and everyone can use it's pretty much free uh free to use and sidebox is the API for the for the virtual desktop and we just released today so expect some bugs uh do let us know uh and I I'll give you the invitation code uh for uh today is aie EWF 2026 and please give it a try. Let me know uh the feedback. We do have a credits program like uh happy to collaborate in a diff in different way different capacity and see how to you know like uh make your development better uh in the future.

好的。非常感谢。我想留些时间回答问题。大家可以随时提问，也谢谢大家的耐心。我会确保把照片发出来。如果你们希望我在帖子里提到你们，请告诉我。大家尽管提问，我会在这里再待几分钟。我知道那边还有其他活动。All right. Thank you so much. Uh I I want to leave some time for questions. So feel free to uh ask me questions in there and thank you for your patience. I I'll make I'll make sure the photo is posted. So, let me know if you want to be mentioned in there. Yeah, feel free to ask any I I'll stay here like for a few minutes. I understand like uh there's another events going there.

屏幕理解与记忆问题Screen understanding and memory questions

主持人：嗯。反正是问题记忆。Host: Yeah. Question memory anyway.

嘉宾：空间记忆。Guest: Spatial memory.

主持人：是的。对于 VM，它在视觉上看起来是什么样的？Host: Yeah. for VM how it visually looks that

嘉宾：目前，大多数记忆都被简化成文本了，是的。正如我提到的，在 Agent S2 中，我们确实有一个单独的模型，所以它实际上并不是一个大型视觉语言模型的简单封装。我们还使用其他 70 亿参数的开源模型，有时也会对这个模型进行微调。这个模型专门负责识别屏幕。有意思的是，我们知道基础模型进步得非常快、表现也越来越好，但它们仍然不太擅长理解屏幕。这是个挺有意思的结论。可能是因为互联网上可用的截图数据不多，所以它们没有足够的数据来训练模型。不过我能想象，这类模型会变得越来越好。Guest: for now more most of the memory are reduced to text in there yeah and we do have as I mentioned in agent S2 we do have a model separate model so it's actually not a simple wrapper of a large vision language model so we have other 7 billion open source models sometimes we also like fine-tuning that model and that model is specifically responsible for recognizing the screen and the interesting part is we know the foundation model is progressing so so fast and so good but it's still not very good at understanding the screen. So that's an interesting yeah interesting conclusion. Probably it's because the internet doesn't have many like screenshot data uh available. So so it's it's not available for them to train the models in there. Um but I can imagine the model get getting better and better.

但总感觉始终存在差距。如果，如果你有一个专门针对自己应用的模型，就可以对它进行微调。它总会比那些大实验室推出的通用模型更好。对。问题。But it's just like there's always a gap. If if you have a model that can specifically for your own application, you can fine-tune that model. is always going to be better than like uh like a general model from big labs in there. Yeah. Question

主持人：几个问题。Host: couple questions.

嘉宾：是的。Guest: Yeah.

状态与多种输入模态State and multiple input modalities

主持人：你经历过这个过程吗？Host: Have you experienced the process?

嘉宾：是的。所以我先回答你的第一个问题。呃，我展示的演示是由 LM 生成代码，这就是我所说的无状态编程方法。过去 20 年里，我们一直都在进行无状态编程。而我所说的神经符号方法，是指有状态编程：也就是说，每次运行一行代码时，你都会查看环境、查看状态，而状态实际上会发生变化。我相信，这将是未来编程的一种新范式。未来的程序会更加稳定。呃，我们尝试过 LM，而问题本质上是多模态的，对吧？因此，可访问性可能具备一些能力，但仍然如此——没有什么灵丹妙药。有些人认为，只用屏幕截图就能构建计算机代理；有些人认为可访问性已经足够；还有些人认为 DOM 已经足够。但归根结底，经过所有评估后，我们发现没有一种方法能解决所有问题。Guest: Yeah. So, so let me answer answer your first qu first question first and uh the demo I showed is LM right generating the code and this is this is the approach I call a stateless programming. So all of the past 20 years we all do stateless programming and the neurosyolic approach I call it stateful programming meaning when you every time you run one line of code you look at the environment look at the state and the state actually change. So this is I I believe this is a new paradigm for programming in the future. So program will be more stable in the future and uh we we tried LM and the problem is this problem is essentially multimodel right so accessibility can have some cap capability of you know like some but it's not it's still the same there's no silver bullet so some people argue you can do computer agent just for screenshots just using screenshots some people say accessibility is good enough some people say DOM is good enough but in the And after all the evaluation there's no silver bullet on everything.

所以你总会遇到这样的情况：一个系统的边缘案例，对另一个系统来说却不是边缘案例。所以，归根结底，我觉得我们实际上采取的是 Waymo 的方案，同时使用激光雷达、雷达和摄像头，对吧？而有些公司可能会采用特斯拉的方案，比如说：“我只用摄像头。”是的。不过，如果你看看如今的自动驾驶汽车，就会发现路上有很多 Waymo 车辆在行驶，而且可靠性更高。只是因为它们有多个系统相互协作，确保不会出错，尤其是避免致命错误。你可不希望有人去你的银行，在你开支票给别人时多加一个零。第二点。是的。So you always has a corner cases of one which is not corner case of the second. So, so in the end I think we are actually actually taking a whimo approach but you have lighter radar and camera at the same time right and some some company they may take the Tesla approach like I I just have camera and yeah but if you look at today's self-driving car uh we do see a lot of whimo uh running on the road with better reliability there it's just that you have multiple systems that help each other to to make sure they don't make mistakes like especially fatal mistake you don't want You don't want agent go to your bank and add another zero when you send a check to other people. Second. Yeah.

主持人：第二个问题。Host: Second question.

嘉宾：是的。所以，无障碍树的一大优点是，它所包含的信息会用于渲染屏幕。基本上，这些信息几乎涵盖了渲染屏幕所需的全部内容，而且如果你能使用这些信息，结果肯定会可靠得多。不过还是有一些边缘情况，比如应用的开发者可以有意规定，某一部分不要暴露在无障碍树中。所以，你仍然可以通过截图捕获那部分内容，但无法通过无障碍功能捕获。你可以想象，有些应用会这么做。比如，LinkedIn 有很多机器人检测机制。我以前常跟朋友开玩笑说，LinkedIn 就是 AGI 的图灵测试。是啊，因为如果 LinkedIn 没把你的智能体识别成智能体，那你基本上就实现了 AGI，因为你通过了图灵测试——毕竟他们处在对抗性环境中，而且有很多检测技术。这种无障碍功能也可能是他们使用的检测手段之一。Guest: Yeah. So uh the good part for accessibility tree is that information is used to render the screen. So basically that information has pretty much all of the information that's needed to render a screen and if you can use that information is definitely more reliable. But there are some uh there still corner cases like the developer of that application can intentionally say this part do not expose in the in a tree. So, so you can still capture that in a screenshot, but you don't capture in accessibility. But you can imagine some applications like uh for example, LinkedIn has a lot of B detection there pretty much the I used to tell a joke with my friends like LinkedIn is is the Turing test for AGI. Yeah. Because Yeah. If if LinkedIn doesn't recognize your agent as agent, then you pretty much achieve AGI because you pass the Turing test in because they they're in adversarial setting and uh they have a lot of techniques to detect and like this kind of accessib could be one of them in there.

是的。所以这就是为什么你仍然需要多种模态的组合。嗯，有一个人……我先让那位提问。谢谢。Yeah. So that's why you still need multiple you know combination of the modality. So yeah, there there's one person I I'll yeah I'll let that person ask question first. Yeah, thank you.

主持人：是的，非常感谢。首先，Host: Yeah, thank you very much. First of all,

准确性与验证循环Accuracy and verification loops

客人：所以我们来自德国。Guest: so we're from Germany.

主持人：我们在德国有一个应用场景，你会获取信息，然后用这些信息做很多事情。Host: We have a use case in Germany where you take information and then a lot of things with them.

嘉宾：但问题是，拿医疗保健行业作比较的话，你今天展示的这一切在现实世界中到底有多可靠？因此，你需要非常准确的信息和非常精准的流程。Guest: But the question is how reliable is all of that that you have shown in the real world today when you compare it with the healthare sector for example. So you need very accurate information and very accurate processes

主持人：对。Host: right

嘉宾：你对此有什么看法？是的。所以，基本上这个问题是，我们需要非常准确的信息。也就是说，我们需要做到百分之百准确，对吧？所以，即使是录入数据，也需要做到百分之百准确。嗯，这正是当前基础模型面临的最大问题，不过我们确实有办法解决。基本上，正如你所看到的，一切都是一个循环，对吧？所有基本原则都是如此。当你做某件事时，总会进行验证、批判自己，人类也是一样。所以，当我向朋友提出一些想法时，他们会提出批评，这样我就能形成一个循环，从而更少带着偏见进行推理。准确性问题也可以用这种方法解决。Guest: what is your opinion on that? Yeah. So, so basically the question is so we need a very accurate information. So basically we need everything to be 100%. Right. So even when you do data entry you need to be 100%. So uh that problem is the biggest problem of the current foundation models but we do have a solution for that. So it's basically uh as you can see everything is a loop right. So all the all the fundamental principles everything is a loop. when you do something you always verify criticize yourself and it's the same for humans. So when I when I propose some ideas to to my friends they they criticize so I have the loop so I can like reason with less bios in there. So the the accuracy problem can also solve by this approach.

我们确实做过一些实验。我可以告诉你，我们采用的一种方案是：每当你填写完一个表单，就把表单反过来，再问模型这样是否合理，让模型进行验证。大多数时候，模型都能发现其中的问题，因为验证比生成容易得多。发现问题后，你就把结果交回去，让智能体再执行一次相同的任务。这实际上大幅提升了性能。 另外，你还可以考虑无障碍树。对我们来说，要编写代码并让代码能够确定性地重放，只能通过无障碍树来实现，不能通过截图实现，对吧？因为树中的内容是文本，可以做到 100% 匹配。So we we do have some experiments that I can tell you one solution we did is uh uh whenever you have something you fill out the form you take a form reverse it and ask the model does that make sense right so ask the model to verify if the mo most of the time the model recognize something wrong in there because verification is is much easier than generation generation general action and if there's something wrong you take the results back and ask the agent to do the same task again. It actually improved the performance a lot. And um another thing you can think about is the accessibility tree like for us to write the code and for the code to be deterministically replayed you can only do in accessibility tree. You cannot do in a screenshot right because the tree is text can you can do 100% matching there.

因此，这种 100% 的文本匹配也很重要，能确保一切可靠。所以最终的结论是，你必须有一个系统。它不会只是一个模型，你把它放在那里，然后就指望它能达到 100%——这种事不会发生。不过，你的使用场景肯定有解决方案。对。抱歉。轮到你了。对。你是怎么做到的？So that kind of a 100% text matching is also important to ensure everything is uh reliable. So in the end the conclusion is you have to have a system. It's it's not going to be one model. You put it there and then you can hope that's 100% it's not going to happen in there. And uh but there's definitely a solution for your use case in there. Yeah. Sorry. Your turn. Yeah. How do you kind of keep it?

不可逆操作和操作系统限制Irreversible actions and OS restrictions

主持人：是的。所以在产品侧，我们确实设置了一道防护机制。每当系统识别到某个操作是不可逆操作时，防护机制就会在那一步叫停。因此，还有另一个系统在监控这个智能体。如果智能体想要执行某项操作，系统基本上会将其拦截，并要求先获得批准。不过我认为，你们的问题更具挑战性，因为它涉及更底层的操作系统层面。比如，智能体会说要打开某个应用，而你们希望在系统中屏蔽这个应用，对吧？因此，这需要在操作系统层面再增加一层防护。这就是为什么我们认为，使用由 Claw 管理的 Windows 机器是更好的方向，因为我们有虚拟机。如果你们需要，我们也可以为你们提供虚拟机镜像。这样一来，在虚拟机镜像中，你们就有很多机会去修改操作系统底层的结构。Host: Yeah. So, so in the product side we do have a guard rail. So whenever there's a action that recognize that irreversible action the guaro will say stop there. So it's another system another system monitoring this agent. If the agent want to do a certain action it'll basically cut off and say ask for approval. But I believe your your problem is even more challenging is more at the low level of the operating system level like for example the agent will say open this applications and you do want this application to be blocked in there right so this requires another layer uh on the OS level that's why we believe have the claw managed uh Windows machine is a better direction because we have we we have the VM so if you need we can also produce uh the image the VM image for you so So in a VM image you have lots of opportunity to uh change the underlying you know structure of the operating system.

这样你们就可以屏蔽某些操作，并在其中设置白名单。所以我认为这就是最终的方向。呃，VM 这部分我们还处于早期阶段。我是说，我们正在尝试探索这个方向。呃，如果你们感兴趣的话，我们也可以考虑未来开展一些合作，看看如何设置更完善的防护措施。是的，安全防护绝对是其中一个重要方面。谢谢。So you can actually block some of the actions have the white list in there. So I believe that's the that's the final direction. Uh we are still early on the VM part. I we we try to explore that direction and uh like yeah if you're interested like we can also think about some partnership in the future uh on how to have better guard rails. Yeah, it's definitely like a security security guard is definitely like one important aspect in there. Yeah. Thanks.

嘉宾：是的。只是为了保护某种不可逆的东西。Guest: Yeah. Just the protection of an irreversible something.

主持人：是的，是的。所以这也遵循了同一个原则，也就是我一开始写下的那句话：基本上，所有东西都是神经符号式的。如果你希望一切都可靠，那一切都需要是神经符号式的。你总是先用代码，因为代码可以保证百分之百可靠，对吧？代码让人困惑时，你就转向灵活的智能体，然后再回到代码。所以代码仍然非常重要。保障机制的设计也遵循同样的原则。因此，保障机制也采取代码优先的方式。那里会应用很多规则；如果规则无法覆盖某种情况，就交给智能体处理，然后由人来验证，之后再回到代码。这是我们试图在整个系统内部遵循的一项设计原则，甚至系统里的每个组件都如此。谢谢提问。Host: Yeah. Yeah. So, so that that's that goes to the same principle that the first line I put in the so basically everything is neurosymbolic. If you want everything to be reliable, everything needs to be neuros symbolic, you always go to the code first because code can guarantee 100%. Right? When the code is confusing, you always go to the flexible agent and then come back come back to the code. So that's why the code is still super important. It's the same principle for the design for the guar. So the guar is also like code first. there lots of rules happening there and if the rule doesn't capture it goes to agent and then goes to human for verification and then comes back uh it's a it's a design principle that we try to follow in all of the system inside uh even every components uh in the in the system thanks question

嘉宾：是啊，我是说，你知道的。Guest: yeah so I mean you know

主持人：重复循环，会变得更多吗？Host: repeating cycles Does it get more?

嘉宾：对。对。对。所以，这就是发展轨迹。基本上，我们希望 token 消耗最终降为零。没错。事件数量也应该降为零，而且有时你会看到代码实际上开始出现更多分支。Guest: Right. Right. Right. So, so that that's the trajectory. So, so basically we want the token consumption to eventually become zero. Yeah. event should become zero and you can see sometimes you can see the code actually start to have more branch

主持人：展示出来时，你会看到不同的场景。你看得越多，就越像人类学习，对吧？当你看到……呃，你有更多的数据样本时，那么……Host: show showing up when you will see different scenarios you see more it's just like human learning right when you see uh you have more data samples then

遗留工作流和流程Legacy workflows and procedures

在决策方面，你的偏见会少一些。对，这种方法也是一样的。you are less biased in terms of decision making yeah it's the same for this approach yeah

主持人：哦，所以问题是，他们有专门用于遗留程序的专有程序，对吧？Host: oh so question has proprietary program that they for legacy programs, right?

嘉宾：初始提示必须非常具体，因为它从未接触过这个程序。比如“去……去这里，勾选这个框”之类的。你觉得呢？尤其是流程很长时，是要写一段很长的提示，还是实际演示给它看该怎么做？Guest: On the initial prompt, you have to be very specific because it's never seen this program. Go to go to this check this box in how do you like especially a long process is that a long promp or do you actually show it how to do something?

主持人：是的，对于这个版本，你可以看到这里有一个提示词，以及正在执行任务的智能体。嗯，也许再过几个月，我们也会考虑这件事——这在研究社区里相当常见。基本上，如果你能复现……比如人们在谈论世界模型，对吧？如果你能复现这些应用，而且能更好地控制它们，确保它们不会做出不该做的事；如果你设置了护栏，确保智能体不会做坏事，那么你就可以在环境中进行探索。基本上，你让智能体在环境里自由探索不同的……比如 100 次，然后智能体就会对环境更加熟悉。这样一来，你的提示词就可以没那么……你不需要把问题描述得非常精确，比如要点这个按钮还是那个按钮。你也能成功地……比如找到第一个成功的轨迹。一旦找到了，之后就都能成功。你已经知道这个流程，也就可以准确地说明具体步骤了。Host: Yeah, for for this version you can see there's a prompt and the agent doing the task uh well maybe in a couple months we we also think about it's it's uh it's pretty common in research community. So basically if you can replicate like people talking about word models right if you can replicate this applications uh that you have better control on not doing if you have guard rails that the the agent is not going to do like bad things and then you can do exploration we call exploration in the environment. So basically you let the agents do free exploration on the envir environment for different uh like maybe a 100 times and then the agent become more familiar with the environment and then your basically your prompt will be less you you don't need very accurate problem like whether this button or that button you you can also al also like succeed in terms of like uh finding the first basically the goal is to finding the first successful trajectory once you found it it can be successful all the And you already know that process you can already tell the exact process.

嘉宾：是的。Guest: Yeah.

主持人：对，对，对，对，对。对很多公司来说，他们称之为 SOP。所以他们确实有内部的 SOP。我们实际上做过一个有意思的实验：把一份 SOP 直接交给编码代理，一次性完成所有事情，结果直接就 100% 成功了。所以这显然是可行的。不过我认为，现在这一切仍然有点处于演示阶段。我们想要一种足够通用的技术，让每位客户来到这里时，我们都不必担心哪里会出问题。因此，今后我们还有很多边缘情况需要解决。我记得刚才有位先生举过手。Host: Yeah. Yeah. Yeah. Yeah. Yeah. For for many companies they call SOP. So yeah they have internal they have SOP. So we we actually did a interesting experiment. We feed one SOP directly to the uh coding agent right and one shot in one shot everything and it's 100% successful directly. So it's pos is there definitely possible but I I hold the uh the possibility that like everything now is still kind of demo phase. We we want a technology that's general enough that every customer coming here and we don't have to worry about anything breaks in there. So there's still lots of corner case that we have to solve going forward. I remember there's one gentleman raised hand before.

客人：哦，好的。是的。好吧，好吧。非常感谢。哦，还有一个问题。其实，我想听听你对那个软件的看法，不过你看，我们实际上最终看到的是像素之类的东西。所以这台电脑甚至看不到它在上面运行。我明白你是想让智能体运行起来，但这是……你……是的。嗯，呃，我的印象是，它有点像一张地图，然后你试着让智能体在地图上行走。这样，如果你让它读取屏幕截图，它就能直接过去，弄清楚应该怎么走，比如它不会……Guest: Oh okay. Yeah. All right. All right. Thank you so much. Oh, one more. And actually your opinion on that actually software but you can see like we are actually the pixels certain things at the end. So this computer don't even see it working on it. I understood that you want to agent working but is it something you Yeah. Um uh I I so uh my impression is kind of like a map and then you try to let the agent to walk through the map. So he can just go if you ask us to read screenshot go there figure out just how we should go like what it doesn't

适应应用程序的变化Adapting when applications change

ff

主持人：对，对，对，这就是编码层的整体思路。编码层基本上会把整个轨迹都编码成代码，对吧？我想，我们试图解决的问题是：假设存在一些遗留软件，它们不会再改变，那么这种方法就行得通。但如果我们说的是社交媒体网站或其他不同的网站，它们实际上会发生变化；一旦它们变了，你的路径就行不通了。我们有没有办法在这种情况下让智能体适应变化？Host: yeah yeah yeah so that that's the whole idea of the coding layer so the coding layer basically encode all of the trajectory into the code right and I think I think the problem we try to solve is uh suppose there's legacy software then it's not going to change then this this approach work but if we talk about like websites right for social media or like different websites and they actually change and when they change then your path is not going to work in there is there a way for us to actually adapt adapt the agent uh during that scenario

嘉宾：我的意思是，我对那个问题的回答是：如果你的主代理再去检查一下，那你就能持续改进。Guest: I mean my answer to that problem was if your main agent again like check it out. So you have continuous improving.

主持人：是的，是的，是的。我明白你的意思了。所以，基本上他的解决方案就是，每次发生变化时，我都会派一群智能体进去探索。这样我就总能知道最好的那个，呃Host: Yeah. Yeah. Yeah. I get your point. So, so basically his solution is okay, every time there's a change, I have a swarm of agents doing the exploration in there. So, I always know the the best uh

嘉宾：是的，里面的路径。Guest: Yeah. the path in there.

主持人：你得，呃，真的……Host: You have to like actually

嘉宾：是的，是的，是的。这正是我对那位先生的回应。基本上，这个想法是，如果你有一个遗留系统，而且探索的成本为零，对吧？比如说，如果是银行，那探索是有成本的，我不可能尝试所有操作。但如果你没有任何探索成本，就可以直接进行探索，比如把智能体放进去，让它们尝试不同的操作，并记录所有轨迹。这样一来，问题实际上就更容易解决了。所以我认为，目前 80% 的应用其实都存在于遗留系统中。如果我们能做到这一点，面向通用系统的问题实际上会更容易解决。这样也已经能够在当前行业中创造出很大的经济价值。Guest: Yeah. Yeah. Yeah. That that's exactly my response to that gentleman. So, basically the idea is if you have legacy system and you have is the zero cost for exploration, right? Because for example, it's if it's a bank, then there's a cost for exploration. I cannot explore for all of the actions. But if you don't you don't have any like cost for doing exploration. You just do exploration like put agents in there and like let them try different things and map all of the trajectory there and it's actually a simpler problem to solve. So basically I I think 80% of the the applications is actually living in the legacy system right now. And if we have you can like it's it's actually easier problem to the general uh system. So that can also already capture a lot of economic value in the current industry there.

是的。谢谢你的反馈。是的。此外，进一步来说……Yeah. Thanks for your feedback. Yeah. Further furthermore

训练数据与代码复杂度Training data and code complexity

主持人：你觉得数据够了，就可以表示数据充足。Host: you think enough data you can signal enough data.

嘉宾：是的。所以问题是，假设我们有足够的数据，能不能真正训练智能体，让它们取得非常好的表现？围绕这个问题有过很多讨论。基本上，我的结论是，要做到这一点，你需要的钱比 OpenAI 和 Anthropic 多得多，甚至多出好几个数量级，因为视觉数据是二维的，文本是一维的，所以我们讨论的是大得多的规模，对吧？所以，如果你能给我几万亿美元，我们当然可以考虑一下。是的，不过这很有挑战性；也许借助一种新的范式是有可能的。但考虑到目前的深度学习范式，这件事短期内还做不到，至少我觉得现在还实现不了。Guest: Yeah. So, so that the question is suppose we have enough data can we actually train uh trend the agents to be very successful. So there was a there was a lot of discussion around that. So basically my conclusion is you need more exponentially more money than open eye and anthropobic to do it because uh vision data is two dimensional tax is one dimensional so we are talking about like something like much bigger right so so you uh if you have if you can give me like couple trillion dollars we can we can certainly consider that yeah but it's challenging maybe it's possible with the with a new paradigm of but given the current paradigm of deep learning it's not something that's uh like I don't think it's achievable in the near term right now.

主持人：是的。最后一个问题。Host: Yeah. Question final question.

嘉宾：对。Guest: Right.

主持人：有很多。没错。Host: There's a lot of Right.

主持人：效果怎么样？Host: How does it do?

嘉宾：哦，最终你的代码里会有很多 if/else 逻辑。所以你可以想象，代码肯定会变得难以让人管理。不过，智能体仍然能够编写包含大量 if/else、while 循环和复杂逻辑的代码，而且这些代码依然能正常运行。没错，这就是这个系统最终的目标。当然，你也可以把问题拆分成更简单的小问题，一次解决一个。但这样就会遇到边界问题：比如，怎样定义一个更小的问题？边界在哪里？输入和输出是什么？对吧？所以有时候，如果能找到一个简单的解决方案，那其实更好。好，谢谢大家。Guest: Oh, eventually your code will have a lot of eels logic in there. So, so you can imagine the code is the code is not going to be manageable by humans for sure. Uh but the code is still be able the agent is still be able to write that code with lots of eels while loop uh complex logics in there and it'll still work and yeah that that's the eventual goal of this system in there. Yeah. But you can also chop it chop it to make a simple problems each time. But then you have a boundary problem like okay how do you define a smaller problem like what's the boundary? What's the input and output? Right? So sometimes you if you have a simple solution that's that's actually even better. All right. Thank thank you everyone.