烧掉你的 Flags：给智能体打造交互式 CLI | PayPal，Mark Lummus 与 Navinkumar Patil

简要说明Brief Description

告别命令行标志：面向智能体的交互式 CLI——Mark Lummus 和 Navinkumar Patil，PayPalBurn Your Flags: Interactive CLIs for Agents — Mark Lummus & Navinkumar Patil， PayPal

目录Table of Contents

• 简介与交互式 CLI 挑战
  Introduction and the interactive CLI challenge
• 为什么智能体会悄无声息地错过终端提示符
  Why agents silently miss terminal prompts
• 命令行标志失效的情形
  Where command flags stop working
• 智能体、脚本与无障碍性
  Agents, scripts and accessibility
• Barista 9000 与研讨会竞赛
  Barista 9000 and the workshop competition
• 纳文库马尔·帕蒂尔谈 CLI 的回归
  Navinkumar Patil on the return of CLIs
• 获取 CLI 并登记一笔咖啡订单
  Getting the CLI and registering a coffee order
• 模型选择与 token 预算
  Model choice and the token budget
• 提示一：tmux 会话与终端状态
  Hint one: tmux sessions and terminal state
• 提示二：IPC 侧信道与 JSON
  Hint two: IPC side channels and JSON
• 提示三：上下文注入
  Hint three: context injection
• 评分准确度和响应时间
  Scoring accuracy and response time
• 受众解决方案与基准局限性
  Audience solutions and benchmark limitations
• 保持 CLI 的确定性
  Keeping the CLI deterministic
• 获胜方案及基于智能体构建的求解器
  Winning solution and an agent built solver
• 参与者的第一个 CLI
  A participant's first CLI
• 上下文、侧信道与 tmux
  Context, side channels and tmux

简介与交互式 CLI 挑战Introduction and the interactive CLI challenge

大家好，欢迎参加。我的名字是 Mark Lumus，在 PayPal 负责开发者工具。我身边这位是 Naveen Kumar，是我们开发者工具团队的技术负责人。今天我们想和大家聊聊 CLI。我们演讲的标题稍微有些挑衅意味，听完演讲你们就会明白原因。今天我们会介绍一种特定类型的 CLI：交互式 CLI，还会安排一个练习。毕竟这是一场工作坊，所以对于愿意参与的各位，我们会提供一个 CLI，让你们尝试改进它，使其在智能体工作流中发挥更好的作用。这就是今天工作坊的目的。我们会留出充足的时间，先聊几分钟，尽量缩短讲解时间，然后就让大家开始动手。Hi everyone, welcome in. My name is Mark Lumus. I'm head of developer tools at PayPal. With me I have Naveen Kumar. He's our tech lead for the developer tools team. And today we're going to talk to you a little bit about CLI. The title of our presentation is meant to be a little provocative and you'll understand why at the end of the presentation. So today we're going to talk a little bit about a specific kind of CLI, interactive CLI, and we're going to set up an exercise and we're going to give you for those of you who wish to participate, this is a workshop after all. We'll give you a CLI that you can make work better in an Agentic workflow. So that's the purpose of this workshop today. So we're going to give you plenty of time. We'll talk for a few minutes. Try to keep the talking to a minimum and then we'll get you started.

现在这事还挺有意思的。我记得有人说过，旧的东西又变新了。我刚开始职业生涯时还是很久以前的事了，那时我已经有了白头发。一切都是 CLI，一切都是命令行界面。你们有人记得在 Microsoft Windows 出现之前用 MS-DOS 吗？对，就是这样。我大学毕业后的第一份工作是设计嵌入式系统。所以你会通过串口连接到一台计算机，打开一个兼容 VT100 的终端，然后开始敲命令。这就是你的 CLI，也就是你与计算机交互的方式。如今，CLI 又有点重新流行起来了。我想，可能是 Claude Code 真正让它们变得更受欢迎，现在我们还有 Codex。嗯，所以很多人都在添加 CLI 工具，让访问……变得更方便。Now it's kind of interesting. I think someone said what's old is new. When I started my career long time ago, I have gray hair. Everything was a CLI. Everything was a command line interface. Any of you remember working with MS DOSs before we had Microsoft Windows? There you go. My first job out of college, I was designing embedded systems. And so you would connect to a computer over a serial port and you would pull up a VT100 compatible terminal and you would start typing. And that was your CLI. That was the the the way you interfaced with the computer. So with today CLIs have kind of had a resurgence I guess you know with the maybe it was clawed code that really made them much more popular and now we have codecs um and so a lot of people are are adding CLI uh tools so that it's easier to access the

为什么智能体会悄无声息地错过终端提示符Why agents silently miss terminal prompts

他们想要向智能体开放的能力。现在说到交互式 CLI，这就是我们今天想讨论的话题。嗯，智能体经常会调用 Bash，这很好。但如果它们运行交互式 CLI，情况可能就不太妙了，因为并不是所有执行框架都能很好地处理交互。如果处理得不好，它们就会悄无声息地失败。你不会知道，既没有报错，也没有任何提示。 但我们为什么要在意呢？交互式 CLI 很棒，因为下一个问题可以根据你对上一个问题的回答而变化。如果你试图创建标志来预先涵盖所有可能的流程，标志就会变得越来越多。选项取决于上下文，也可以根据环境中正在发生的事情灵活调整。所以，如果你的 CLI 调用 API 并收到响应，然后再根据 API 返回的内容提出问题，那么用标志来处理就会非常非常棘手，而交互流程本身可以包含各种防护措施。capabilities that they want to expose to an agent. Now, with interactive CLI, this is why we're here today with what we want to talk about. Um, agents shell out to Bash constantly, and that's good. But if they run an interactive CLI, that may be bad because not all harnesses handle interaction very well. And if they don't handle it very well, they'll fail quietly. You won't know. No errors, no nothing. But why do we care? Interactive CLIs are great because the next question can emerge based on what way you answer a previous question. Flags multiply if you try to create flags that predict all the possible flows. The choices are contextual and they can also be responsive to what's happening in an environment. So if your CLI is calling an API and getting a response and then the question comes up based on what the API returns, that's really really challenging to handle with flags and the guard rails can live in the interactive flow.

这里左边有个例子。我们看到一个 CLI 示例，它是交互式的。CLI 打开了一个交互式提示，也就是一个错误按键提示。人能看到它，也可以使用方向键，但代理看不到。代理看到的是一个重绘循环，而不是一个问题。于是 harness 丢失了 UI 状态。这就是故障模式：它悄无声息、无从察觉。So here's an example over here on the on the left. We see a a sample CLI. It's interactive. The CLI's opened an interactive prompt, an error key prompt. The human sees it. The human can use the arrow keys, but the agent doesn't see it. The agent sees a redraw loop, not a question. and the harness loses the UI state. So here's the failure mode. It's silent and invisible.

命令行标志失效的地方Where command flags stop working

所以常见的解决办法就是添加标志。对于简单的使用场景，标志完全没问题。你知道，流程是预先确定好的，只需要执行一遍，而且所有选项彼此独立。这就是使用简单标志的绝佳场景。但如果答案会根据上下文变化，或者会随着对实时事件的响应而变化，那就会变成一个问题。这些情况下，你确实没法使用标志。So the common fix will just add flags. And flags are fine for simple use cases. You have uh you know what the you know what the flows are up front. It's a single pass. All the options are independent. Great great use case for a a a simple flag. But when the answers change based on the context or they're changing based on reactions to real time events, that's when it becomes a problem. You really can't use flags for those situations.

智能体、脚本与无障碍功能Agents, scripts and accessibility

现在，智能体并不像自动化脚本。自动化脚本什么都不知道。所有决策都预先设定好了。智能体可以推理，可以应对，可以提问，可以重试，也可以退回上一步。最重要的是，如果你给它提供一棵决策树，智能体就能沿着它逐步执行。光靠标志位是做不到这一点的。所以，我们其实已经有一个模型可以解释现在发生的事情了。它叫作无障碍访问。因此，如果你把我们为人类所做的事情映射到我们需要为智能体做的事情上，解决方案就会显现出来。这是同一门学问，只是换了新的使用者。那么，挑战就在这里：我们如何既保留安全护栏和指导，又不再假装智能体和人类拥有相同的……Now, an agent is not like an automation script. An automation script knows nothing. All of the decisions are pre-baked up front. The agent can reason. It can react. It can ask. It can retry. It can back up. And most importantly, an agent can walk a decision tree if you provide provide one. You can't do that with flags. So we already have a model today for what's happening here. It's called accessibility. And so if you map what we do for humans to what we need to do for agents, then the solutions become apparent. It's the same discipline with a new reader. So here's the here's the challenge. How might we keep the guard rails and the guidance and stop pretending that agents and humans have the same

Barista 9000 与工作坊竞赛Barista 9000 and the workshop competition

能力？现在，我不会回答这个问题。这正是这场研讨会的目的。那么，挑战来了。我们有一个 CLI，叫作 Bris Barista 9000。它会接收咖啡订单，但你尝试从智能体运行它时，它就会失败，因为它是交互式的。如果你选择参加挑战，获胜者将赢得一台 Mac Mini。今天就能把它带走。就在那边那个袋子里。abilities? Now, I'm not going to answer this question. That's the whole purpose of this workshop. So, here's the challenge. We've got a CLI. We call it Bris Barista 9000. It's going to take coffee orders and it's going to fail when you try to run it from an agent because it's interactive. And if you choose to participate in the challenge, the winner will win a Mac Mini. Walk walk away with it today. It's in that bag right there.

主持人：弄到一台 Mac Mini 本身就是个挑战。我直说吧。好，现在说下简单规则。我们会给你一个 CLI。你对这个 CLI 做出修改，通过一个智能体运行它，它就会向我们的服务器发送一些信息。然后它会更新排行榜。我们稍后会把排行榜放出来。我们没有两个输入源，所以会把幻灯片换成排行榜，让大家能看到。现在有请 Naveen，他会给大家讲解。Host: That was a challenge in and of itself getting a Mac Mini. I'll be straight up. Okay. Now, [clears throat] simple rules. We're going to give you a CLI. You make changes to that CLI, run it through an agent, and it's going to send some information up to our server. And then it's going to update a leaderboard. And we'll put put the leaderboard up in a moment. We don't have two inputs, so we're going to change the slide deck out for the leaderboard so you can see it. Now, here's Naveen. He's going to talk you through this.

纳文库马尔·帕蒂尔谈 CLI 的回归Navinkumar Patil on the return of CLIs

嘉宾：谢谢你，马克。好的。Guest: Thank you, Mark. All right.

主持人：太棒了。是啊。呃，我觉得这对大家来说都是个令人兴奋的任务，而且这个话题对我来说非常重要、很有意义。不过，先把这些放一边，Mark，我们做完这个之后去喝咖啡吗？Host: Cool. Yeah. Uh I think it's an exciting assignment for everybody and this topic is very near and dear to me but uh keeping things aside Mark are we getting any coffee after we build this

嘉宾：如果我们成功了，我希望如此。Guest: I hope so if we're successful

主持人：好，太棒了。那在我们开始讲作业之前，我想先带大家回顾一段历程。对吧？我刚开始工程师生涯时，就开始接触各种 IDE。那时我主要用 Java，所以开始使用 Eclipse。随着时间推移，我转到了 IntelliJ，也就是 JetBrains 的产品。它真的很棒。回头看，大概五六年前，情况开始发生变化，市场上出现了新的 IDE，尤其是微软的 VS Code，它也催生了很多其他 IDE。所以看看现在，微软有自己的 VS Code 编辑器，但后来也开始涌现出新的 AI IDE，比如 Cursor、Windsurf 等等。在这一切发展过程中，CLI 一直都在，对吧？有段时间我想，嗯，你知道，CLI 正在慢慢消亡，也许它们不会再回来了。不过，我其实很喜欢《宿醉》电影里的那句台词：四个朋友都在车里，然后他们说：“嘿，我们回来了。”Host: okay cool so yeah let me uh before we start into um the assignment I want to take to a journey right uh when I started my engineering career I started to play with ids. I started to use Eclipse because I was primarily using Java. And as time went by, moved into Intelligj uh Jet Brains. It was amazing. Looking back, five or six years behind uh things started to evolve. there were new ids in the market especially Microsoft VS code which gave birth to a lot of u other ids as well. So if you look right now Microsoft had its own VS code editor but later on new AI id started to pop out like cursor windsurf to name a few. In all these journeys, CLIs were constant, right? At some point, I thought, okay, you know, CLIs are having a slow death of their own. Maybe they are not going to come back. But, you know, uh I I I actually like to see this hangover movie dialogue that um you know, all all the four friends are in the car and they say, "Hey, we are back."

想象一下 CLI 彼此看到这一点。在 AI 的世界里，它们实际上是 VC，对吧？至于是谁买下了它，嗯，我不知道。但整个情况是，现在 CLI 已经变了。谁该为此负责？是 Agents，对吧？所以这次研讨会主要聚焦于 Agents，也就是你们的 CLI。这将是一项动手实践练习。这里有规则。也没有规则。所以尽管尽可能地折腾，你们知道，奖品就在这里。嗯，所以我们有了……Imagine CLI seeing that to each other. And in the world of AI, they are actually VC, right? Whoever bought it, yeah, I don't know. But the whole story is now the CLI have changed. And who is responsible for that? Agents, right? So this workshop is primarily focusing on agents, your CLIs. And this is going to be a hands-on exercise. There are rules to it. There are no rules to it. So feel free to hack as much as possible and you know the prize is here. Um so yeah we have

获取 CLI 并登记一份咖啡订单Getting the CLI and registering a coffee order

也许，呃，就算我能拿到笔记本电脑也行。我们有几件事要做。嗯，我们有一个名为 PayPal cafe 的代码仓库，里面有一个你可以下载的 CLI。这是一个最基础的 CLI。不过，要在 Mac Mini 上用它可没那么容易。对。你会拿到一些初始命令。用它试试看，注册一下，提供你的用户名，随便你喜欢什么。呃，你的昵称也行，什么都可以，然后试着用它煮杯咖啡。好，这就是目标。在整个练习过程中，我们也希望你能用创新的方式来解决问题。接下来，结合使用 CLI 的智能体会让事情变得很有意思，所以请从这个角度来思考。我改了一下。我刚换了笔记本电脑。现在，在这次课程中，我们不会给你任何提示。maybe uh ven if I can get the laptop. So we have couple of things. So we have yeah um so we have the repo which is PayPal cafe. It has the CLI that you can actually download. That's the barebone CLI. Now, it's not going to be that easy uh for a Mac Mini. Yeah. Uh so, you will get some initial commands with it. Uh play around with it. Register yourself. Uh provide your username, whatever you like. Um your nicknames, anything, and try to brew a coffee out of it. Okay, that's that's the goal. And in this whole exercise, we want you to also be innovative in your approaches and going forward, it's going to get interesting with agents using the CLI. So think it from that perspective. I change this. I just change my laptops here. Now during the during the session, we're we're not starting you out with any hints.

等我们觉得已经有足够多的人安装了这个仓库，并且能直接运行 CLI 之后，也许每隔 15 或 20 分钟，我们就会分享一个好点子。现在，我不会把它称为最佳实践，只会称它为一个好点子，因为我觉得我们现在还处于探索旅程的早期阶段，还不知道最佳实践是什么。我们现在正在摸索。它们都是好点子。我也希望最后能听到观众朋友们分享一些很棒的点子。请记住这一点。我们都还在学习，这是一个探索的过程。我们会在快结束时再聊到这个。它是不是没在运行？再给我们几分钟。我想我们会解决技术问题的。看起来大家确实在用它。所以在右侧，你会看到 Get CLI；如果你看到 PayPal cafe/rules，所有规则都列在那里。Once we feel like enough people have gotten the the repo installed, can run the the CLI as is, then maybe every 15 or 20 minutes, we'll share a good idea. Now, I'm not going to call it a best practice. I'm just going to call it a good idea because I feel like we're early enough in the journey here where we don't know what the best practices are. We're kind of feeling them out now. They're good ideas. I'm also hoping that at the end we'll hear some great ideas from you in the audience so we can share. So just keep that in mind. We're we're all learning. This is a journey. We'll come back to that towards the end. Is it not working? Give us a couple of minutes. I think we'll get the technical issues resolved. seems like people are actually using it. So on the right side you will actually see get CLI and um if you see PayPal cafe/ rules that's where you will have all the rules outlined to it.

在我们弄清楚之前，你可以先开始处理 CLI。好的。Feel free to start working on the CLI till we figure it out. Okay.

主持人：我们有欧洲那份吗？哦，是那个吗？Host: Do we have the Europe? Oh, is it that one?

嘉宾：是啊。嘿，好了。Guest: Yeah. Hey, there you go.

主持人：不错。Host: Nice.

嘉宾：太棒了。Guest: Awesome.

主持人：谢谢。非常感谢。嘿，太棒了。你们可以看到这是……哦，我们确实看到了一些签约。好的。右侧可以看到“获取 CLI”。你们可以下载，希望大家都能看到。它在右下角。这里还有一些规则。点击这些规则，就会弹出具体内容。我们大概会花 25 到 30 分钟。嗯，中间我们还会穿插一些幻灯片，让内容更有趣。Host: Thank you. Thank you so much. Hey Cool. So you can see this is Oh, we do see some signings. Okay. So on the right side you will see get CLI. You can download that and I hope you guys can see it. It's at the bottom right. And there are some rules here. You can click on these rules and it will pop out some rules. So, we'll probably take around about 25 to 30 minutes. Um, and in between we'll interrupt with more slides and make it interesting.

来宾：其他幻灯片用的笔记本电脑在哪儿？Venode拿着呢。好的，没关系。我们会在这里为大家提供帮助。如果你遇到困难，请举手。我们会尽力为你提供帮助。简单介绍一下这些模型。不要使用模型。Guest: Where's the laptop for the other slides? Venode's got it. Okay, that's fine. and we're here to help. If you're having trouble, please raise your hand. We'll try to assist you as best we can. a little bit about the models. Don't use models.

模型选择与 token 预算Model choice and the token budget

现在，我们内部确实有过一番小小的讨论。市面上有一些很棒的模型。这个挑战有意思的地方在于，如果你在最前沿的大语言模型上使用最高思考模式，大概可以靠蛮力硬闯过去。但我觉得在当前的环境下，各团队开始考虑自己的 token 预算了。所以你未必想把高思考模式的 token 浪费在运行 CLI 工具上。真正的挑战是：你能不能让 Haiku、Sonnet 之类处于那个级别的模型运行你的 CLI？我们知道最高级别的模型能做到。那更居中一些、更快也更便宜的模型能做到吗？如果你想接受挑战，这才是有意思的地方。Now, we had a little bit of a debate internally. There are some great models out there. The interesting thing about this challenge, if you use the highest thinking mode on the bleeding edge LLMs, you can probably brute way brute force your way through this. But I feel like in the current climate, teams are starting to think about their token budget. And so you don't necessarily want to waste your highinking tokens on running a CLI tool. So the real challenge is can you make Haik coup or sonnet or something at that level run your CLI? We know you can do it with the highest. Can you do it with something that's somewhere in the middle that's faster and cheaper? That's the That's the interesting challenge if you if you want to take it on.

主持人：听起来很有趣。是啊。Host: Sounds fun. Yeah.

主持人：太棒了。既然我们聊到了咖啡，呃，马克，你知道旧金山市有全美国最好的面包店吗？你知道这个事实吗？Host: Awesome. Since we are on the coffee topic, uh Mark, were you aware that San Francisco City has the best bakeries in the entire United States? Were you aware about that fact?

客人：我不知道这件事。Guest: I was not aware of that.

主持人：是的，你肯定应该这么做。Host: Yeah, you should definitely.

嘉宾：继续。嗯。继续，多跟我说说。Guest: Go on. Yeah. Go on, tell me more.

主持人：嗯，我在城里逛过几家面包店，真的特别有意思。你们一定要去看看这里所有超棒的面包店。Host: Um, I have I've done some bakery crawls uh around in the city and they are so fun. Like you should definitely check out all the amazing bakeries out here.

嘉宾：好的。Guest: Okay.

主持人：是的。有什么最喜欢的吗？黄油和酥粒。那是我最喜欢的。不过我没法去那里。太棒了。目前 J Gray Codex 排在最前面。所以各位，请赶上他或她。使用 init 命令，你们实际上可以注册自己的名字。希望你们当中有些人已经弄明白了。你想谈谈第一个……你可能得想一想……等我把它插上之后，我……Host: Yeah. Any favorites? Butter and crumble. My favorite one. Although I couldn't go there. Awesome. So as of now we have J Gray Codex at the top. So guys please catch up with him or her So with the init command, you could actually register your names. I hope some of you have figured that out. Do you want to talk about the first You might have to think about Once I plug it in, I

嘉宾：[叹气]Guest: [sighs]

提示一：tmux 会话和终端状态Hint one: tmux sessions and terminal state

主持人：请开始。Host: Go ahead.

嘉宾：好的，Guest: Okay,

主持人：很好。嗯，我们接下来进行一些演示。与此同时，你们可以开始动手了。我的意思是，你们可以继续使用手头的 CLI。不过，先给大家一个提示：开始使用 T-Max 是个不错的主意。现场有多少人了解 T-Max？好，看来不少观众都知道。嗯，简单来说，先运行 T-Max，然后输入 help 命令。help 非常重要，因为如今每个 AI 智能体在开始使用某个命令之前，都会先依赖 help 命令。所以，没错，send-keys 和 capture-pane 这两个命令能帮你做很多事。对于还不了解 T-Max 的人来说，它可以让你使用多个终端会话，还提供分屏功能，你可以好好利用 T-Max 的这个功能。Host: cool. Um, let's move on to some presentations. Meanwhile, you can start working. I mean, you'll continue working with the CLI that you have. But as as a first hint, you know, it's it's a good idea to start using T-Max. How many of you know about T-Max here? Okay, quite a fair amount of audience out here. Um, so yeah u in short start with running T-Max and um start by putting help command. Now, help is really important because every AI agent in today's world relies on the help command before it even starts using that command. So, yeah, send keys and capture pain gets you far. Now, T-Max actually um for those who actually don't know about it is like a multiple you could actually use multiple terminal sessions uh with the T-Max and it's like a split pane feature that you could leverage with T-Max.

所以，通过这个示例，你可以看到，嗯，我觉得屏幕对我来说太小了。我稍微放大一点。好，tmux -d。这个标志的作用是将整个 CLI 分离出来，并把它放到另一个 tmux 实例中。你可以看到，tmux 支持捕获窗格；就像我说的，你实际上可以在 tmux 中使用分屏窗格。这样你就可以在 CLI 中处理各种工作，而这也是一项很酷的功能，可以帮助你的智能体完成许多任务。嗯，想象一下你需要运行多个智能体的情况。你可以启动不同的会话，而 tmux 会记录这些会话，了解你在做什么，并在需要时恢复它们。所以，当智能体同时打开多个会话时，这实际上有助于它们相互交流并开始协作。So, with this example, you can see here um I mean the screen is very short for me. So, let me just zoom in a little bit. Yeah, T-Max D. Now, what this flag does is it detaches the entire uh CLI and puts it under a different emox instance all together for you. So you could see size uh T-max capture pane like I said you could actually have split panes in T-maxes. So you could actually take care of u working around with um in in the CLI and uh that's one of those cool features that can help your agents perform lot of activities. Um, think of it when you want to run multiple agents. You could actually spin up different sessions and T-Max actually takes a log of those things where it knows what are you working on and resumes those sessions whenever it's needed. So, it's actually helpful for agents to communicate and start communicating when they have multiple sessions open.

而这些是一些长时间运行的任务，它们……And these are some long running tasks that

提示二：IPC 侧信道与 JSONHint two: IPC side channels and JSON

你实际上可以在 T-Max 内部运行。[嗤笑] 第二个好点子是 IPC 侧信道。就像我之前提到的，T-Max 是一次不错的升级。因此，你可以开始使用侧信道，而不是进行屏幕抓取——T-Max 确实会为了视觉推理进行屏幕抓取。现在，[嗤笑] 侧信道可以帮助你定义已有的方向键，也能帮助你理解它的工作原理；我会在下一张幻灯片中讲解。不过，智能体应该知道输出的 JSON 是什么。在整场演讲中，输出 JSON 变得非常关键，而智能体也需要它，因为现在我们有了多种使用者。以前我们只面向人类，现在则有了智能体。因此，请同时考虑这两类使用者，让你的 CLI 能充分满足这两种使用场景。对于不了解侧信道的人，我来用简单的话解释一下：可以把它想象成给你的 Web 应用添加一个 REST API，而网站则为人类提供可视化界面。you could actually run inside T-Max. [snorts] Second good idea is IPC site channels. So like I mentioned before, T-Max was a good upgrade. So instead of doing screen scraping, which T-Max actually does um for visual in inference, you could start using side channels. Now [snorts] site channels would help you define your arrow keys that you have and help you understand and I'll go in the next slide uh how it works but the agent should know what is the output JSON now this becomes really critical in the entire talk is output JSONs and Asians want that since we have multiple personalities now earlier we used to only cater to human beings now we have agents. So think of both these personas so that your CLI can actually take advantage of both these use cases. So site channel for those who don't know about it let me explain to you in in simple words think of it like adding a REST API to your web application and the website gives humans a visual interfere uh interface.

API 会提供一个机器结构化的接口，对吧？所以可以把 side channel 看作完全相同的东西——这是我可以用来说明的一个类比：读取状态、发送输入，就像处理数据一样。[清嗓子] 现在我们有了 barista brew-agent，这是一个命令，可以帮助你在 CLI 中使用后台代理运行程序，对吧？嗯，你会希望代理在后台运行，然后你输入这些命令，它们会帮助你在当前的 CLI 中尝试使用它。看看你能不能在笔记本电脑上的 Barista 应用中取得一些进展。现在，这里有几个你可以用作示例的东西。你可以看到 barista brew ask，然后还有一大堆信息。哦，这是什么？好，字段，然后是小、中、大，也许是咖啡的杯型大小；没错，默认值是大杯。The API gives you a machine structured interface right so think of side channel as exactly the same that's one of those analogies that I can refer read state send input as a data [clears throat] so we have the bista brew hyphen agent now this this is a command that will help you run your CLI with background agents, right? Um, you want your agents to be running in the background and uh you put all of these commands, they are going to help you try using it in your current CLI. See if you are able to um make some progress in the barista application that is out there for in your laptops. Now, there are a couple of things that you could actually provide as an example out here. You can see barista [clears throat] brew ask and then there's a bunch of information out there. Oh, what is it? Okay, fields then small, large, medium, maybe the sizes of the coffee and yeah, default as large.

这到底是什么意思？那是输入数据，对吧？所以你要做的就是把这些输入数据传给代理，让它真正了解这些信息，而且你得用机器可读的格式来提供。这就是那个东西派上用场的地方。所以别只告诉代理人类能看到什么，好吗？跳出常规思维想一想：代理会怎么想？What does that even mean? That's input data, right? So all you have to do is feed in this input data so that agent actually knows that and you got to do it in machine readable formats. So this is where um that thing comes in play. So don't just tell the agent what the human sees, okay? Think it out of the box. What are the agents going to think?

提示三：上下文注入Hint three: context injection

他们有不同的技能组合，你必须开始发挥这些技能组合的优势。同一个提示，不同的大脑。现在，[清了清嗓子]，你看左边，我刚才提到了不同的角色。一类是人类，另一类是智能体。CLI 是一样的，但从它的结构方式来看，人类想要一组选项；另一方面，对于智能体，你可以把所有这些信息都放在一个输入文件中。[哼了一声] 好的，这就是我们所说的上下文注入发挥作用的地方。你要以这样的方式提供上下文，举个例子。你其实快没燕麦奶了。好吧，你只剩下两加仑燕麦奶了。告诉你的智能体需要询问用户什么信息，并确认两盎司够不够。They have different skill sets and you have to start leveraging those skill sets. Same prompt with a different brain. Now, [clears throat] so on the left side you see I was talking about different personas. One is the human beings and the other one is the agents. The CLI is the same but when you look at the way it is structured human beings want set of options but on the other hand you have agents wherein you could provide all this information in one single input file. [snorts] Okay. So this is where we call this as context injection comes in play. You provide context in such a way that let me take an example. You you are actually low on oat milk. Okay. And uh you have only two gallons of uh oat milk left. Tell your agent what kind of information it needs to ask the user. confirm it if if two ounces are good enough.

要我帮你选一个备用方案吗？这项信息应该通过上下文注入这种技术来提供，你也可以在你正在构建的 CLI 中利用这一点。试试看吧。说不定你会登上排行榜榜首。我们从 T-Max 开始了一段旅程，它基本上就是屏幕抓取，之后逐步发展到注入技能。T-Max 是一种低成本的做法。我想你们很多人都试过。不过，侧信道也能帮助你向智能体提供不错的反馈，而上下文注入则能为你提供不同的场景供你思考——这些场景在人类的情形下并不会出现。所以，继续努力提高分数吧。你想展示一下这个吗？Do you want me to pick a backup for you? This information should be provided using context in injection as a technique and you can leverage this in the CLI that you are actually building. Try using it. Maybe you'll be on the highest on the leaderboard. So we started a journey from T-Max which is basically screen scraping to injecting skills. So T-Max is low effort. I think a lot of you might have tried it. But this is where site channels [snorts] also will help you in providing [clears throat] good um sort of feedback to your agents and context injection will provide you those different scenarios that you can think about which doesn't happen in the human case. So yeah, keep pushing your scores. You want to show this?

嘉宾：是的。这确实需要一些时间。Guest: Yeah. It does take some time.

主持人：给你。Host: There you go.

主持人：好，[嗤笑] 不错。我们这边确实看到很多流量。[清嗓子] 哦，有人快到 5,000 了。是 bank 吗？好。[清嗓子] 到目前为止有什么问题吗？有人想要解答吗？有什么疑问吗？Host: Okay, [snorts] cool. We do see a lot of traffic out here. [clears throat] Oh, somebody is really close to 5,000. Is it bank? Okay. [clears throat] Any questions so far? Anybody wants to get it answered? Any queries?

客人：抱歉。Guest: Sorry.

评分准确性和响应时间Scoring accuracy and response time

主持人：好，我再问你一次这个问题。[清嗓子] 如果有多个人都达到 5,000 分，会发生什么？马克，你来回答吗？Host: Okay, let me repeat that question to you. [clears throat] What happens if multiple people reach a score of 5,000? Mark, you want to take that?

嘉宾：所以，我们的想法是先看分数，如果出现平局，就看哪一个用时最少。现在，根据这个结果，我看到有好几个都在零秒内完成了三项中的三项。我看到有一大堆都在零秒内完成了。所以，我得找人商量一下，因为这不应该是可能发生的。但我们的想法是先看高分，也就是分数，然后再看响应所需的时间。这样，快速返回的准确响应就会比缓慢返回的准确响应更占优势。所以目前我们只有一个。Guest: So, the the idea was we would look at score and if there's a tie, we would look at which one took the least amount of time. Now, based on the resolution of this, I see several that are that have completed three of three in zero seconds. I see a bunch of these that have completed it in zero seconds. So, I need to consult with someone because that's not supposed to be possible. But the idea is that we would look at the high score the the score first and then the time it takes to respond second. So that an accurate response returned quickly would be biased over an accurate response returned slowly. So we have only one for now.

主持人：好的，抱歉。Host: Okay, sorry.

嘉宾：我们只有一个。最终会变成这样，这就是排行榜的底部。Nicholas [哼了一声]，游戏开始。[清了清嗓子] 0秒。你想再等两分钟吗？我们可以开放讨论，听听大家对他们参与度有多高有什么想法。我们有……？Guest: We have only one. eventually comes to this is the bottom of the leaderboard. Nicholas [snorts] game up. [clears throat] 0 second. Do you want to wait two more minutes? We can open it up ideas how engaged they are. Did we have

主持人：[清了清嗓子]Host: [clears throat]

嘉宾：再过几分钟，我们就会开放时间让大家分享想法。我希望能听到一些很棒的创意，甚至超越那些不错的点子，因为我知道肯定还有不少这样的创意。Mark，你预料到会有这么多人，人数达到4,000吗？Guest: In just a few minutes, we'll open up the floor for some idea sharing. So, I'm I'm hoping that we'll hear some great ideas that go above and beyond our good ideas because I know that there's some out there. Mark, were you expecting so many people to be in the 4,000?

主持人：是的，当然。Host: Yes, absolutely.

主持人：在这次会议上？Host: At this conference?

嘉宾：当然。Guest: Absolutely.

主持人：太棒了。你不是吗？Host: Awesome. Were you not?

嘉宾：不，我当时是。Guest: No, I was.

受众解决方案与基准测试的局限性Audience solutions and benchmark limitations

你们觉得怎么样？有人能达到 5,000 吗？What do you think? Anybody can make it to 5,000?

主持人：我不知道。我们先不做决定。Host: I don't know. We'll hold it open.

主持人：我们现在开始开放提问吧？Host: Why don't we open the floor up now?

嘉宾：嗯，当然。我们很乐意听听大家的想法。那么，现场有人想到其他办法，能让他们这么快拿到这么高的分数吗？大家都不愿意离开桌子。哦，有人举手了。来，我走过去你那边。[清嗓子]因为我几乎听不清你说话。Guest: Yeah, sure. We'd love to hear ideas. So, did anybody in the room come up with other ideas that were able to get them to such a high score so quickly? Nobody wants to leave the desk. Oh, we have Yes. Let me let me come over to you [clears throat] because I can barely hear you.

主持人：好。这是一个非常容易钻空子的基准测试，因为唯一重要的就是你能在很短的时间内完成一笔订单。所以，如果你用一个确定性解析器——嗯，这完全违背了这个挑战的初衷——但如果你用了确定性解析器，而且你的东西离服务器非常近，只要直接暴力尝试，就能很快把延迟降下来。Host: Okay. it it's a very easy to game benchmark because the only thing that matters is you fulfill one order within like small like a very small latency. So if you do a deterministic parser um it which defeats the spirit of this challenge entirely but you deterministic parser and you have something like really close to the server you can get the latency down uh really quick if you just brute force it.

主持人：好，那就是你做的吗？Host: Okay, which is that what you did?

嘉宾：我现在正在做的就是这件事。Guest: That's what I'm doing right now.

保持 CLI 的确定性Keeping the CLI deterministic

主持人：好。行。你刚才提到了一个有意思的东西——在你的 CLI 中运行的确定性解析器。好。我确实喜欢 CLI 具备确定性的这个想法。[清了清嗓子] 原因是，如果 CLI 是供 LLM 使用的工具，那你就一定希望它具有确定性。这里不确定的是 LLM。我们希望 CLI 像有人形容的那样，是一件小巧锋利的工具。它的功能应该非常非常专注。每次你给它一条命令，都希望它返回相同的响应。所以，你不会想在 CLI 里内置一个智能体，再把这个 CLI 交给另一个智能体，因为你不知道最后会得到什么。还有什么有意思的想法吗？显然，很多人已经摸索出了如何钻系统的空子。Host: Okay. All right. Now you said something interesting the the deterministic parser that's running in your CLI. Okay. I actually like the idea of the determinism in the CLI. And [clears throat] the reason is that if the CLI is a tool to be used by an LLM, then you absolutely want it to be deterministic. The LLM is the non-deterministic player here. We want the CLI to be as someone described it a small sharp instrument. Should be very very focused on what it does. And every time you give it a command, you want it to come back with the same response. So you don't want to build an agent into the CLI and then hand that CLI off to another agent because you don't know what you're going to get. Any other interesting ideas? Obviously we a lot of people have figured out how to game the system.

嘉宾：有 Mac Mini 的地方，就有办法吧，我想。Guest: Where there's a Mac Mini, there's a way, I guess.

主持人：是的。再等五分钟。五分钟。网站崩溃前，比分已经有一阵子没变了。Host: Yeah. Wait five minutes. Five minutes. The score hasn't changed in a while before the site crashes.

主持人：你刷新了吗？Host: Did you do refresh?

嘉宾：[咳嗽] [清了清嗓子]Guest: [cough] [clears throat]

主持人：为什么显示正在重新连接？哦，好了。你看，网络有点卡。我们这边的网络有点卡。Host: Why does it say reconnecting? Oh, there we go. Okay. See, the network is a little laggy. The network is a little laggy out here.

主持人：好多好多电脑。听起来不错。崩溃了。我们现在卡住了。[清嗓子] 好了。结果没有变化。好。我们要不要再等几分钟，还是就到此为止？Host: A lot of lot of computers. Sounds good. Crash. And we're frozen right now. [clears throat] Here we go. No change in the results. Okay. Do we want to wait a few more minutes or call it?

嘉宾：是的，五分钟。Guest: Yeah, five minutes.

主持人：五分钟。好。你想在结案陈词时把 PPD 带到 PPD。Host: Five minutes. Okay. You want to bring the PPD to PPD for the closing arguments.

嘉宾：我们再给它大约五分钟。Guest: We're going to give it about five more minutes.

主持人：嗯。Host: Yeah.

嘉宾：然后我们就收工。Guest: And then we'll call it.

主持人：邦在哪儿？邦克在哪儿？举手。Host: Where's Bon? Where's Bonk? Raise your hand.

嘉宾：Bon是谁？好，我们有点东西了。Guest: Who is Bon? Okay, we have something.

主持人：好。笑得灿烂点。笑得灿烂点。现在，Host: All right. Big smile. Big smile. Now,

嘉宾：我们确实想听听你是怎么做到的。所以等我们叫到你时，我们就会让你解释你做了什么。我很想听听。我得去那边一下。Guest: we do want to hear how you did it. So once once we call it, then we'll we'll we'll we'll let you explain what you did. I'd love to hear. I need to go over here.

主持人：是的，嗯，</finalHost: Yeah,

嘉宾：等一下。抱歉。我应该再要一个……我应该再要一个供现场提问用的麦克风。Guest: hold on. Sorry. I should have asked for another I should have asked for another microphone for the room for questions.

获胜方案和一个由智能体构建的求解器Winning solution and an agent built solver

主持人：嗯，是的。所以，Bonk 就是一个构建在 cloud agent SDK 之上的 agent 东西。嗯，它把整套东西都构建出来了。在此之前，我已经有一个类似 eval bench 的 skill，可以创建某种环境并进行优化。所以它最后基本上构建出了一个确定性的 CLI 求解器，而现在它只是在 T-m 上运行，不停地猛打你的 API。Host: Um, yeah. So, Bonk is just a an agent uh thing built on top of cloud agent SDK. Um, it built the whole thing. Prior to this, I already had a sort of eval bench skill that will create sort of an environment and um optimize. So, it basically ended up building a uh deterministic CLI solver and then now it's just running on a T-m slamming your APIs.

访客：好的。Guest: Okay.

主持人：是的。Host: Yeah.

嘉宾：好吧。也是。我想我运气不错，因为有一次测到了大约 29 毫秒。呃，可能是因为我们在同一个区域。Guest: All right. aso. I think I got lucky because I hit it at one point at like 29 milliseconds. Uh probably because we're in the same region.

主持人：好的。Host: Okay.

客人：这只是网络延迟，Guest: It's just a network latency,

主持人：对吧？而且这一切都是在 Claude 上完成的。Host: right? And this is all done on claude

嘉宾：呃，Claude Agent SDK。好。就是通过 Slack 来驱动它。Guest: uh claude agent SDK. Okay. Just driving it through Slack.

主持人：不，哪个模型？Host: No, which model?

嘉宾：Opus 4.8。Guest: Opus 4.8.

主持人：好的。Host: Okay.

嘉宾：嗯。Guest: Yeah.

主持人：你试过其他模型吗？Host: Did you try it with any other models

嘉宾：关于构建，呃，构建 CLI，还是Guest: for the build uh for building the CLI or

主持人：为、为了你的……为了你的代理吗？我们该怎么做？CLI 完全是确定性的。没错。没错。所以，所以这个代理构建了 CLI，现在只是在 T-Max 上运行它。Host: for for your a for your agent? How would we do that? The CLI is completely deterministic. Sure. Sure. So, so the agent built the CLI and is just running it on T-Max now.

嘉宾：好的。Guest: Okay.

主持人：是的。Host: Yeah.

嘉宾：太棒了。Guest: Awesome.

主持人：是的。Host: Yeah.

嘉宾：恭喜。Guest: Congrats.

主持人：我们来看下一个。Danny。这里谁是 Danny？你看到上一个了。[清嗓子] 我们想知道什么？Host: Going to go to the next one. Danny. Who's Danny here? You see last one. [clears throat] What do we want to know?

主持人：嗯。你是怎么搭建它的？Host: Yeah. How'd you build it?

参与者的第一个 CLIA participant's first CLI

嘉宾：是啊。所以，呃，我完全不懂编程。我基本上算是个“氛围编程”用户。我有几个智能体，然后我就跟它们说：“嘿，这是问题。读一下，然后修好它，修好它。”所以，我就是……对，在 VSC 上开着 Cloud Code。Guest: Yeah. So, I don't know anything about uh coding at all. I'm I'm using uh I'm basically a Vibe coder. I have a couple agents and I just said, "Hey, here's the problem. Read it and then fix it and fix it." So, I just have Yeah. Cloud code up on uh VSC.

主持人：你以前构建过命令行界面（CLI）吗？Host: Have you ever built a CLI before?

嘉宾：呃，不是。我几个小时前才看到，显然有个奖可以拿。所以我就进去了，然后说：“嘿，呃，把这事弄明白。给我解释一下，就当我是……你懂的，就当我什么都不懂。”然后再用给五岁小孩讲解的方式解释。对，我我我也不知道。然后呢，我们就想出了，呃，对，我们拿出了一些不错的结果，我说：“嘿，我们得让它更快点，再给我一些建议，对，认真点。”所以我我猜 [哼笑] 我们有两个脚本在运行，第一个得了 49.98 分，然后我说：“嘿，复制一个，再给我一些建议，告诉我怎么才能拿到 49.99 分。”Guest: Uh no. I I saw a couple hours ago that obviously there was a prize to be won. So, I got in and I said, "Hey, uh figure this out. Explain it to me like I'm, you know, like I don't know anything." And then explain like I'm five. Yeah, I I I don't know. So, and then we came up with uh yeah, we came up with some good results and I said, "Hey, we need to make it faster and give me some suggestions and yeah, seriously." So, I I guess [snorts] we have uh two scripts running and the first one got the uh 49.98 and then I said, "Hey, make a copy and give me some suggestions for how we can get the 49.99."

主持人：对。Host: Right.

嘉宾：你有什么建议吗？呃，这个智能体的自动化程度挺高的。所以我就说，嘿，这是开头，这是结尾；如果需要的话，开始时问我几个问题，然后把问题解决掉，这样我就不用操心了，大概就是这个意思。抱歉，这说得不太好……Guest: Did you get any suggestions? Uh it's this agent's pretty automated. So I just say like hey here's the start here's the end give me a couple you know questions at the beginning if you need it but then fix the problems so I don't have to deal with it kind of thing. So sorry that's not a very good

主持人：不，这挺好的。我确定很多人都在想，写 CLI 这件事你会亲自做吗，还是会让你的智能体替你做？显然，现场有很多人，只要把 Claude 或 Codeex 指向这个任务，然后说：“构建一个 CLI，让它更快、更具确定性。”这就是三个想法，对吧？好。你想换幻灯片吗？就是这张。Host: No it's good. I'm I'm sure a lot of people are wondering like writing a CLI is that something that you would ever do or would you ask your agent to do it for you? So clearly we have many people in the room and just going to point Claude or Codeex at this and say build a CLI, make it fast, make it deterministic. And those are the three ideas, right? Okay. You want to change the slide? There it is.

嘉宾：[清了清嗓子]Guest: [clears throat]

上下文、旁路通道与 tmuxContext, side channels and tmux

主持人：好了，我们已经讨论了所有建议，也听到了一些想法。最主要的想法是让你的智能体替你想办法。我很喜欢这个想法。嗯，在结束之前还有什么要补充的吗？好，那就这样。根据我们的说明和建议，给它提供良好的上下文，给它一个旁路通道，并告诉它使用 T-Mox。这是三个不错的起点。谢谢大家。Bonk，你要上来吗？Host: So, we talked about all of our suggestions. We heard some ideas. The predominant idea was let your agent figure it out for you. I love that idea. Um, anything else before we close it out? All right, there you go. Takeaway for us based on our instructions, our recommendations. Give it good context, give it a side channel, and tell it to use T-Mox. Those are three good ideas to start with. Thanks everyone. Bonk, you want to come on up?

嘉宾：赢家还是 Monk，对吧？没错。好吧。不，不，我们已经宣布结果了。五分钟前就宣布了。Guest: We still have Monk as the winner, right? Yeah. Okay. No, no, we called it. We called it five minutes ago.

主持人：好的。Host: Okay.

嘉宾：对。对。我不会改[清了清嗓子]我不会改它。Guest: Yeah. Yeah. I'm not gonna change [clears throat] I'm not going to change it.

主持人：是的。Host: Yeah.

嘉宾：我们，我们叫它。Guest: We We called it.

主持人：太棒了。Host: Awesome.

嘉宾：我们……[清了清嗓子] 我们会拍——我们就拍张照片，Guest: Let us [clears throat] We'll get We'll just get a picture,

主持人：你知道，社交媒体。没错。这是一台真正的 Mac Mini。Host: you know, social media. That's right. It's a real Mac Mini.

客人：这是真正的 Mac。天哪，这种东西居然真的存在。Guest: It's a real Mac. Oh my god, these exist.

主持人：好，太棒了。Host: All right, awesome.

主持人：它是从哪儿来的？Host: Where'd it come?