为智能体构建数据上下文层：Typedef 演示知识图谱、连接安全与代码影响分析

简要描述Brief Description

The Data Context Layer for Agents — Yoni Michael & Brandon Callender， Typedef

目录Table of Contents

• 数据代理所缺失的上下文
  The missing context for data agents
• 影响分析、连接与安全变更
  Impact analysis, joins, and safe changes
• 分布式上下文与表粒度
  Distributed context and table grain
• 有效的 SQL 仍可能解释错了原因
  Valid SQL can still explain the wrong cause
• 代理需要了解的四件事
  Four facts an agent needs
• 连接语义与扇出
  Join semantics and fan out
• 确定性提取与数据剖析
  Deterministic extraction and data profiling
• 持久化一个可查询的上下文图
  Persist a queryable context graph
• 行动前分析下游变更
  Analyze downstream changes before acting
• 安全连接与指标定义
  Safe joins and metric definitions
• 评估数据工程任务
  Evaluate data engineering tasks
• 五种上下文工程模式
  Five context engineering patterns
• Brandon 的编码智能体工作坊
  Brandon's coding agent workshop
• 两个同名的类型
  Two types with the same name
• 提取、持久化并查询代码上下文
  Extract, persist, and query code context
• 语法、符号解析与含义
  Syntax, symbol resolution, and meaning
• 笔记本一：构建代码图谱
  Notebook one: build the code graph
• 解析规范符号标识符
  Resolve canonical symbol identities
• 将节点和边加载到 Neo4j 中
  Load nodes and edges into Neo4j
• 构建一个爆炸半径分析工具
  Build a blast radius tool
• 将代理接入图中
  Wire an agent to the graph
• 编写其他图工具
  Write additional graph tools
• 使用 Fenic 进行语义摘要
  Semantic summaries with Fenic
• 维护和语言限制
  Maintenance and language limitations
• 数据工程评估与范围
  Data engineering evaluations and scope
• 代理仍然选择 grep
  The agent still chooses grep
• 围绕智能体的习惯设计工具
  Design tools around the agent's habits
• 笔记本二：公平地比较智能体
  Notebook two: compare agents fairly
• 确定性评分与证据
  Deterministic scoring and evidence
• 一次正确运行与重复运行的一致性
  One correct run versus repeated consistency
• 衡量图表工具的采用情况
  Measure graph tool adoption
• 调用图、目录和数据基准平台主管
  Call graphs, catalogs, and data benchmarks
• 查看追踪记录，并质疑排行榜
  Read traces and challenge the scoreboard
• 拓展到 Rust 之外，并提出一些小问题
  Extend beyond Rust and ask small questions
• 边界、索引与稳定的评估快照
  Boundaries, indexing, and stable eval snapshots
• 本体论与更广泛的数据上下文愿景
  Ontologies and the broader data context vision

数据代理缺失的上下文背景The missing context for data agents

我们开始吧。呃，我叫 Yonyi Michael，是一家名叫 Typede 的初创公司的联合创始人之一。呃，今天我们会和我们的创始工程师 Brandon Calendarer 一起举办这场工作坊。嗯，我们有两个小时的时间。我们会尽量让内容有趣、好玩。呃，我知道现在确实很容易分心。你们的笔记本电脑都打开了，而且世界杯比赛里德国队目前以 0 比 1 落后。不过我们保证会尽量把这场工作坊办得有趣，让你们觉得没必要去看直播。嗯，那么，今天我们要聊的是——在真正开始有趣的动手环节、撸起袖子参加更偏技术的工作坊之前，我想先带大家大致了解一下我们在 Typedev 想要解决的问题。We'll get going. Uh my name is Yonyi Michael. I'm one of the co-founders of a startup called Typede. Uh we're here to do along with uh our founding engineer Brandon Calendarer this workshop. Um we have a two-hour time slot. We're going to try and keep it as interesting and fun as possible. Uh, I know it's very tempting right now. You guys all have your laptops out and Germany's down one- nothing in the World Cup game, but it's our promise to try and make it as fun as possible, so you don't feel the need to be streaming that. Um, so, um, what we're going to talk about today, uh, is we're going to do like before we actually get into the fun, roll up your sleeves, do more of a technical workshop, I wanted to walk through a little bit around the problem set that we're trying to solve here at Typedev.

嗯，先为后续内容铺垫一下，并介绍一些背景，讲讲一些概念和原则，之后 Brandon 带领我们进行实际工作坊时会用到这些内容。嗯，这场演讲的核心是介绍 Typed Defeaf 的背景。我们正在构建数据代理访问数据平台和数据系统、并在其中执行高复杂度任务所需的信任层。嗯，所以演讲的标题是“数据上下文层”。这是我们实际构建的一项核心基础设施，旨在帮助我们实现这一使命。嗯，这其中很多内容都源于这样一个想法：也许你们对数据领域没那么熟悉，但一旦尝试让编程代理或软件工程代理处理数据，就会发现这其实是一个细节更多、复杂得多的环境。在构建合适的支撑框架时，有很多事情需要考虑；还要为这些代理设计上下文，才能让它们真正发挥作用。um and kind of set the stage and give some context for concepts and principles uh that will later be applied to the actual workshop that Brandon will lead us on. Um so the main idea behind this talk um is the background on typed defeaf. We're building uh the enter the trust layer for data agents to be able to access and perform uh high complexity tasks on data platforms and data systems. Um and so the title of the talk is the data context layer which is a core piece of infrastructure um that we've actually built to help us with that mission. Um, and so, uh, a lot of it, uh, comes from the idea that perhaps maybe you guys aren't as much into the data space, but once you try and unleash, uh, coding agents or software engineering agents on the data, it's actually a lot more of a nuance, complex environment, and there's a lot of things that you need to think about when building the right harness, when context engineering for these agents to be really effective.

这就是我们要讨论的内容。核心结论是，模型并不是瓶颈。模型的能力还会不断增强。但要让这些智能体真正发挥作用，缺失的上下文才是关键。这就是我们所说的 data context layer（数据上下文层）。 好，下面说说我们这家名为 typed defeaf 的初创公司正在押注的方向。过去大概两年里，我们所有人基本都在这个行业里思考如何为编程智能体构建上下文工程，对吧？比如 Anthropic、Codex、Cursor 这些产品，让我们能够理解如何有效地映射代码仓库、进行检索，并设计工具，从而让这些编程智能体真正发挥作用。That's what we'll be talking about. The punch line is like the the model isn't the bottleneck. The model's going to continue to grow in capabilities. Um but the missing context to make these agents actually effective is the important part in this. Um so this is what we call the data context layer. Um all right so here's the bet that we're taking as a startup called typed defeaf. Um all of us have spent let's say like uh the last two years generally in the industry um going and thinking about context engineering for coding agents right so we have like the the anthropics the codeexes of the world the cursors um where you're able to understand how to map repositories effectively do retrieval and then tool design for these coding agents to be really effective.

嗯，我们做得还不够的地方，以及开始发现需求更加迫切的地方，是也需要为那些能够访问和操作数据的智能体做同样的事情。数据恰恰是问题所在，因为如果智能体信心十足地给出错误答案——而这种情况经常发生——最终可能会造成巨大的经济损失，对吧？因为我们要求智能体分析实际执行的许多指标和计算，最终都会出现在董事会演示材料之类的东西中；或者成为模型中的指标，或出现在那些悄无声息地发生故障的数据管道里，而其中包含着他们试图计算的非常非常重要的特征、指标和度量。嗯，我们的核心判断是，智能体将成为这些数据系统的核心操作者。要让它们真正有效，我们就必须确保它们在执行读写操作时，能够安全、准确地完成这些操作，并且让人们能够大规模地信任这些结果。um where we're falling short and where we're starting to see there's more of a need for is that the same thing needs to be done for agents that are able to access and touch the data and the data is exactly where uh if an agent is confidently wrong which it often time is it can actually end up costing a lot of money right because a lot of these metrics and calculations that we're asking agentic analytics to actually do wind up on things like board decks um or their metrics in a model or their pipelines that fail quietly that have really really important important features and and um and metrics and measures that they're trying to calculate. Um so the core bet that we're saying is that agents are going to be core operators of these data systems. In order for them to be really effective, we need to be able to ensure that when they do read and write actions, they're able to do these actions safely and accurately and be able to trust these results at scale.

而这只有在你能为智能体提供推理层时才真正可行，这样它才能真正端到端地理解整个数据系统，就像那些具备实际知识、能够从头到尾构建该系统的数据工程团队或数据团队一样。And that only really works if you're able to give the agent the reasoning layer so that it can actually understand the data system end to end very much like how a data engineering or a data team that actually has working knowledge that was able to build the system from end to end is able

影响分析、联接与安全变更Impact analysis, joins, and safe changes

也能对它进行推理。这就是这里构建数据上下文层的目标。嗯，过去一年半左右我们在这一领域不断尝试，发现如今的编码智能体在处理数据任务时，往往会在三个具体方面表现不佳。第一是影响分析，也就是：如果我真的做出这项更改，会有什么东西被破坏？接下来是连接建议：在我的数据系统中，这种连接对我来说真的安全吗？此外，还要确保代码更改总体上是安全的，对吧？也就是确保转换逻辑的更改安全。那么，这次编辑的含义仍然相同吗？它是否保留了这个指标的含义？还是说，由于我提交了这次数据更改，这个指标的定义也会随之改变？to also reason about it. So that's the goal of the data context layer here. Um, so where I'm going to say what we found over let's say the last year and a half of us building in this area is that there are three general concrete areas where today's let's say coding agents fail over when you're trying to get them to be effective in data. The first thing is impact analysis. So essentially like what breaks if I actually make this change. Um the next thing is join recommendations. So if the is this actually is this join actually safe for me to make across my data system and then safe code changes generally as well right safe transformation logic changes. So does this edit still mean the same thing? Is it preserving the meaning of this metric or is this metric definition now going to change because I'm making this data commit and this data change that's happening.

这些并不是什么边缘任务，对吧？它们是数据团队日常要处理的任务和工作流程，而他们也在思考如何真正让企业数据与 AI 协同工作。如果你的智能体无法可靠地推理并完成这些任务，就很难真正信任它。很多时候，甚至很难放心地授予它写入权限，乃至读取权限。而真正重要的是，要让它们成为我们数据系统中高效的核心操作者，而不只是那种只会问答的智能体。These aren't fringe tasks, right? These are like the day-to-day uh tasks and workflows of like data teams um as they're thinking of how to actually get enterprise data to be working with AI. And if your agent can't reason and do these tasks reliably, it's very hard to actually trust it. It's very hard to actually trust it to give it right and even read access many times. U and that's what actually matters is be able to get them to be effective core operators of our data systems and [clears throat] not just like question and answers types of agents.

分布式上下文与表粒度Distributed context and table grain

那么，这就是我们在这里通过工作得以界定并真正发现的核心差距。嗯，数据所需的这些上下文分散在许多不同的系统中，对吧？你的数据仓库可以存放数据；它保存着数据行和实际数据本身。你的转换代码包含真正重要的业务逻辑。你的 BI 工具中存储着指标、维度和一些业务定义。除此之外，还有文档、Slack 频道、Slack 讨论串，以及往往存在于人们脑海中的隐性知识，对吧？但智能体实际上还需要这些内容之间的一层东西。它需要真正理解一个模型意味着什么、它与其他模型有什么关联、如果我进行某项更改会导致什么问题，以及哪些操作实际上是安全的，对吧？So here's the gap the core gap that we've been able to define and like really discover through our work here. Um this context that the data needs is uh is uh distributed across many different systems, right? So your warehouse can have the store. It stores the the rows and the actual data itself. Your transformation code has the actual business logic that's really important. Your BI tool stores the metrics, the dimensions, some business definitions in there. And then you have your docs and Slack channels and um and Slack threads and like tribal knowledge that oftentimes lives in like people's heads as well, right? But the agent actually needs something in between all of that. It needs to actually understand what a model means, how it relates to other models, what breaks if I make this certain change, um and then which operations are actually safe to make, right?

嗯，接下来大概15分钟，在我讲解这些内容的时候，我们会用到一个叫作“粒度”的术语。所以，对于在场不太熟悉数据的各位来说，从数据团队的角度看，“粒度”指的是什么呢？嗯，就是，呃，抱歉，就是表里的每一行究竟代表什么？每一行实际表示的是什么？嗯，所以，如果粒度是按客户、按月，而我不小心把它变成按客户、按周，那么之后所有的汇总都可能出错，尽管 SQL 能成功编译，而且……Um and for the next let's say 15 minutes while I'm going over this like we'll be using like um a term that's called grain. So for all of you like non-data folks in the room what grain actually means from like a data team perspective is um is uh um sorry is like what does each row of the table actually mean? What does each row actually represent? Um and so if the grain is customer by month uh and I accidentally turn it into customer by week then every downstream sum after that uh can become wrong even though the SQL compiles and

有效的 SQL 仍可能解释错了原因。Valid SQL can still explain the wrong cause

一切都运行正常，对吧？所以这就是我说粒度变化时的意思。我们把它说得更具体一点，比如从可靠性的角度来看，如果有人问智能体，为什么某条数据管道的行数突然激增，一夜之间涨了 40%，这到底意味着什么？嗯，智能体可以写出有效的 SQL，也能运行它，还能生成一张漂亮的图表，但最后它还是会报告错误的根本原因，错误地分析实际出了什么问题。它会说：“看起来是有更多数据进来了。”但实际上，发生的是上游的某一列把一对一连接变成了一对多连接，对吧？这意味着粒度实际上发生了变化。嗯，智能体根本无法判断这是实际的数据激增，还是粒度真的发生了变化，因为粒度本身并不包含在数据里，对吧？everything runs correctly right so that's what I mean when I talk about grain change um and so let's make this a little bit more concrete like in terms of reliability what does it actually mean if someone asks an agent why a pipeline's row count row count suddenly spiked and it jumped 40% overnight. Um, the agent can go and write valid SQL. It runs it. It could render a beautiful chart, but at the end it's going to report the wrong root cause, the wrong analysis of what actually went wrong. It says, "Looks like there's more data that came in. Um, but what actually happened is there was an upstream column that uh turned a onetoone join into a one to many join, right? So that means the grain actually changed. uh and the agent literally cannot tell a real spike like a real data spike and differentiate between that and like an actual grain change because the grain isn't in the data itself, right?

嗯，所以智能体可以运行查询、获取一些样例行、抽样查看数据，也可以仔细阅读 SQL 来理解它，但它并不知道一行数据原本应该代表什么，也不知道现在它的含义已经完全不同了，对吧？所以，SQL 有效还不够，对吧？对吧？当你想让智能体发挥作用时，语义其实非常非常重要。 那么，针对这种情况，我们不妨想想，有两种显而易见的解决办法。但为什么这两种显而易见的办法实际上都行不通呢？你可以把代码提供给智能体，对吧？但问题在于，比如说你把自己的 DBD 项目或 Airflow 项目交给它，它每次收到提示时都会从头重新注入上下文。这意味着它的行为具有概率性，而且受上下文窗口的限制，效果并不总是很好。另外，每次开启新会话，你都得从头重新构建上下文。Um and so the agent can run the query, it could get some sample rows, it could sample the data, it could read the SQL very well to understand, but it didn't know that what one row was actually supposed to mean and that now it means something completely different. Right? So this is why valid SQL isn't enough, right? Right? The semantics are actually very very important when you're trying to get an agent to be effective. So then let's say like the like thinking on it, there maybe two obvious fixes for what you can do for this type of scenario. And this is why neither obvious of like neither of these obvious fixes actually works, right? So you can give the agent the code, right? But the problem with that is if you go and give it let's say like your DBD project or airflow project it's redriving the context every time from scratch right on every single prompt which means it's probabilistic um it's bounded by the context window which doesn't always work very well and then every time you start a new session you have to go and rebuild that context from scratch.

那我们就说，好吧，试试看给它数据库访问权限。所以我要把 Snowflake 或 Databricks MCP 接入它。嗯，这样它就能看到数据行，也能抽样查看一些数据，但它并不真正理解每一行数据的含义。所以它能读取数字，却未必能正确解释它们。第一种情况是，有的系统能读代码，却没有记忆能力，无法保留你花了很多时间来回沟通、逐步建立起来的上下文知识；另一种情况是，它能看到数据，却无法将数据与其背后的含义联系起来。所以，两者都没有形成关于系统实际运行方式的持久模型，而且……Then let's say okay let's try and give it database access. So I'm going to connect my snowflake or datab bricks MCP to it. um and instead it can see the rows and then it can pro it can it can see sample some data but it doesn't actually have an understanding of what each row actually means. So it can read the numbers but not necessarily explain it right. And so the first situation is one can read uh uh one can read the code but doesn't actually have memory to be able to persist the knowledge of the context that you spent a lot of time going back and forth and having bu build up and the other sees the data but doesn't actually associate the meaning behind it. So neither has like a persistent model of how the system actually behaves and

代理需要了解的四件事Four facts an agent needs

这就是我们着手解决的使命。那么我们先退一步，准确说说为了让智能体能够有效地推理这些复杂的数据变更，你需要向它提供的上下文中究竟缺少什么。可以说，有四条事实决定了一个答案是否正确，但它们实际上都没有写在任何智能体能够读取或推断的地方。第一，一张表中的一行究竟代表什么。这就是我们之前谈到的粒度。接下来是某个连接是否安全。然后是某项指标究竟是什么意思，以及各项之间有什么关联，对吧？在数据系统中，依赖关系非常重要。[嗤笑] 智能体完全可以读懂你的 SQL，但在查询时——也就是它最需要收集准确上下文的关键时刻——它却很难看清这四件事。that's the mission that we set out to try and solve. So let's like take a step back and name exactly what's missing from the context that you need to actually give the agent to be able to reason effectively across these uh uh complex data changes. There are four facts let's say that decide whether an answer is right and none of them are actually written down anywhere that the agent can read or infer. So the first thing is what what one row actually represents in a table. This is what we talked about the grain. The next thing is whether a join is safe or not. And then there's what a metric actually means and what relates to what. Right? The dependencies are very important um in data systems. [snorts] And so an agent can read your SQL just fine. Um but it can't see these four things very easily at query time when it's the most important time for it to gather the the exact context that it needs.

所以，我们所做工作的目标是看看如何让它们变得可计算、可查询，从而使智能体能够在需要的时候获取所需的上下文，而不必在每次会话中都重新构建。接下来的几张幻灯片会介绍我们具体是如何做到这一点的。嗯，然后。So the goal of what we did was let's see how we can make them computable and queryable so that the agent is effectively able to grab just the context it needs just at the time it needs and not have to rebuild it during every single session. So the next few slides are about how we actually went about doing that. Um and

连接语义与扇出Join semantics and fan out

那么我们以 join 这个概念为例。这就像是上一张幻灯片中提到的第二个事实。它是个很好的例子，因为它最清楚地说明了为什么仅靠语法实际上不足以让智能体有效运作。这里有一个 join，我们称之为等值连接，它是按某种特定方式写出来的；但根据底层数据的不同，它可能产生四种截然不同的结果，对吧？它既可能保留原有行，也可能查找一个维度；如果是一对多关系，还可能悄无声息地复制每一行。SQL 中经常用到的 between 子句也是一样。它既可能是安全的“截至某日期”查找，也可能是一个实际上会让数据粒度膨胀的日期序列，对吧？so let's take for example the join idea. This is like the the second fact that we talked about in the previous slide and this is because let's this is a good example because it's a cleanest example of like why syntax isn't actually enough to be able to get the agent to be effective. So there's one join, we call it like an an equa join that's happening here that's written a certain way, but it could have four completely different outcomes depending on the underlying data, right? So it can either preserve your rows, it can look up a dimension, or it can silently replicate every single row when it goes from like a one to end type of relationship. The same thing is for this between clause that happens oftentimes in SQL. So it could be a safe as of lookup or a date spine that actually explodes your grain, right?

这样，智能体不仅能看到这种语法，还能实际理解这种语法会导致哪种情形。这其实并不是模型或能力方面的问题，而是上下文方面的问题；我们实际上是在努力为智能体提供这类上下文。这里的第四个事实，也是我比较喜欢拿来展示的一个有趣事实，就是理解哪些模型、哪些数据资产彼此相关。对吧？比如，哪些模型属于营收领域，哪些属于产品领域；而这些通常也与实际尝试访问或运行数据、以获取这些领域核心指标或数字的人所对应的角色有关。如今，团队往往会根据文件夹名称来组织这些内容，而很多时候，弄清楚哪些模型与哪些业务部门相关，靠的都是口口相传的经验。So the agent can see this syntax but it can actually understand which of these scenarios is going to result because of this syntax syntax. That's not really a model or capability problem. That's like a context problem that we're actually trying to arm the the the agent with. The fourth fact in here and this is one of the more fun ones that I like to um to show off is that um understanding of like what models what data assets relate to other ones. Right? So this is which models are in the revenue domain for example, which are in the product domain and those oftentimes correlate to the personas of the people that are actually trying to access or run the datas to access these uh the core metrics or numbers that are in these uh in these domains. Oftentimes today like teams will organize it based on folder names and a lot of times it's tribal knowledge to understand which models relate to which um business units.

但实际上，你可以通过数据的结构来发现它，还能做得更多。例如，你可以看看模型之间是如何关联的，比如查看血缘关系，包括表级血缘和列级血缘；还可以查看它们是否共享键或命名方式。这样，系统就会自动将它们聚类到各个域中。于是，当你问代理：“对于 RevOps 团队，我的收入域里有哪些模型？”时，这就变成了代理可以直接执行的查询，而不是一个必须反复向人类追问的问题，代理不必通过来回沟通才能弄清楚哪些模型属于同一个域。这其实就是上下文工程中反复出现的同一原则：把存在于某个人脑海中、或散落在其他文档里的隐性知识，尝试通过计算得出，并让代理能够查询。But you can actually discover it from the you can actually do more and actually discover it from the structure of the data. So for example you can take a look at how the models actually connect looking at lineage for example table level lineage column level lineage looking at shared keys looking at shared naming. So the system clusters them automatically for you into the domains so that when you go and ask the agent okay which models are in my revenue domain for the revops team um it becomes a query that the agent can actually run and not a question it has to ask like you know that has to go back and forth with the human to be able to understand which are in the same domain. So this is kind of the same principle for context engineering that you see over and over again. It's you take something that is tribal knowledge that lives in someone's heads or lives in some documentation elsewhere and you try and actually compute it and then make it queryable for the agent.

确定性提取与数据画像Deterministic extraction and data profiling

嗯，那么这些事实实际上是怎么生成的呢？这里我们经常谈到的一个关键原则是：首先要尽可能采用确定性方法，只有在需要消除歧义，或者要完成模型确实特别擅长的事情时，才使用 LLM。这样，你就可以直接从实际的 SQL 中计算出粒度和血缘。我们的做法是获取所有转换逻辑，并一直解析到底层的 AST，从而真正理解数据的结构。这样一来，智能体就不会偏离实际实现，因为我们提供给它的图和上下文，实际上都是从实现本身推导出来的。嗯，然后你还可以做一些事情，比如根据实际数据验证连接是否安全。Um, so how do you actually produce these facts, right? And the key principle here that we talk about a lot is like you want to always be as deterministic as possible first and then only use LLMs when you're trying to resolve ambiguity or do things that models are actually really good at, right? And so you're able to compute grain and lineage straight from the actual SQL itself. So what we do is we actually take all the transformation logic and we parse it all the way down to the as to actually understand how the data is actually structured right and that way the agent can't drift from the actual implementation itself because the the basis of the graph and the context we're giving it is actually derived from the implementation itself. Um and then uh and then you can do things like verify join safety against the actual data.

对吧？那么连接键真的唯一吗？这件事可以通过分析表来完成，对吧？你可以去分析数据仓库中的表。这样一来，智能体就不必猜测诸如基数这类信息。然后，对于其中那些需要人类可读内容的部分，你就可以使用 LLM，对吧？比如，如果你想了解业务名称、概述模型的用途，或者消除前两层中的任何歧义，这时 LLM 就能发挥作用。这些事实会先计算出来，再与你的数据进行核对，因此 LLM 最后只是在我们构建的图中为这些数据资产命名，从而确保我们没有偏离核心实现。Right? So is the join key really unique? This is something that you can do by profiling the tables, right? So you're able to go and profile the the tables that you have in your warehouse. Um, so that allows the agent to not really guess about things like cardality. And then you can use the LLM for things where the human readable parts of it, right? So if you're trying to understand business names or summarize the utility of models or resolve any ambiguity from like the first two layers in there, that's where the LLM can really come into the picture. um these facts are computed and then checked against your data so that the LLM um uh just kind of names these data assets at the end on the graph that we're building and that allows it to make sure that we're not actually drifting from the core implementation.

这种区分很重要，因为，呃，它……That separation matters because um it

持久化一个可查询的上下文图Persist a queryable context graph

它会让核心事实与实际实现以及数据本身关联起来，某种程度上就像我们之前讨论的那样。这一切都很好。那么，这些内容都存在哪里？我们把它称为数据上下文层。说到底，它是一个图，对吧？它只计算一次，然后在每次提交时原子性地刷新，对吧？因此，随着数据团队不断提交更多数据变更，我们会逐步修改这个上下文图。这样一来，它始终能够反映生产企业数据的最新状态。嗯，之所以采用图，是因为访问模式，对吧？之所以用图，而不是一堆文本或一套内容丰富的 Markdown 文件，是因为访问模式。这样，智能体只需遍历一次，就能准确提取所需的子图。嗯，这也避免了提示词塞得过满或上下文塞得过满之类的问题，而且扩展能力远超 500 个模型，因为你不必每次构建上下文层时都重新读取整个项目。keeps the core facts tied to the actual implementation kind of like we talked about and to the data itself. So that's all good and great. Where does this all live? What we call we call it the data context layer. At the end of the day, it's a graph, right? And it's computed once and then on every commit, it's refreshed atomically, right? So, we're incrementally modifying the uh this context graph as data teams are making more data commits to it. So, it's always representative of the the freshest state of uh your production enterprise data. Um, and the reason is a graph. The reason it's a graph and not just a pile of text or the robust set of markdown files is because of the access pattern, right? So the agent's able to pull exactly the the subgraph it needs in one traversal. Um, and it avoids things like prompt stuffing or context stuffing and it scales way past 500 models because you're not like rereading the project every time you're trying to build the context layer.

对吧？过去几年，我们显然看到图数据库又大火起来了。我一直都在想，甚至在 AI 出现之前，它的主要用途到底是什么。但事实证明，模型和 AI 非常擅长获取上下文、编写 Cypher 查询，然后遍历图。而且数据资产的性质决定了它们之间有很多依赖关系，所以非常适合在图中创建这些有类型的对象、节点和边。嗯。哦，你们看不到幻灯片。哦，Right? over the last few years, we've obviously seen a huge resurgence of like graph databases. And I've always like wondered even like before AI like what the main utility of it is, but it turns out that models and AI is very very good at being able to to grab context and be able to write cipher, be able to write a query against it and then traverse. Um, and the nature of how data assets are, it's very dependency based. So it yields really nicely to be able to create these uh typed objects and nodes and edges in this graph. Um. Oh, you guys can't see the slides. Oh,

主持人：大家能把灯关一下吗？我不知道。不行。哦，好吧。嗯，这样好多了。好一些了。我们再把灯调暗一点。抱歉啊各位。我应该想到黑色背景和其他颜色的问题。嗯，不过我很乐意之后把这个演示文稿也发给大家。好吧。嗯，真抱歉。嗯，那么，嗯，我们来看看，你们能看到吗？也许看不到，我想。嗯，这里有一个子图的示例，而你们实际上看到的是，边携带了很多溯源信息，对吧？所以这些边定义了数据资产之间的不同类型的关系。比如说，嗯，呃，“由……物化”以及“具有语义定义”。这些节点都代表图中不同类型的数据资产。Host: can you guys turn off the lights? I don't know. No. Oh, okay. Um, oh that's better. Getting a little better. We dim them a little bit more. Sorry guys. I should have uh thought of the the black background and other colored. Um, but I can happy to send this out afterwards also the deck. Okay. Um, apologies for that. Um so um yes let's see so you can see or maybe you can't see I guess um but there there's like a sample of like a subgraph in here and you can and what you're is actually represented is that the edges carry a lot of provenence right so the edges are defining different types of relationships amongst the the data assets themselves we have like for example um uh is is materialized by has a semantic definition and these are all nodes that are representing different types of data assets in the actual graph itself.

所以会有一个节点，比如说代表一个语义视图，然后还会有一个节点表示一个物理表，也就是表本身逻辑代码的实际物理表示。之后我会把这个发给你们，这样你们就能仔细看看，或者我们再想办法发给你们。嗯，所以，没错，这个就像是……So there'll be like a node that represents a semantic view for example um and then there will be a node that will present like a physical table which is like the actual physical represent representation of the logical code um of the table itself. Um I'll send this to you guys afterwards so you can take a closer look at it or we'll figure out how to get it to you guys. Um and uh and so yeah and so this is like

行动前分析下游变更Analyze downstream changes before acting

本质上就是子图的样子。嗯，所以我们回过头来想想，有了这个上下文层之后，智能体现在能够解决哪些相关用例。嗯，第一个就是影响分析，对吧？智能体提出一项变更，比如修改模型的粒度，对吧？在它真正执行变更、写下一行 SQL 之前，它可以沿着图向下游遍历，并询问每个下游依赖节点：它试图修改的这项变更，是否会破坏你在这个节点中表示的粒度？essentially like what a subgraph will look like. Um and so if we go back and think about the use cases that are relevant that the agent can now solve because of this context layer that we have um is is impact analysis is the first one right so the agent proposes a change um for modifying like a model's grain for example right before it actually does that before it writes a line of SQL to make this change it can actually walk downstream the graph and ask each node that is a downstream dependency of this node it's trying to change does this actually break the grain that you have represented in this node here.

所以这里它能够捕捉到这样的问题：如果我试图把月度粒度的变更转换为周度粒度的变更，那么月度 AR 指标就会在每周的行中重复出现；而下游有某个 ARR 指标会按月汇总这里的所有不同值，结果可能会让这个指标达到实际准确值的 4 倍左右，对吧？接着，这个问题可能会波及下游的所有内容，比如你在董事会层级的仪表板，因为上下文图涵盖了整个数据系统，从源系统一直到 BI 层和仪表板层。因此，智能体能够查看整个数据生命周期中的数据沿袭，并一路追踪过去。所以关键在于，这一切都发生在正式执行之前，对吧？So here it's able to catch that um uh a monthly AR measure would get repeated across weekly rows if I'm trying to go from the monthly grain change to the weekly grain change and that downstream there's some ARR that's actually summing all of the different um values for this on a monthly basis which would result in let's say the metric having like a 4x um of the actual value amount that it's actually accurate right and then it could land on let's say everything downstream on like a board level dashboard that you have because the context graph is inclusive of your entire data system. Everything from like source systems all the way down to the BI layer in the dashboards layer. So the agent has visibility and able to walk the lineage all the way across the the data life cycle if you will. So the key part is this all happens pre-flight, right?

这不是在构建时发生的，也不是在生产事故期间发生的。它甚至还没采取行动，就已经知道影响范围了。然后这里的第二个用例是联接建议，对吧？所以这是事实二。嗯，所以这就像是这个智能体……It's not at build time. It's not in a production incident. It knows a blast radius before it actually even acts on it. And then the second use case here is join recommendations, right? So this is fact two. Um so this is like the agent

安全连接和指标定义Safe joins and metric definitions

想要进行归因——这里的练习是，智能体想要按营销渠道对 ARR 进行归因，对吧？它不会仅仅因为能理解列名就选定一条路径，而是会实际检查每个连接键是否唯一。这是通过分析实际表本身来完成的，并且会根据数据进行验证，对吧？然后，它会给出建议，指出要实现这一点，实际安全的连接路径是什么，同时标记出那些会导致数据扇出、让行数变成 7 倍，或者会在下游悄无声息地丢失 12% 行的路径。所以，关键在于，我们给出的建议有实际的推理和证据作为依据，并且立足于数据本身。然后是安全的代码更改，对吧？这是第三点。wants to attribute the the exercise here is like the agent wants to attribute a arr by marketing channel right so it doesn't pick a path because of like column names that it's able to understand it actually checks whether each join key is actually unique and this is done because we're profiling the actual tables themselves um and it's verified against the data right so then it has the recommendation of what the actual safe join path is to make this happen while flagging the ones that will actually cause like a fan out or like a 7x um in in rows or one that will suddenly like silently drop 12% of the rows downstream as well. So we're making recommendations with actual reasoning and evidence is the key here and it's grounded in the data itself. And then safe code changes, right? This is the third fact.

你经常会发现，业务定义其实往往嵌套在 SQL 本身中，对吧？所以，如果你让一个智能体去整理某个度量或指标，而它删掉了一个谓词，那么从语法上看是行得通的，SQL 仍然可以编译通过。但实际发生的情况是，这个谓词可能看起来无伤大雅，却其实是业务定义中非常重要的一部分。比如这个例子里，它删掉了“付费”的定义，于是悄无声息地重新定义了这个指标，把试用期流失也纳入其中，而这个指标一路下游被三个仪表板所使用。这个上下文层实际上能在业务定义和业务指标的变更真正实施之前捕捉到这些变化。Oftentimes you'll find like business definitions. The reality is they're oftentimes nested in the SQL itself, right? And so if you're asking an agent to let's say tidy up some measure or some metric that you have and it removes a predicate um syntactically that will work right the the SQL will still compile um but what it's actually happening and it it could could look harmless but the predicate was actually a very important part of the business definition itself right so here in this example it's dropping the definition of um of paid right so it it's silently redefining what the metric uh the metric to include trial churn uh across all the way downstream three dashboards that are actually happening representing this metric. Um so this context layer actually catches this uh change in business definitions and business metrics before it's actually implemented.

嗯，所以这就像是，如果你在构建智能体，就要把这些设为一等约束。它们不只是写在某个 YAML 文件里的注释之类的东西。这……Um and so this is like if you're making the agent ser like have these as like first class constraints. It's not just a comment that is in some YAML file or something like that. That's

评估数据工程任务Evaluate data engineering tasks

关键就在这里。这就是语法层面的重构，与人类真正能够信任的改动之间的界线。所以这些都很好，但它真的能用吗？对吧？所以你必须对它进行衡量。每当你构建任何具有智能体特性的东西时，都必须能够对它进行衡量。我们采用评估驱动开发，这也是其中非常重要的一环。基本上，我们会创建一系列真实的生产环境数据工程用例、任务和工作流，难度从简单到盲测不等。每项都会运行多次，并且都在实际数据环境的隔离副本上运行，这一点非常重要，这样这些环境之间就不会发生实际的数据泄漏。之后，我们会运行通过/失败检查，核验实际输出。the key here. And that's the line between like a a syntactic refactor and a change that a human can actually trust. So all good and great, but does it actually work? Right? Um and so you have to you have to measure this. Anytime you're building anything agentic, you need to be able to measure it. Um and so we practice like eval driven development, which is a very important part of it. We're essentially going and creating a bunch of like real production data engineering use cases and tasks and workflows that range in difficulty from like easy to blind and they each run multiple times and they're is they're each running on isolated clones of the actual data environment which is a very important part so that there's no actual leakage that happens across these environments and then we have pass fail that's running checking the actual output of it.

我们也有一些 LLM-as-judge 在做影响分析之类的事情。嗯，这里的关键发现是，虽然使用的是相同的模型、执行的是相同的任务，唯一变化的只是智能体能否访问数据上下文层。因此，当它拥有合适的上下文时，你会看到明显的提升。它能够针对这些非常复杂的变更进行推理，任务成功率也大幅提高。配备数据上下文层后，成功率几乎高出 35 个百分点。随着我们与更多团队合作，摸索他们各自定制的用例，这个基准测试显然还会不断扩大和演进。不过，信号是一致的，对吧？你能提供给智能体的上下文越好、越丰富，结果就会发生显著变化；相比之下，仅仅更换模型，或者给模型本身添加更多通用工具，比如 MCP 工具，效果就没那么大。We have some LLM as judge doing like impact analysis things as well. Um, but the key findings here is that although it's the same models, it's the same task, the only variable that's changing is where the agent has the access to the data context layer. And so you see a material lift um when it has this right context. It's able to reason across these really complex changes and the task success rate goes up tremendously. It's almost 35 points higher when it's armed with the data context layer. And this exact benchmark is obviously going to keep growing and keep evolving as we're working with more teams to figure out their kind of um bespoke use cases in there. Um but the signal is consistent, right? The better context um the more rich context you're able to give the agent, the outcome changes tremendously than just let's say like swapping models or adding more generic like tools, MCP tools um to to the model itself.

五种上下文工程模式Five context engineering patterns

那么，我就讲到这里。不过在结束之前，显然我会把时间交给 Brandon；但先来说说这五条经验。即使你们没有在构建完全相同的系统，这些也是我们在过去大约一年的开发过程中，确实吃了不少苦头才总结出的关键经验。第一点是：只要可能，就始终采用确定性的方式。不要试图把整个 DBT 项目或 Airflow 逻辑都扔给模型，让它去完成这些棘手复杂的任务。那行不通。你真正需要做的，是以确定性的方式，深入、细致地理解转换逻辑。因此，彻底解析它，以便理解其结构、关系和依赖关系，这一点非常重要。So, I'll wrap up here, but before I wrap up, the the five patterns, and I'll hand it off to Brandon, obviously, but the five patterns that are even if you're not building this exact system, these are some key takeaways that we've really learned the hard way actually over the last like year of us building this, right? The first thing is like deterministic always when you can, right? Don't ever try and take like your entire DBT project or your Airflow logic and just throw it to the model to be able to try and do these tough complex tasks. That that won't work, right? Um what you really want to do is be able to get a really deep and detailed understanding of the transformation logic in a deterministic way. So parsing it all the way down to the as to understand the structure the relationship the dependencies are very very important.

然后像能够正确计算粒度这类事情，也是可以用上的。我们花了很多时间进行改进，比如使用关系代数，针对数据生命周期的每个阶段进行实际的计算。嗯，在需要的时候再使用模型——比如总结模型、总结不同的数据资产，或者对模型进行分类或归类，这些都是模型非常擅长的事情。第二点是，存储证据，而不是结论。因此，很多时候最好只保留信号，然后让智能体自行推理，因为这些模型能力很强，也很聪明，能够在查询时有效地进行推理。第三点是，图遍历总是胜过往提示词或上下文里塞内容，对吧？嗯，应该让它提取所需的子图。And then things like being able to compute grain right like that's something that you can use. We spent a lot of time refining and like using like relational algebra to actually calculate and compute for each stage of the data life cycle. Um and then use the model when you need to when it's like summarizing models or summarizing the different data assets or categorizing or classifying models are really great for that. The second thing is store evidence not conclusion. So it's oftentimes good to just keep the signals and then let the agent be able to reason because these models are very capable. They're very smart. They can reason at query time effectively. And then the third thing is that traversal always beats prompt stuffing or context stuffing, right? um it's going to want to let it pull the subgraph that it needs.

不要在运行时把整个代码库都 dump 出来。它会拉取子图，并能够遍历其中的内容，从而找出如何解决当前任务，因为所有依赖项和其他内容都清楚地映射在那里。这样一来，随着任务逐渐被这些智能体解决，上下文会小得多，自然也就能让 AI 运行得更高效。第四点是掌控工具层。我们发现，经过精心筛选的一套工具，总是胜过一大堆现成的 MCP。比如，即使是我们已经使用的、能让智能体访问 Snowflake MCP 或 Databricks MP 的那些 MCP，我们也发现自己重写了其中很多部分，以便让它们更符合我们想要实现的能力和功能。Don't dump the entire repository at runtime. It'll pull the subgraph and be able to traverse to figure out how to solve the task at hand because all the dependencies and everything is clearly mapped out there and the context when naturally you'll have be running more efficient AI because the context is going to be a lot smaller every time um the task is trying to be solved by these agents. And then the fourth thing is like own the tool surface. So like what we found is a curated a curated type tool set always beats like a buffet of just like off-the-shelf MCPS. Um even like for the MCPS that we do have that give our agent access to Snowflake MCP or data bricks MP we found ourselves like rewriting a lot of it to kind of match the capabilities and functionality of what we're trying to accomplish.

嗯，然后，嗯，假设第五点是我们刚才谈到的评估，对吧？这是最重要的一点：无论智能体在做什么，如果你无法衡量它，就无法真正构建出可靠的智能体；你也无法从任何事情中吸取经验。嗯，所以要尽早并持续投入评估基础设施。它是你软件中一个持续运转、不断演进的组成部分，你必须始终维护、更新，并持续运行。好了，作为本次工作坊这一部分的收尾，到目前为止，这还算不上真正的工作坊。我只是有点漫无边际地说了一通，不过现在我们要进入有趣的部分了。嗯，我们发现，智能体在数据工程或数据任务上表现不佳，并不是因为它们不会读写 SQL。它们会，而且做得非常好。它们失败的原因在于，不了解或不理解数据的实际含义、数据如何流转，或者数据发生变化时会有什么东西出问题。Um and then um let's say the fifth thing is we talked about like eval right this is uh the most important thing you can't really build a reliable agent regardless of what the agent is doing if you can't measure it you can't learn from anything you can't measure um and so invest in like the eval infrastructure early and often and it's a continuous like living breathing uh piece of your software that you have to always maintain and up and and update um and run continuously. So to close uh this portion of the workshop, this hasn't been a real workshop so far. It's just me kind of rambling, but we'll get to the fun part now. Um what we found is the agents don't really fail at data engineering or data task because they can't read or write SQL. They do that. They do a very good job of that. They fail because they don't have or they don't understand what the data actually means, how it flows, or what breaks when it changes.

对吧？所以，如果你把这个上下文真正提升为一等公民——提取它、计算它并持久化它——那么智能体就能通过图谱以非常高效的方式查询它。这样一来，你就能跨越一道界限：让智能体从只能读取代码，变成真正能够操作数据系统的智能体。这就是我们通过这个数据上下文层构建出来的东西。好了，感谢大家听我做这个简短的分享。接下来我想把时间交给 Brandon，他会带大家开始一场更偏实践的工作坊，并延续我们刚才提到的一些概念。我就不在这里留时间提问了，因为我想确保 Brandon 有足够的时间完整地带大家做完工作坊。不过结束后我会在附近，大家随时可以来找我。Right? So if you make that that context really a first class citizen um you extract it, you compute it, you persist it, and then the agent's able to query it in a really efficient way because you have a graph. Uh then you're able to cross a line between from an agent that only reads your code to one that's actually ready to like operate your data system. And that's like what we've built with this like data context layer. All right. Um so thank you for that like little brief talk um that we've given here. I want to hand it over to Brandon who's actually going to like start walking you guys through more of like a hands-on workshop that carries over some of the concepts we talked about in here. Um, I won't take any questions out because I want to make sure that we have enough time for Brandon to go through his entire workshop, but I'll be around like after this and whenever like feel free to reach out to me.

我很乐意就此继续聊。谢谢。I'm happy to chat more about this. Thank you.

Brandon 的编码智能体研讨会Brandon's coding agent workshop

做得好。好啦，大家今天过得怎么样？状态不错。太棒了。抱歉，请稍等我一下。好了，我今天要讲的主题，嗯，肯定和 Yonyi 刚才讲的内容有关。我们会尝试把这些概念结合起来，整理成一种我们能在接下来的时间里实际动手实践的形式。嗯，我叫 Brandon，来自 Typed。这周的会议上，你会听到很多演讲都在讲：智能体的能力取决于你提供给它的上下文，而构建这些上下文本身就是实际要做的工作。所以，这基本上就是对这一课题的实操版。在接下来的 90 分钟里，我们将一起为一个编码智能体构建一个上下文层，使用真实的代码库来做真实示例。今天我们会用到 quadrant 代码库。Good job. Okie dokie. How is everybody doing today? You're doing good. Fantastic. Sorry, give me one moment. All righty. So, the topic that I'm going to talk about today um is uh definitely related to what Yonyi was talking about. We're going to try to bring these concepts together in a form that we can actually do in uh the rest of our time here. Um so, my name is Brandon. I'm at Typed. Um, and you're going to hear a lot of talks here at the conference this week about how agent is only as good as the context you give it and that building that context is in fact the actual job. And so this is basically a hands-on version of that exact same lesson. Um, over the next 90 minutes, we're going to build a context layer for a coding agent together, a real one over a real example codebase. We're going to be using the quadrant codebase today.

嗯，接下来我要展示最后一部分：即使你构建了这些很棒的工具，嗯，智能体大多还是会忽略它们，除非你在过程中引导它。嗯，真正让它使用这些工具，才是工作重点。嗯，在我们继续深入之前，如果你想跟着一起操作，可以扫描这个二维码，它会链接到我的 GitHub 仓库，里面有所有相关代码。嗯，仓库里有几个 Jupyter 笔记本，方便大家跟着操作。再说一次，如果你不想配置好所有东西，也不用担心。这里的依赖项是用于运行 Neo4j 的 Docker Compose 或 Docker。嗯，如果你想实时试用其中的 LLM 部分，可以使用 OpenRouter 密钥。不过，这不是必需的。Um, and then I'm going to show you uh the last part, which is that even if you build when you build these amazing tools, um the agent's mostly going to ignore it unless you help it along the way. Um, getting it to actually use the tool is going to be uh where the work really is. Um, so before we go ahead and dive in, um, if you want to go ahead and follow along, this QR code links to my GitHub repository, which does have all of the relevant code. Um, that repository has a few Jupyter notebooks for people to follow along with. Again, don't worry if you don't want to have set that set everything up. The dependencies there are uh docker compose or docker for running neo4j. Um and if you do want to try out the llm portions of it live, you can use an open router key. However, that's not required.

如果你愿意，整个过程都可以离线运行；也可以跟着我一起，看我在这里一步步搭建。嗯，我再把这个页面多留一会儿，然后会稍微讲解一下，方便想设置的人做好准备，接着我们就开始看笔记本本身。好，大家都扫好了吧？准备开始了吗？再等一分钟。It'll all run offline if you desire or you can just follow along with me uh as I build it up here. Um so, uh I'm going to leave this up for just a little bit longer and then I'm going to start doing a little bit of talking for people to be able to set things up if they want to and then we'll dive into the notebooks themselves. Okay, everyone got the scan in. Ready to go? Give it one more minute.

两个同名类型Two types with the same name

好，太好了。那我们就从每个编码代理都会开始的地方——文本搜索——开始吧。嗯，我要查一下这个名为 posting list 的结构体类型。它是 Qdrant 中一个真实存在的类型。代理运行时会执行 ripgrep。你看，太好了，它找到了两个完全匹配的结果。它们都正好是 posting list。不过我们还能看到另外十几个相似项，比如 posting list view、posting list iterator，等等。你的代理也会做同样的事。它会用 grep 搜索，得到这一大堆结果，然后选一个，对吧？它可能选对，也可能选错。它会先去探索选中的那个。也许还得回头、回溯，再读下一个，才能弄清楚你真正想要的是哪个，对吧？All right, perfect. So, let's start where every coding agent starts, which is text search. Um, I'm going to ask about this type for this strct called posting list. Uh, it's a real type in quadrant. Um, so when the agent goes, it runs rip grip. Um, and you see great, it found that we have these two different um, answers that are exact matches. They're both exactly posting list. Well, we see it has a dozen more look alikes. You know, we have a posting list view, a posting list iterator, etc., etc. Your agent's going to do the same thing. It's going to GP. It's going to get this wall back and it's going to pick one, right? It may pick the right one. It may pick the wrong one. It's going to explore that one first. Maybe it has to go back, backtrack, read the next one to actually figure out what you want, right?

但它其实没有任何办法知道你想要的是哪个倒排列表，对吧？它、它、它不行，文本本身没有任何关于身份的概念，也不知道哪个倒排列表才是真正有意义的。嗯，你关心的那些东西，比如用了哪些方法、它在哪里被使用、它属于哪个 crate，并不一定只存在于文本里，而是存在于代码本身的实际结构中。嗯，我们刚才看到 Yonyi 针对数据提出了一个非常类似的观点。而让智能体每次运行都在没有记忆的情况下重新推导整个系统，是有问题的。嗯，因为首先，这会很昂贵，对吧？它每次都得重新读同一批文件。如果你把同一个问题问它五次，它每次都会回去把那些文件再读一遍，对吧？But it doesn't really have any means of knowing which posting list you want, right? It it it can't the text itself doesn't have any concept of identity or you know which posting list is actually you know meaningful. Um this the the things you care about you know what methods use it where it's used what crate that it's in that's not necessarily just in the text it's in the actual structure of the code itself. Um, so we just watched Yonyi make this, you know, a very similar case for data. And it's it's problematic to have the the agent redive the entire system for every run with no memory. Um, because for one thing, it's going to be expensive, right? It's going to go through it's going to have to reread the same set of files every time. If you ask it the same question five times, it's going to go back and read all those same files every time, right?

但重要的是，它很脆弱，对吧？它回过头来读取那五个文件时，下次可能会按不同的顺序读取，也可能先读到一个给出错误答案的文件，进而让它去另一个文件里徒劳地追查，对吧？所以，我们在为这类智能体构建上下文时，真正想要解决的就是这个问题：如何让它具有确定性，而不是那么脆弱，对吧？我们的目标不一定是让智能体变得准确得多或高效得多，尽管有时确实会如此。真正的目标是让它具有更强的确定性，对吧？因为我们正试图尽可能地为这个高度非确定性的系统增加确定性。嗯，如果没有一个持久且可信的代码模型，它每次都会靠猜。But the important thing is that it's fragile, right? when it goes back and it reads those five files, it might read them in a different order next time or it might read one file first that gives it the wrong answer and sends on a goose chase to another file. Right? So, it's that issue of actually having this thing be deterministic and not as fragile uh that we're really trying to accomplish when we're trying to engineer context for these kinds of agents, right? The goal isn't necessarily to get the agent to be a lot more correct or a lot more efficient, even though that's sometimes the case. The real goal is to have it be a lot more deterministic, right? because we're trying to apply as much determinism as we can to this very non-deterministic system. Um without you know this persistent trustworthy model of the code, it's going to guess um every time.

而且你知道，我们看到的是，随着时间推移，模型在猜测方面变得越来越厉害，对吧？你知道，我们都能看到，一年前运行 cloud code 和现在运行它时的情况有什么不同，对吧？And you know what we see is that over time the models have been getting better and better and better at guessing, right? You know, we can all see what happened when we ran cloud code a year ago versus running it now, right?

提取、持久化并查询代码上下文Extract, persist, and query code context

这是完全不同的体验。所以，如果我们想为代码构建一个上下文层，基本上就是要遵循完全相同的做法，只不过层级稍微不同一些。我们会把规模缩小，这样就能搭建一个工作坊，让你今天就能实际运行，而不是试图复刻我们整个产品。基本上，我们要采用“提取、持久化和查询”这个概念，开始从代码库中提取信息。我们会提取符号相关的信息，也就是每一种类型、每一种 trait、每一个函数，以及它们真实的身份和位置。我们还会提取这些不同符号之间的关系，在图中建立可以遍历的边。It's completely different experience. So like if we want to build a context layer for code, we basically want to follow this exact same recipe um but um just at a slightly different uh level, we're going to scale this down so that we can build a workshop and you can leave it something that you can actually run today um instead of trying to replicate all of our uh our entire product up here. So we're basically going to take um that that concept this exact this extract persist and query um to basically go ahead and start extracting information out of our codebase. So we're going to pull out information about the symbols you know every every type every trait every function uh with its real identity and location. Um we're going to the relationships between these different symbols get uh edges inside of our graph that we can traverse.

这些内容只会计算一次，但之后会被查询很多次。这和我们处理庞大数据架构时采用的整体方法是一样的，只不过这里用的是一个小到足以在这个房间里运行的示例。嗯，所以没错，这基本上就是我刚才说的内容。呃，我们今天就把它展示出来。所以，这是一个你们在这次会议期间会经常听到的概念。嗯，我觉得这真的很酷。我提到了——我做了张幻灯片，讲上下文的三个层级，结果一看日程安排，发现明天有人要讲上下文的四个层级。所以我感觉自己还没开始就先被比下去了，不过我们今天还是会讲这个，因为我觉得它对这里的语境仍然挺有用的。嗯，你可以把编码智能体的上下文大致理解为这样几个层级。These get computed once and they end up being uh queried many times. This is the same uh overall approach as what we do when we're tackling a sprawling data architecture just with a small enough example to run in this room. Um and so yeah, that's basically what I was just saying. Uh so we're going to send it up today. So uh this is going to be a concept that you're going to hear about a lot during the conference. Um I thought it was I thought it was really cool. I I brought up I made the slide of three levels of context and look at the schedule and someone has a talk about four levels of context tomorrow. So I feel a bit preemptively oneuped but we're still going to go through this today because I think it's still uh pretty useful for this context here. Um so you can think about context of a coding agent in sort of this terms of levels.

嗯，我喜欢这样理解。第一级就是你刚才看到的那种。它是 GP，是 glob 原始字符串，没有结构，对吧？第二级是如今大多数代码 RAG 解决方案所处的层级，对吧？你会用到语义搜索，对代码建立索引、生成嵌入，对吧？还会进行一些不错的词法搜索和语义搜索，然后得到排序后的结果，也就是所有提到这个东西的位置。但归根结底，这仍然只是一个扁平列表，对吧？这个扁平列表能给出更相关的结果，但它终究还是扁平列表。它不会展示不同代码之间的关系，对吧？所以我们认为，第三级，也就是最高级，是一个可遍历的图。Uh that's how I like to think about it. So level one is sort of what you just saw. It's the GP, it's the glob raw strings, no structure, right? Level two is where you get, you know, most of the code rag solutions that we have today, right? you get um semantic search, you index the code, you embed it, right? You do some nice lexical search, semantic search, you get ranked results, right, of all the, you know, the locations that this this thing was mentioned in. But at the end of the day, that's still going to be a flat list, right? It's going to be a flat list that gives you more relevant results. It's going to be still a flat list. It's not going to show you the relationships between these different code, right? So the the top level or or the highest level at level three is in our opinion a traversible graph.

嗯，一种将符号与真实身份关联起来，并在它们之间添加可供实际遍历的类型化边的东西。嗯，我想在本次工作坊接下来的部分里，我们会逐步构建这些不同层级的内容，最后得到一个能够使用……来回答有关代码库的问题的智能体。Um something that links the symbols with real identity and adds typed edges between them that you can actually traverse. Um so I think for the next part of this workshop we're going to go through and build up through these different levels and by the end we'll get to an agent that can answer questions about the codebase um using

语法、符号解析与含义Syntax, symbol resolution, and meaning

这个可遍历图。嗯，呃，是的，我们来把它梳理一下。构建这个图时，要经过几个不同的台阶或层级，才能达到第三级的理解，也就是理解我们是如何做事的。第零级是语法层面。首先，你需要解析自己试图理解的代码。最简单的办法是使用像 Tree-sitter 这样与语言无关的工具。它能对代码库的不同部分以及它们之间的交互方式，进行非常基础的高层 AST 拆解。不过，它仍然不知道该如何处理名称冲突，对吧？posting list.rs 里的 posting list 符号，和 quant.rs 里的 posting list 符号，是两个不同的 posting list。this traversible graph. Um, and uh, yeah, let's uh, let's let's lay it out. So, as we're building this, there's going to be a few different uh, stepping stones or rungs to get to that level three understanding of how we do things. So, rung zero is sort of the syntactic level. First, you need to actually parse the code that you're actually trying to trying to understand. And so, the easiest way to do that is with something that's language agnostic like treesitter. And that gets you a very basic high-level ASD breakdown of the different um parts of your codebase and how things interact. Um, however, that still doesn't really have any idea how to deal with collisions, right? Posting list, the symbol that's in posting list.rs and the posting list, the symbol that's in, you know, quant.rs are two different posting lists.

它们在第零层还是会发生冲突。所以，关键就在于第一层。GP 真正做不了的就是解析，对吧？解析能把那两个含糊不清的倒排列表变成两种不同且可识别的类型，对吧？所以用 Rust 有点像是在作弊。你基本上可以免费得到第一层，因为 Rust 编译器会帮你完成很多这类工作。嗯，我们基本上会用 Rustdoc JSON，开箱即用地获取所有这些信息。那么，这算作弊吗？算。嗯，我特意选了 Rust，因为我得搭建这个工作坊；而且你知道，要为另一种语言从头构建一个完整的解析器，或者尝试构建类似的东西，难度会大一些。不过最后我会讲一个更实用的 Python 用例。仓库里有一个示例 notebook，大家可以看看；如果今天的工作坊时间允许的话，我们也会讲到它。They're still going to collide at rung zero. So, the name of the game is going to be rung one. So resolution is the one thing that GP can't really do, right? It's the thing that turns those two ambiguous posting lists into two distinct identifiable types, right? So with Rust, it's a little bit cheating. You kind of get wrong one for free because Rust compiler will do a lot of this work for you. And uh you know, we'll basically be using Rust do.json to give us all that information basically out of the box. Now, is this cheating? Yes. Um I picked rest on purpose because I needed to build this workshop and you know building an entire parser for a whole other language or trying to build something out is a little bit harder. However, at the end I will talk about a more useful case of this on Python and there's an example notebook in the repo for you guys to peruse on that if we don't get to it uh during this workshop today.

嗯，接下来的这一层呢，我们还有一个笔记本，但我不确定今天能不能讲到。它是第二级，也就是我们提取的这些不同内容背后的实际含义，对吧？所以，前两层完全是确定性的。我们基本上始终都在努力以尽可能确定性的基础方法来构建我们的理解。因此，我们使用 Tree-sitter 和 rest.json JSON 来构建前两层；如果时间允许，我们还会探讨如何构建第二级——也就是如何获取这些信息并输入到 LLM 中，让它进行分类、总结，以及完成 LLM 擅长的那些事情，比如……Um the next layer uh that we're there is a there is a notebook for I'm not sure we'll get to today is rung two which is the actual meaning behind um these different things that we're extracting. Right? So these first two layers are solely deterministic. We're basically always trying to be ground our understanding in as deterministic a base approach as possible. So we build out these first two layers using tree sitter and rest.json JSON and then if we have some time we'll explore um how we build out that rung two layer like how do we take that information feed it into an LLM to classify and summarize and do the things that the LLM are good at as

笔记本一：构建代码图谱Notebook one: build the code graph

而不是只想着把所有东西都塞进去，然后让它自己理出头绪。嗯，太棒了。那么现在我们来打开 notebook 01，可能会有点困惑。把它挪到这边来。哦，其实它已经在这儿了。好，行。大家能看到这个吗？还是我把它打开在你们那边了？太好了。那么，这个 notebook 要做的就是带大家逐步了解如何为我们的 Quadrant 项目构建不同层级的上下文。我们今天不会为整个代码库建立索引，它有点大。嗯，我们打算看看其中的一个样本，我想是五个 crate。这些代码足以作为我们测试的语料库。嗯，我们可以开始运行这些单元格了。opposed to just trying to throw everything in there and asking it to make sense of it. Um fantastic. So, now we're going to go ahead and open up our uh notebook 01 and going to be confused. Bring this over here. Oh, actually it's already here. Okay. All right. Can folks see this? Okay. Or did I bring them up on your side? Great. So this uh this notebook, what this notebook is going to do is it's just going to walk through the process of building up those uh those levels of different context for our uh our quadrant project. We're not going to be indexing the entire codebase today. It's a little bit big. Um we're taking a look at uh a sampling of I think five crates. Um it gives us a good corpus of code to look over uh for our testing here. Um, so we can start going ahead and running these cells.

所以我们首先要做的就是进行一系列设置。嗯，现在设置已经完成，我们可以开始看看了，嗯，对。啊，它在 GitHub 上。是的，我可以马上再把链接贴出来。把它打开。嗯，链接在这里。如果你不想在本地跟着操作，每个不同的部分里都有 Colab 链接。是的。嗯，如果你想在笔记本的 Colab 中使用图表，就需要 Neo4j 或者一个登录账号。好，这只是那个部分的特殊要求。好啦。那么我们可以看到，这里还有一大段额外的代码，用来在 Rust 本身内部完成所有解析工作。今天我不一定会讲解这部分，但你可以随时在代码仓库里查看。嗯，今天我们这里只用一个神奇的函数，extract rust gate，它会运行 tree-sitter 扫描 Rust 代码库，并提取其中的信息。And so what we do first, we just do a bunch of setup. Um, and now that the setup is good, um, we can go ahead and start looking at the, um, yes. Ah, it is in the GitHub. Yes, I can put up the link again to that real quick. bring that up. Um, so the link is up here. Uh, if you don't want to follow on locally, there are collab links in each of the different um, things there. Yes. Uh, for the collab for the notebook one, if you do want to use the graph, you will need a Neo4j or a login. Okay, that's just a that's just a special thing for that. Okie dokie. So, we can see here um there's a bunch of additional code in here to actually do all of the um parsing inside of uh Rust itself. I'm not going to necessarily go over that today, but it's all available in the repo for you at your leisure. Um we just have a magical function here today uh extract rust gate that will uh pull out the information from running trees sitter across the rust codebase.

呃，然后再来一个，它也会对我们来自 REST 文档的信息执行同样的操作。我们可以看到，前几个 crate 中，我们提取出了 FreeSitter 的一堆符号，然后还有一堆……Uh and then another one that will go ahead and do the same thing but for our information from uh rest dock. So we can see here that from the first couple of crates we've pulled out um a bunch of symbols from freesitter and then a bunch

解析规范符号身份Resolve canonical symbol identities

这些符号的……呃，来自 Rustto do 的这些符号的已解析版本。现在我们可以开始运行一些代码，尝试弄清楚为什么要区分第 1 层和第 0 层。比如，当我们在第 0 层查询 posting list 时，可以看到这个导出的 posting list 版本。不过，我们其实不知道它最初来自哪里，也无法反向解析到它实际定义的位置。呃，这就是第 1 层的作用，对吧？它能指出：这是 posting list，它的规范位置就在这里——posting list、posting list、posting list；而它的实际位置在这个路径的这一行，对吧？这样一来，当某些内容被重新导出到 crate 外部时，我们就能找出它的来源，并将其关联回原始 crate。of symbols uh resolved versions of those symbols from Rustto do. So now we can go ahead and start running some code that's going to start trying to understand why we're differentiating between rung one and rung zero. So we can see when we ask about something like posting list in rung zero we see that okay we see this exported version of posting list we don't know actually where that originates from though we can't resolve that backward into the actual um location where it was defined. Um, so that's what rung one is for, right? It's going to be able to say this is posting list. This canonical location is here at posting list posting list posting list and that actual location is at this path at this line, right? So this is what's going to be able to let us figure out when things get reexported outside of a crate, link it back to the original crate.

一旦我们真正构建出图本身，智能体就能够遍历这些不同事物之间的所有关系。嗯，所以现在我们先快速搭建一下我之前提到的第二层。嗯，我们会有一种索引，在其中构建一个非常……你知道的，我们基本上会收集所有符号，然后让一个函数执行全文搜索。对吧？所以我们进行搜索，就会遇到在图中看到的同样问题。对吧？我们可以搜索 posting list，并获得非常有用的信息。我们可以看到这是 reexport。这是 struct。这也是 struct。但它仍然无法告诉我们这些不同事物之间的关联，对吧？智能体可以推断出来，并且可以……你知道的，想明白：哦，如果 posting list 是一个 struct，那它很可能是在这里被重新导出的。The agent's going to be able to traverse all the relationships between those different things um once we actually build out the graph itself. Um so now we'll just build very quickly the level two that I was talking about before. Um where we have a kind of index where we build a uh very you know where we basically take all the symbols and then we let um a function go that just does a full text search. Right? So we do a search and we get the same problem that we saw in grap. Right? We can search for posting list and we can get very useful information. We can see this is the reexport. This is the strruct. This is the strruct. But it's still not going to tell us the linkages between those different things, right? And the agent can infer that and it can, you know, make the make the figure out, oh, probably that if the posting list posting list is a strruct, it's probably reexported up here.

所以这大概就是它的基础。但你不希望它每次都得做出这种逻辑上的跳跃。So that's probably the basis of it. But you don't want it to have to make that logical leap every

将节点和边加载到 Neo4j 中Load nodes and edges into Neo4j

只要有一次出错，它就会完全变成另一个离谱的猜测。所以我们接下来要做的是构建图本身。我们把刚才提取出的所有不同符号和信息，转换成一系列节点和边，然后填充到 Neo4j 图中。接着，我们就把这些内容全部加载进去。加载到图中之后，我们甚至可以打开 Neo4j，看到一个漂亮的小图，展示代码库各个部分之间的引用关系。现在有了这样一个图，我们就可以专门构建一些工具……single time because it's going to be that one time where it gets it wrong and then it's going to end up on a completely different wild gase. So our next step here, we build out the we start building out the actual graph um itself. So we take that all those different um uh symbols and uh information that we just extracted, we transform that into a bunch of nodes and edges that we can populate into our Neo4j graph. Um and then we go ahead and actually um load all of that in. So once that's in the graph, right, we can even go into Neo4j. Um we can see we have you know a beautiful little graph of u references to all the different parts of our uh of our of our codebase here. Um so now that we have this sort of graph, we can build tools that specifically

构建一个影响范围分析工具Build a blast radius tool

利用图的能力，让我们的智能体能够访问某些有助于它更轻松完成工作的内容。比如，如果我们构建一个工具，想要查看某个函数，并了解修改这个函数会带来哪些变化、导致哪些内容失效，对吧？那么我们可以构建一个函数影响范围工具：传入某个对象的限定名称，也就是规范名称，它就会沿着我们定义的不同边遍历图，尝试准确找出有哪些内容依赖于这个对象，以及如果它被修改会导致什么失效，对吧？这样一来，你就可以让智能体回答一个问题，而这个问题原本需要在多个不同文件中进行多次 grep 才能解决。理论上，智能体只需一次就能给出答案。harness the capabilities of the graph to give our agent access to something that's going to make it easier for it to do its job. So if we build a tool and we want to say okay I want to look at a function and I want to see what's going to change what's going to break if I change this function right so we can build this function blast radius where we say okay you pass in the qualified name of something the canonical name it's going to go in and it's going to try to figure out walk down the graph through those different edges that we defined try to figure out exactly um what depends on this thing what will break if this thing gets changed right and so in that sense you can ask the agent to answer a question that would have taken multiple grapes across multiple different files and in theory um the agent can just answer it in one in one go.

所以这当然没成功，毕竟为什么会成功呢？为什么会成功呢？现场演示就是这样，各位。好，你们可以看到，这里我们说要把影响范围传给这个 struct 时，可以说，好，没问题。它通过这个……关联到这里，这个……有个方法会做这件事……它会在这里返回。因此，只要把内容放进图中，我们就能看到可以建立的所有关联。最后，我们终于解决了碰撞问题。现在，如果我们说有两个不同的倒排列表，想看看它们之间的差异，就可以对这两个倒排列表调用新的 blast radius 工具，然后得到一组不同的结果。更改一个倒排列表，并不会……实际上，如果你更改另一个倒排列表，影响到的改动程度也不会相同。So of course that didn't work because why would it why would it work live demos everybody? Okay, so you can see here when we say we're going to pass the blast radius to this strruct, we can say, okay, cool. It's linked here via it gets returned by this um there's a method that does this um it gets returned here. So we can see all the linkages that we're able to establish um just by having the content inside of the graph. So finally um we get to resolve our collision problem. Um now if we say we have two different posting lists and we want to see the difference between them um we can call our new tools blast radius for this posting list and this posting list and we get a different set of results. Changing one posting list is not going to in fact the same level of changes if you impact change the other posting list.

所以，尽管这些东西的底层名称相同，对它们进行实际的规范解析后，我们就能更明确地说出，当你进一步破坏这个东西时，究竟会发生什么变化，而不只是……So even though these things have the same underlying name, the actual canonical resolution of them results in us being able to more definitively say what's going to change when you break this thing more than just

将智能体连接到图中Wire an agent to the graph

而不是逐个读取所有文件，再试着猜测。嗯，现在我们可以利用目前构建的内容，搭建一个功能非常有限的小型代理来实际使用这套新工具。嗯，既然我们已经有了这个 blast radius 工具，就可以快速接入一个小代理。嗯，这个代理非常简单。它只是一个 Pydantic 代理，配置了一个系统提示，说明它可以运行这些不同的工具。我们还为它编写了一些非常基础的工具，让它能够使用 blast radius，以及 implementer 的方法；这些方法可以处理非常简单的 Cypher 查询，以便提取信息。只要这些信息已经存储在图中，编写查询将其提取出来就会变得相当简单。嗯，只要你能表达出自己想要实现的目标，无论你使用的是哪种图数据库，都很容易编写 Cypher 查询或该数据库所用的等效查询语言，来提取这些数据。going a grap reading all the files and then trying to guess. Um so now we can actually take what we built so far and build a very small limited function agent to actually use this new tooling. Um now that we have uh this blast radius tool um we can wire up a very quick little agent. Um this agent is very simple. It's just a pyantic agent um that uh is just configured with a system prompt that says you have the ability to uh run these different tools and we wrote some very basic tools for it to be able to use blast radius the implementers methods of um that work with very basic cipher queries in order to get the information out. So once you have this information in the graph, writing the queries to actually pull it out becomes quite simple. Um if you can express the idea that you want to uh do then it's pretty easy to write cipher or the equivalent language in whatever graph you're using um to actually pull that data out.

这表明你已经为这项工作创建了合适的 schema。现在，如果你运行这个智能体，它实际上会花几个回合调用工具并尝试弄清楚情况。然后它会尝试回答我们的问题。现在，我们可以问：“如果我更改 posting list，会有哪些内容出问题？请告诉我每个依赖符号及其所在的文件和行号。”接着它会根据影响范围分析，准确告诉我们会有哪些内容出问题。我们还可以在这里看到它调用了哪些工具：它先进行了搜索，在调用影响范围分析之前做了几次搜索，之后又调用了一次影响范围分析。由此可见，如果智能体足够擅长使用工具，就可以利用这些新工具来逐步加深对代码库的理解。And that's a sign that you have created the right schema for the job. So now if you go ahead and run this agent, this agent will actually spend a few turns um calling the tools and trying to figure things out. Uh and then it will go ahead and uh try to answer our question. So for now if we can say what breaks if I change posting list tell me each dependent symbol with its file line and then it will tell me exactly uh what will break um based on the blast radius finding and we can see here what tools it called. It started with search did a couple of searches before calling the blast radius uh and then doing another blast radius call. So we can see that if you have an agent that is uh skilled enough in tool usage um it can use these new tools to uh build out its understanding uh of the codebase.

现在这部分是，呃，更多的……Now this part is a uh more of

编写其他图工具Write additional graph tools

一个交互式区域，你可以在这里构建任何你想要的工具。嗯，它的基本工作方式是，我们可以编写一个能够执行任意 Cypher 查询的工具。嗯，这里我们正在编写一段 Cypher 查询，让它查找这个文件路径中的所有符号，嗯，然后把它们列出来。这个工具就会照做：找出这个文件路径中的所有符号，并把它们列出来。嗯，是的。所以，如果我想编写一个新工具，其实甚至不一定要懂 Cypher，对吧？嗯，我可以让智能体使用一个包含 schema 的系统提示，对吧？它会告诉智能体，好的，这些是不同的节点，这些是不同的边。然后我可以用自然语言来写，嗯，我可以提出这样的请求：我想查看每个实现了某个 trait 的 struct，以及它实现的 trait，对吧？an interactive section where you can build whatever tool that you would like. Um, how this basically works is we can write a tool that basically executes arbitrary arbitrary cipher. Um, so here we're wr writing some cipher that says find me all of the um symbols within this file path um and then just lay them out. So this tool will do just that. It will find all of the symbols in this file path and lay them out for me. Um, yes. So then if I wanted to write a new tool, I wouldn't even necessarily have to know cipher, right? Um I could give the agent access to a system prompt that has a schema in it, right? That tells the agent, okay, these are the different nodes, these are the different edges. And I can write um I can write my um my requests in natural language like I want to see uh every strct that implements a trait along with the trait that it implements, right?

如果我运行，嗯，如果我运行那个，呃，它会花一秒钟，因为这个智能体会实际尝试弄清楚。它确实做到了。它实际上生成了这个 Cypher 查询，它可以……And if I run um if I run that uh it'll take a second because the agent's going to actually try to figure it out. And it does. It actually generates this cipher query that it can

使用 Fenic 生成语义摘要Semantic summaries with Fenic

运行之后，它就可以继续从图中提取这些信息。今天我们还讲不到第二级，不过可以稍微体验一下第二级。到目前为止，所有内容都是完全确定性的：你会得到结构，也会得到解析结果。我刚才提到的第三级，指的是真正的语义含义。你实际上可以让 LLM 总结这些符号，并为它们生成嵌入向量。Typed DF 实际上有一个名为 fenic 的开源库，可以实现这一点。它是一个 DataFrame 库，用于将传统的 DataFrame 操作与 LLM 结合起来。这里我们快速配置一下 fenic，让它连接到 OpenRouter。我们会导入一些信息，然后尝试对这个特定的 Rust 项目进行分类：提取它的类型、限定名称和文档字符串，接着询问：“用一句简洁的话说明这个 Rust 项目的用途是什么？”当我们用 fenic 处理这个语料库，也就是这些行时，就会得到 LLM 生成的摘要结果。run and it can go ahead and pull that information um out of the uh out of the graph. Um we're not going to get to uh rung two today, but we can get a little taste of rung two. So so far everything has been very deterministic. Um you get your structure and you get your resolution. Um that third rung I was talking about was the actual semantic meaning. um you can actually have an LLM summarize these symbols and uh embed them. And uh typed defaf actually has an open source library for this called fenic. It's a dataf frame library for u marrying traditional dataf frame operations and LLMs. Um so here we're just doing a quick configuration of fenick where we say we're going to have it talk to open router. Um we're going to pull in some information. We're going to say uh we're going to try to classify this specific rust where we're going to pull in the kind and the qualified name and the dock string and then we're going to ask in one concise sentence what is this rust item for and so when we run fenic over this over this corpus over these rows we get back um the summarized outputs from the LLM.

所以从长远来看，对吧，我们的想法是把这些信息整合到你的图谱中，这样代理也能使用这些信息。这样做的好处是，在生成这些信息时可以进行审计。如果你看到这里并说：“嘿，关于这个函数功能的解释是错的，我们可以直接把它修正过来。”如果代理临时推断这个函数的功能，而且推断错了，你可能要等很久之后，等它写出无法运行的代码，才会发现它错了。嗯，太棒了。So in the long run, right, the idea would be to bake this information up into your graph so that this information is also available um for the agent. And the nice thing about this is that this is auditable when you generate it. So if you're looking at this and you say, "Hey, this explanation of what this function does is wrong, we can go ahead and actually get that fixed." If the agent on the fly figures out what that function does and it's wrong, you're not going to know that it's wrong until much later down the line when it writes code that doesn't work. Um, awesome.

维护和语言限制Maintenance and language limitations

是的，当然。嗯，那是第一个笔记本。我们还有一点内容要讲，不过我很乐意回答大家关于刚才内容的任何问题，我马上把它再放出来。好，我看到后面有两位。我们会拿着麦克风过去。Yeah, sure. Um, that was uh notebook one. We have a little bit more stuff to talk about, but I'm happy to take any questions about the content that was there and I'll put it back up real quick. Yeah, I see two in the back. will come back with a with a microphone

来宾：测试。Guest: testing.

主持人：是的。这个图会在运行时重新建立吗？也就是说，每当你在构建这些东西、呃，遍历它们时？Host: Yeah. Is the is the graph reestablished at runtime when whenever you're uh building these a uh traversing them?

嘉宾：定义一下运行时。运行时是指，呃，当 PR 或 CI/CD 通过时，还是……Guest: Define runtime. runtime as in uh when the PR or CI/CD comes through or

主持人：是的，我的意思是，理想情况下，虽然我们不是靠开发这个产品谋生的，但它应该以规范形式缓存并保存下来，比如说，作为基于 main 的黄金版本，对吧？然后，当你调整和修改本地内容时，可能需要维护一份本地副本，理想情况下这份副本可以增量刷新，这样你就不必一开始就把所有内容都处理完。Host: yeah in an ideal world I mean we don't build this product for a living but like in an ideal world that would be uh cached and saved in the a canonical form as like say the gold version based on main for example right um and then as you tweak stuff and modify things locally you need to probably maintain a local copy of that that would ideally be incrementally refreshed so you wouldn't have to do it all up front

主持人：是的，我……我的问题也类似。你觉得，呃，保持这些结构更新所需的成本，和性能及保真度方面的提升相比，该怎么权衡？嗯，至少对我来说，你知道，我是在用 AI 做大规模改动。所以……Host: yeah I My question is similar. How do you uh think about the cost of keeping these structures up to date versus you know whatever the increase in performance and fidelity um because at least for me you know I'm using AI to make uh large changes. So

嘉宾：嗯，所以我很难一下子判断，你知道，如果必须维护这些结构，是否真的会更好。Guest: um so it's hard for me to know um offhand you know whether it would really be better if you have to maintain these structures.

主持人：是的，完全同意。我觉得关键在于，[嗤笑] 嗯，很多这类问题都可以拆解，并采用增量处理，因为你做的很多事情，其实不一定需要重新索引已经索引过的内容——只要它们没有变化，对吧？而且，当你对整个代码库进行大范围修改时，确实会改动很多文件，但也有很多文件并没有改动，对吧？所以，如果我们能聪明地进行增量重建索引，确保只跟踪发生变化的内容，只对那部分重新索引，那么就目前的情况来看，这样做确实值得。尤其是随着需要保持索引的上下文规模不断增大，这种做法会变得更有用，因为每次都实时处理的成本只会越来越高。Host: Yeah, absolutely. I think I think the key to that is [snorts] um a lot of these problems can be boiled down and incrementalized because a lot of the things that you're doing is you you don't necessarily have to reindex the things that you've already indexed if they're not changing, right? And when you're making broad sweeping changes across your codebase, like you are changing a lot of files, but there also are a lot of files you're not changing, right? So if we can be smart about incremental reindexing, making sure we're only keeping track of what's changed, only reindexing that part, then what we've seen so far is that the trade-off is in fact worth it. Especially as the context size of what you need to keep indexed grows, it actually becomes more useful because the cost of doing it live every time just becomes even more expensive.

嗯，非常感谢。这真的特别有帮助。嗯，我想知道它在实际代码库中是怎么工作的，尤其是那种比较复杂的语言。比如说，有动态分派，或者并非所有类型在编译时（或者说，在分析代码时）都能确定。你们试过这种情况吗？有什么限制吗？有什么建议吗？Um, thank you very much. That was super useful. Um, I I'm I'm wondering how it works on kind of like a real code base in a in a messy language where uh for example you have dynamic dispatch or like not not all types are available at you know at compile time or before sorry when you just analyze code. So have you tried like what are the limitations any any any tips there?

嘉宾：是的。嗯，我这里确实还有第三个笔记本，专门讨论这个问题。嗯，基本上讲的是，如何在 Python 这样的语言中实现这件事——它不像 Rust 那样，能轻松地访问类型图。嗯，我们可以使用其他工具，比如 Griffe 和 some type sitter。Types 仍然可以为你提供……而 Griffe 可以对类型等其他一些内容进行静态解析。你也可以用像 tai 这样的工具来进行类型检查，从而获得更多洞见。是的，换成不同的语言，难度确实会呈指数级上升。嗯，我稍后会稍微谈谈这一点。嗯，不过，回答你的问题：是的，我考虑过这个问题。这是一个难得多的问题，所以我今天选择解决一个稍微简单一些的问题。Guest: Yes. Um we I do have a third notebook here which is talking about that in particular. Um which is basically how you do this thing on a language like Python which doesn't have the same uh uh easy access to the types as graph as as rust. Um we use we can use other tooling. We can use tools like griff and same type sitter. Uh types will still give you a and griff will give you static resolution of some of the other stuff like the types. You can use something like tai uh to do type checking as well um to gain more index there. Yes, it does become exponentially harder with different languages. Um, so the and I will talk about that a little bit later on. Um, but yes, the answer to your question is yes, I did think about it. It is a much harder problem which is why I chose to solve a slightly easier problem today.

是的。我觉得你不应该把这个东西做出来，先说清楚，这只是一个用于教学的实例。嗯，人们经常会构建这类代码图谱之类的东西，我不会为了做代码分析就从头开始构建一个。呃，我不知道这是否回答了你的问题。我觉得构建这类东西是值得的，但不确定每个人都值得自己构建一个。我想，这个领域里已经有不少人在做这类工作了，你知道，就是只在代码层面进行分析。所以我会说，如果是为了把它做成产品，一个人去构建它可能不太值得；但如果是为了学习如何构建智能体、工具链和上下文图谱，那就很值得。Yeah. I don't think you should be building this to be clear. This this is this is a this is a worked example for educational purposes. um people build these kind of coding uh graph things like on the reg on the regular and I wouldn't try to build one from scratch um just for doing code analysis. Uh I don't know if that answers your your question and I think they are worth it to build. I don't know if it's worth it for everyone to build their own. I think there is probably a and there are a bunch of folks in this game already that are are doing this sort of you know analysis just on the code layer. Um, so I I would say it's probably not worth it for a single person to build if for the purpose of being a product. It is probably worth it for someone to build for the purpose of learning about how to build an agent and how to build tooling and how to build context graphs.

嗯，我觉得这个问题的规模大概就是它更有用的地方。这样回答了你的问题吗？Um, I think the scale of this problem is like where it that's where it's more useful. That answer your question?

数据工程评估与范围界定Data engineering evaluations and scope

主持人：太棒了。Host: Awesome.

主持人：那么，呃，我的问题是关于你之前提到的数据上下文层。呃，我猜它对开发者会有帮助。也就是说，你开发了这个上下文层，它能帮助生成更好的 SQL 程序之类的。嗯，我有两个问题。第一，你有没有发现它在运行时也有帮助？呃，比如在推理时？这是第一个问题。第二，嗯，你们有没有基准测试结果可以展示？Host: So uh my question is about the data context layer uh that you talked about earlier. Uh so uh I'm assuming that that's useful in a for a like a developer. So you develop this this context layer and it helps uh produce uh better SQL procedures whatnot. Um so uh two questions number one um does it have you seen if it helps at runtime as well? Uh like inference time uh that's number one. And number two, um do you have any benchmark results to demonstrate?

嘉宾：是的，当然。所以，Yonyi 在他的演讲中谈到的内容，我们确实有相应的基准测试结果。我们做的是，尝试寻找具有代表性的数据架构——这其实挺难的，因为你可以构造一个很简单的示例，然后说：“哦，我的智能体在我的 30 个模型语料库上的测试中拿了 100%。”这就……好吧，也没什么了不起的。所以，我们试着寻找规模更大的开源数据架构示例。最后，我们决定采用一个案例，不知道大家有没有听说过 Mattermost。他们是一家即时通讯和聊天软件公司。他们实际上把整个 DBT 项目都开源了，里面有大约 300 到 350 个模型。因此，我们能够把它拉下来，在其中模拟数据，然后针对里面指定的场景进行评估。Guest: Yeah, absolutely. So what Yonyi was talking about in his talk, um we do have benchmark results for that. Um what we do is we've uh we've tried to find a representative data architectures which is kind of hard because you can build an easy tour example like in your in your in your and say oh my agent aces 100% of my tests on my 30 model corpus and it's like okay that's not very impressive. Um, so what we tried to do is find uh open source examples of data architecture that was larger. Um, so we ended up settling on using uh I don't know if anyone's heard of Mattermost. They're a instant response and chat company. Um, they actually open source their entire DBT project with like several 300 350 models in it. Um, so we're actually able to pull that down, simulate data in there, and then run evaluations against specified scenarios inside of that.

嗯，让它执行一项特定任务，在里面故意弄坏一些东西，然后再让它修复。让它构建一组新的任务，让它去构建新东西。通过构建我们的评估层，并逐层评估它做得有多正确、多完整、多高效，我们就能看出这一点，对吧？我们能够看到，在提供适当上下文的情况下，我们的智能体整体表现更好；而且在某些特定类别中，它们完成这些相同任务的效果也优于单独运行 Claude Code。嗯。Um, asking it to do a specific task, break something in there, and then ask it to fix it. ask it uh build out a set of you know new tasks for it to build new things. And we're able to see that uh by building out our evaluation layer and layering out you know evaluations of how correct it is, how complete it is, um how efficient it is, right? We are able to see that uh with the appropriate context our agents are able to do a better job uh overall and in certain particular categories uh to uh like accomplish those same tasks better than just cloud code running alone. Um

主持人：那之前的问题呢，或者说关于推理时间的另一个问题呢？Host: and what about the earlier question or the other question about inference time?

嘉宾：所以，呃，它确实会……你说的是智能体实际运行的时候。是的，智能体实际运行的时候。Guest: So uh it does so you're talking about like when the agent is actually running. Yes, when the agent is actually running

主持人：这正是我们发现它最有用的时候，因为智能体运行时会查询我们构建的所有工具，而真正的价值就在这里体现出来，对吧？如果智能体能在运行时查询所有这些内容，就不必一次性重新推导所有内容了，对吧？Host: that that that's when we find the most use out of it because when the agent is running, it's going to be querying all of the tools that we built and that's where the real value is coming into play, right? If the agent's able to query all this stuff at runtime, it doesn't have to rederive it all at one time, right?

主持人：那么，嗯，你有没有用过 Bird、Spider 之类的 NL2SQL 基准测试来试试这个？Host: So, um, so then have you tried this with like NL2SQL benchmarks like Bird or Spider or one of those?

嘉宾：是的。我的意思是，没错。我们确实在其中一些基准测试上试过。不过我觉得难点在于，产品的目标未必是编写分析型 SQL。我们想为大家打造的东西，更多是面向数据工程这一侧的。所以我觉得你知道，说“哦，我们要做文本转 SQL”，然后着手打造这样的产品，是很常见的想法。我觉得大家都在试图做这个，对吧？但我认为更重要的是，我们想打造能让你真正信任文本转 SQL 的工具，而不一定是要替你编写更好的分析查询。我们希望确保，当你实际投入日常工作、维护数据架构时，我们能帮得上忙。Guest: Yeah. So, I mean, yes. And no like yes we have tried it on some of those benchmarks. I think the the tricky bit is that like the goal of the product isn't necessarily to write analytical SQL. The goal of what we're trying to build for people is more on the data engineering side. So it's I think you know it's it's sort of a very common thing to say oh we're going to build text to SQL and we're going to build that. I think everyone's trying to build that, right? But it's I think it's more important to say we want to build the tools that are going to make it possible for you to actually trust text to SQL, not necessarily that we're going to be able to, you know, come in and write better analytics queries for you. Like we want to make sure that when you are actually going in and doing your day-to-day work of maintaining your data architecture that we're going to be able to help you with that.

这样回答了你的问题吗？好吧，我回到这里来——如果我能找到鼠标的话。Did that answer your question? All righty. So, popping back into if I can find my mouse.

该智能体仍然选择 grepThe agent still chooses grep

回到幻灯片。我们刚才最后看到的是，我们实际上可以把这些不同的倒排列表区分开来。因为我们能够查询新的图上下文。我们还能一步就找出给定符号的所有依赖项，不需要再次抓取。查询的工作方式是：它会匹配完全限定名，然后找出所有依赖项，就这样。各位，工作坊很棒，我们结束了——才怪。问题是，这确实很酷，也确实有效。这个工具真的很棒，对吧？它一次调用就能完成通常肯定需要多个图才能完成的事情。我觉得就算抓取做得再完美，也不可能只抓取一次就做到，对吧？但问题是，你实际上得把它交给一个智能体。Popping back into the slides. So what we saw there at the end, right, is that when we uh we can actually go ahead and tell these different posting lists apart. Um because we're able to uh query our new graph context. Um and we're able to actually figure out all the dependencies of a given symbol with one hop uh without any regrabing. Um and so how the query works you know it matches on the qualifi the fully qualified name and then it goes and finds all the dependencies and that's it guys great workshop we're done for the No. So the thing is like this is cool. This is real. This tool is really great right? This tool can do in one call what it would certainly take multiple graphs to do. I don't think you could accomplish this in one grab even if you did the perfect grab, right? But uh the problem is the agent you have to actually go and give it to an agent.

所以，当你把这个工具交给这个智能体，又把这个工具交给它，并把它和智能体已有的所有工具放在一起时——你知道，我们有一个非常酷的工具，也见过它做出各种了不起的事——它还是照样跑了 GP。嗯，如果你让它使用图工具和 GP，它还是会跑 GP。我们给了它很棒的工具，还给了它一个预先计算好的、可遍历的代码库模型，可它还是照样跑了 grip。它就是用 glob 匹配文件、读取文件，然后运行 grip；有时还会试着查看图，然后被搞糊涂，接着又回去折腾图里的那些东西。这就是没人会真正告诉你的地方。大家都在说：“哦，他做了这些很酷、很炫的工具，它们能做出各种了不起的事。”但智能体得真的去用它们，对吧？你得做提示工程，还得做其他那些苦活儿，以及评估工作。So when you give this agent this tool and you give it this agent this tool and you give it alongside all the existing tools the agent already has and you know we have this really cool tool, we've seen it do this amazing stuff and it ran GP anyway. Um, if you give it access to the graph tools and GP, it's going to run GP anyway. And we gave it amazing tools, a precomputed traversible model of the codebase, and it just ran grip anyway. It just globbed and read files and and ran grip and it would sometimes, you know, poke the graph and get confused and then get back to, you know, waiting through the graph stuff. And this is where no one really tells you. Everyone's like, "Oh, he built these cool sexy tools and they do this amazing stuff." But the agent has to actually use them, right? you have to do the prompt engineering and the rest of the hard work and the evaluations.

剩下的才是难点，对吧？你可以搭建那个笔记本，也可以在一个下午里构建出你那套很棒的图上下文。难的是让智能体真正用上它。嗯，比如说，是的。嗯，那它为什么会这么做呢？这是因为后训练。模型经过训练，被设计成会调用这些工具，对吧？在它们的训练过程中，有很多情况下，正确做法就是用 grep 搜索并阅读，对吧？而你未必能仅仅靠做出更好的工具来扭转这一点。你可以做出更好的工具，但智能体的肌肉记忆不会让它用这个工具。这跟人类非常相似，对吧？你可以给人们造出更好的工具，如果你硬要他们使用，他们不会高兴的。这话题可能有点应景。That's the rest that's the hard part, right? You can build that notebook. You can build the your amazing graph context in an afternoon. The hard part is actually making the agent use it. Um like the uh yes. Um and why does it do this? Well, it's because of the posttraining. The models are trained and designed to call these tools, right? There are so many circumstances in their training where the answer is to GP and read, right? And you're not going to be able to necessarily fight that um just by making a better tool. You can make a better tool, but the agent's muscle memory isn't going to let it use it. It's very similar to humans, right? You can build humans a better tool and if you try to force them to use it, you're going to get they're going to not be happy about it. This may be a little bit topical.

我不知道。嗯，不过，重要的是，你不一定是在比拼你的工具是否比智能体工具集里的所有工具都更好，对吧？你是在奋力争夺……I don't know. Um, but so the the important thing is that like you're not necessarily competing on the merits of your tool being better than all the tools inside of the toolb tool set of the agent, right? You're trying to fight

围绕智能体的习惯设计工具Design tools around the agent's habits

对抗它自身的本能。那么，具体该怎么做呢？对吧？因为我觉得，现在已经没有疑问了：没错，如果它能查询图谱，就会更高效、效果也更好。我们之后会证明这一点，但具体要怎么让它这么做呢？答案不是去硬碰硬地对抗模型已有的经验，而是顺着它的经验来，对吧？我们要让图谱更像模型在使用 grep，对吧？你可以把一个工具叫作“查询搜索”“查询图谱”之类的名字，但你只要把它叫作“搜索”就行了，对吧？这样，智能体就会以某种方式使用它。我发现，如果你把工具叫作“搜索”，即使没有经过训练，智能体基本上总会用多个关键词来搜索。所以，你得跟模型各让一步，对吧？against its own reflexes. So how do you actually do that, right? Because I don't think there's a question anymore that yes, if it could query the graph, it would be more efficient and better. We'll prove that later, but how do we actually get it to do that? And the answer is to not really fight the the prior experience of the model, but to kind of lean into it, right? We we make the graph feel more like the model is using grab, right? You know, you you you take this you take a a tool that maybe called query search or query graph or query whatever and you just call it search, right? The agent will use it in a certain kind of way. What I found is that if you call a tool search, the agent will basically always do multi keyword search into it with no training. So you have to you have to kind of meet the model halfway, right?

你拿这个模型，把它命名为 search，并考虑到它会如何查询，以便做出相应调整；然后你只需要从图谱中获取结果，再以与 ripgrep 相同的格式返回结果，对吧？你使用与 ripgrep 相同的工具结构和响应，并把它返回给模型，对吧？因为模型就是基于这些内容训练的。它知道该如何处理这些东西，你不可能只靠提示词就让它彻底改变自己，对吧？最后一点是，你需要确保你的工具每次都能在第一次调用时发挥作用，因为有时候，智能体沿着某条路径探索，结果发现走进了死胡同，或者它认为那是条死胡同。它调用了 search，却没有得到任何结果。那么，它有时就会觉得 search 坏了。You take the model, you name it search, you account for it to adjust for how the model is going to query it and then you just, you know, you get back the results from the graph and you return the results from the graph in the same form as rip grab, right? You take the same tool structure and response as rip grip and you return it back to the model, right? Because that's what the model's been trained on. It knows how to handle this stuff and you're not going to get it to completely change its own self um just by prompting, right? And the last thing is that you need to have uh your tools pay off in the first call uh every time because when something when the agent goes down a path and it finds a a dead end or something that it thinks is a dead end. It called search didn't get any results. Well, sometimes it's going to think search is broken.

如果我没有得到任何结果，而且我觉得本来应该有结果，那可能是搜索出了问题。我应该试试别的工具，对吧？它不会回头再用那个工具。如果它真的这么做了，可能也会尝试用别的方法。你没法确切预测这一点，对吧？所以，你得尽力确保你的工具能返回非常有用的内容，嗯，对吧。If I'm getting no results and like I there should be results, maybe search is broken. I should try another tool, right? It's not going to go back and use that tool again. And if it does, maybe it tries to use a different way. you can't really necessarily predict that, right? So, you have to do your best to make sure that your tools are returning something that's super useful uh right

笔记本二：公平地比较智能体Notebook two: compare agents fairly

离开。否则，他们就会回到 GP 那里，而且再也不会回来。所以，第二个 notebook 比较短。嗯，它是一个更聚焦的示例，内容都是关于评估的，对吧？你究竟该如何证明这个智能体确实在做你认为它应该做的事？又该如何证明你的工具优于可能已经存在的现有工具？所以，这些内容都已经预先缓存好了。我已经运行过了。如果你确实想在本地运行这个 notebook，完全可以把 sample size 设为大于零，然后实时运行评估。不过这会花很长时间。可别说我没提醒你。嗯，我昨晚用邮件数据集中的 30 个问题，进行了 5 次重试并运行了整整一夜。所以，这个 notebook 里的结果就是这么来的。away. Otherwise, they're just going to go back to GP and they are uh never going to come back. So, the second notebook is uh shorter. Um it's a little bit of a more focused example and this is all about evals, right? How do you actually show that the agent is doing what you think it is supposed to be doing, right? And how do you actually um prove that your tool is better than the existing tooling that may already exist, right? So um all of this stuff is already pre-cached. I did this run. If you do want to run the notebook locally, you can absolutely set sample size to um something greater than zero and run the eval live. It will take a long time. Don't say I didn't warn you. Um but um I did run this overnight with uh five retries and uh 30 questions in my email set. So that's where these results are coming from for this uh for this notebook.

所以，我们完成这项设置后，首先要做的就是设置测试用例，对吧？我们会创建一个能够使用新图工具的智能体，让它访问同一个模型和我们的图工具。还会创建一个对照智能体，让它访问同一个模型，以及一组由 Dantic 维护和付费支持的文件系统探查工具。这就是我们的两个测试智能体。接着，我们会建立一个问题语料库。每个问题都对应一项真实任务。我们会想一些问题，比如：“如果我修改 posting list，哪些公共类型会出问题？”“有两个 posting list，我该用哪个？”之类的。So the first thing that we do um after we do this setup is um we basically uh set up our our test cases, right? So we're going to have an agent with um our new graph tools. So we're going to give the agent access to the same model and our graph tools. We're going to have our control agent that's going to have access to the same model and a set of file system probing tools that's maintained by paid by dantic. Um so uh that's going to be our two test agents and then we're going to build up a corpus of questions. Um each question is a real task. We're going to figure out okay questions like what public types break if I change posting list? There are two posting lists. What do I use? You know this kinds of thing.

嗯，我们会尝试判断：好，这个问题应该非常适合图；这个问题可能非常适合 GP；还有一些对照问题可能两者都适用，对吧？这里的 gold 数据实际上是从图中提取挖掘数据，再回头与源代码交叉核对，所以我们并没有用图的输出污染 gold 的上下文。我们确实需要交叉核对，确保 gold 没有作弊。现在我们可以加载 gold，基本上开始搭建实际的评估结构了。um where we try to you know say okay this this should be a question that should be really good for the graph this is a question that's probably great for for for GP um you know there's a control questions that are probably be good for both right so the gold data for this is actually we do pull the mind data from the graph we do cross reference it back to the source code so it's not that we're polluting the context of the gold with the output of the graph we do have to cross reference it to make sure that the gold isn't cheating um so now we can go ahead and load up our uh our gold and we can basically start trying to set up the actual evaluation structure itself.

确定性评分与证据Deterministic scoring and evidence

嗯，所以这里说明了什么基本上算是好的评测问题，什么又是糟糕的评测问题。理想情况下，你希望评测问题能够被确定性地评分。我认为，在很多使用场景中，让 LM 充当评判者确实非常棒，但这种做法往往被过度使用了；实际上，你应该尽可能努力地让评分具有确定性。这样可以减少多次运行之间的方差，降低运行成本，长远来看也会让你更满意。嗯，所以你要设计的问题，其答案能够从……中提取出来；你能够为这些问题构造出可用的标准答案，并据此核对答案。你不应该只是让一个智能体给出自己的看法，然后再让另一个智能体用自己的看法来反驳它。Um and so this is what lays out what's basically a good eval question versus a poor eval question. Um you want an eval question that can be in an ideal world your eval should be able to be deterministically scored. I think LM is judge is really fantastic for a lot of use cases but it tends to be overused when you really should be trying as hard as possible to have your scoring be as deterministic as possible. It's going to reduce variance in your runs and it's going to reduce cost in your runs and it's going to make you happier in the long term. Um, so you want to create questions where the answers are pullable from the you you're able to structure an answer to them that's gold that works. You're able to check the answer against the gold. Um, you're not just asking the agent to provide its opinion and then asking another agent to counterman that opinion with its opinion.

这些问题通常都不太适合写成电子邮件。嗯，好吧。这是一些草稿文本，你不用太在意。这基本上就是在构建我刚才说的那些测试用例。这是确定性部分，我们会检查它是否引用了所有预期的证据。嗯，我们基本上就是创建一个示例列表，再创建一个它应该引用的内容列表，然后检查它是否都引用了。至于评估本身，我们使用 Fidantic Evals 中的配置，因为它设置起来非常简单。嗯，具体做法是先构建两个不同的智能体，再构建智能体运行器本身，然后构建一个评估器，对吧？它会根据我们给出的任务来评估智能体。Those questions don't really tend to ever work out as emails. Um, all right. So, this is this is some craft text. You don't need to worry about this too much. This is just basically building out the um the the test cases that I was saying. This is the deterministic part where we check to see if it cites all the evidence that we expect it to site. Um, we're basically just creating a list of examples, creating a list of uh uh things it should be citing, and then checking to see if it's citing all of them. For the eval themselves, we're using a configuration in Fidantic Evals just because it's really easy to set up. Um, how this works is we build out uh our two different agents and then we build out um the agent runner itself and then we build out an evaluator, right? So, this is what actually will evaluate the agent based on um the given tasks that we have in front of it.

我们基本上会将智能体的回答与我们对每项具体任务的预期进行比较，对吧？这样一来，我们就能比较 prep 智能体和 graph 智能体的响应和结果，对吧？这就是我们能够确定两件事的方式：一是我们的技能实际有多大用处；二是随着时间推移，我们可以改进什么，或者怎样做得更好。那么，继续运行吧。We'll basically compare what the agent has said to uh what we expect it to for each given task, right? And that way we're able to compare the responses and the results between our prep agent and our graph agent. Right? So that's how we're able to establish um one you know what is the actual utility of our skills and two um how do we uh what what can we improve or or what can we do better as time goes on. So um go ahead and run.

一次正确运行与反复保持一致性One correct run versus repeated consistency

对。所以我就不再跑一遍评估了，因为那大概要花 30 分钟。不过，我们可以看到，第一次跑评估时，我们用的是 k 等于 1。也就是说，对于数据集中的每个任务，我们都使用 onehot graph agent 和 graph agent。这样做的结果是，graph agent 表现得不太好。你可以看到，graph agent 确实比 graph agent 稍微好一点，但表现并不稳定，也算不上特别好——这不是我们想要的，对吧？但我们要明白，目标从来就不一定是构建一个比 Grap 更准确的东西。因为我们这些日常使用 Grap 的人都知道，用我们的编码代理时，Grap 在它擅长的事情上表现非常好。它有理由使用 GP。它不会经常出错，对吧？Right. So I'm not going to go ahead and run the eval again because that would take about 30 minutes. But um we can see that we run the eval the first time and we run it with uh k equals 1. So we just say for every task in our data set onehot graph agent graph agent and we do that the graph agent doesn't do very well. You can see that the the graph agent it does edge out the the graph agent a little bit but it's not really consistent and it's not really like very it's not what we wanted, right? But what we have to understand is that the goal wasn't ever necessarily to build something that's going to be more correct than Grap. The goal because all of us that use Grap on a day-to-day basis when we use our coding agents know that it's very good at what it does. Like it uses GP for a reason. It's not going to be wrong a lot of the time, right?

但重要的不是我这一次试的时候它答对了，而是我连续试 10 次、5 次、10 次、15 次、20 次，它每次都能答对，对吧？所以我们实际上需要开始以这种方式对这些测试进行采样。现在，我们说把同一组测试在同一语料上运行五次，以计算 pass@k 指标。这样得到的答案会稍微不同，对吧？所以我们有两种不同的统计指标：pass@k 和 pass^k。pass@k 指的是，在这五次运行中，它有没有哪怕一次答对，对吧？在这种情况下，两者还是非常相似的。实际上，Graph 稍微占点优势。嗯，Graph 不会对任何私有变量进行采样，只会采样 crate 中公开的部分。因此，有些控制问题 Graph 总是会表现得更好。But the important part is not that it's right the this one time I tried it, but that when I try it 10, five, 10, 15, 20 times in a row that it gets it right every time, right? So that's how we need to actually start sampling these tests. Now when we say we run these same set of tests over the same corpus five times to get our our pass at K metrics. Now we get a slightly different um answer. Right? So we have two different statistics. Our pass at K and our pass hat K. Our pass at K says across these five runs was this ever correct? Right? In that case they're still they're very similar. Actually graph has a little bit of an edge there. Um the graph doesn't sample any of the private variables only the the private part of the crates only the public. So there's some control questions that the graph will always do better on.

嗯，不过真正有意思的是，当你改变指标，问“它每次运行时都能做好该做的事吗？”时，数字就会彻底反转。突然间，图表代理在五次运行中每次都做对事情的表现都好上一倍，比……比图表代理还好，对吧？嗯，我们可以看到，在那个正在做这些事的广告中，它做得更稳定，也更高效，对吧？纵观我们所有不同的运行，平均而言，图表方案使用的 token 数量大约少了 3.1 倍。最后一点是，好吧，我们实际上要怎么……Um but what's really interesting is when you change the metric and say is it doing the right job on every run and when you do that the numbers completely flip right now all of a sudden the graph agent is twice as good at doing the correct thing all five runs than the than the graph agent. Right? Um, and we can see that within that ad that's doing that stuff and it's doing it more consistently, it's also doing it more efficiently, right? We can see that over the course of all our different runs, on average, the graph approach took about three and a 3.1 times fewer tokens. The last part is, okay, how do we actually

衡量图表工具的采用情况Measure graph tool adoption

让智能体使用图吗？这正是评测派上用场的地方，因为你基本上得编写一个评测，说明：“嘿，检查我调用轨迹中的所有工具。我的智能体使用我希望它使用的工具的频率有多高？”然后你就能得到类似采用率的数据，对吧？这样你就能说：“哦，我写了一个提示词，里面完全没有要求使用工具的指示。工具使用率是 34%。我回去修改了提示词。现在工具使用率是 72%。”对吧？所以，评测是你真正了解智能体是否在使用你希望它使用的工具的唯一方法。嗯，这只是对这个概念的一个测试。我们有两个不同的提示词，它们都试图用同一组工具回答同一个问题。get the agent to use the graph? This is where your evals come into place because you basically have to write an eval that says, hey, examine all the tools in my trace. How often is my agent using the tools that I want it to be using? And then you can get your sort of adoption numbers, right? So that's how you can get to a place where you can say, "Oh, I wrote a prompt that didn't have any instructions in it to use the tools. I got 34% of tool usage. I went back. I changed my prompt. Now I have 72% tool usage." Right? So the eval are the only way that you're going to be able to actually understand that the agent is using the tools that you want it to use. Um, this is just a test of that exact concept. we have uh two different prompts that are both trying to answer the same question with the same tool set.

嗯，如果你稍微调整一下提示词，呃，如果你只是很天真地处理，它就会直接去抓取，呃，然后我们就会开始做些别的事情。要是你调整提示词，理想情况下，它会去实际运行搜索，并使用 get symbol 来弄清楚真正的答案。[哼笑] 所以，构建真正可靠、诚实的评估的诀窍是：从你的语料库中挖掘真实任务候选项；尽可能构建确定性的、有锚点的评分器；并先对评分器本身进行单元测试。嗯，我喜欢 Pantic Eval。外面还有很多其他评估工具，但它们的核心结构都一样：你有一个数据集、一些用例和一个评估器。嗯，确保多次运行评估，并报告 pass@k、pass@k，以及所有不同的指标。Um if you if you nudge the prompt uh if you just naive it will just go ahead and grap um and then we'll start doing some other stuff. If you nudge it the ideal world is that it will go and actually run the search and use get symbol to go and figure out the actual answer. [snorts] And so the recipe for building, you know, really solid, honest evales is to mine real task candidates out of your corpus. Uh build a deterministic as much as you can, an anchored scoreer. Uh and unit test the scorer itself first. Um I like pantic eval. There's many many other eval things out there, but the core structure of them is all going to be the same. You have a data set, you have cases, you have an evaluator. Um, make sure you run your evals many times over and reports pass at K, pass at K, uh, and all your different metrics.

还有，呃，确保加入一些你的智能体可能不太擅长处理的例子，因为你不希望它拿到 100% 的通过率。如果你的智能体在评测中能 100% 通过，那就没什么用。说明你的评测太简单了。你的智能体应该处于差点无法通过评测的状态。呃，你的评测能让你看清实际情况，看看你的智能体在尽可能困难的情境下会怎么做。所以，只给它一些非常容易、非常简单的评估，其实对谁都没什么帮助。好，在我回到幻灯片继续讲最后一部分之前，大家对这个评估笔记本有什么问题吗？呃，我只是想问一下，在你举的例子里，或者说你讨论智能体通常会用 GP 而不是 graph 的时候……And, uh, make sure that you are, um, putting in examples of things that your agent may not necessarily be good at because you don't want it to pass 100%. It's not a useful if your e if your agent's passing your email 100%. Your email is too easy. Your agent should be close to failing your emails. um like your emails are your chance to see where the metal hits the road and what your agent's going to do in as hard as possible circumstances. So it doesn't really do a good anyone any service by just uh giving it very easy and very simple evals. All right, anyone have any questions on this eval notebook before I jump back into the slides for the last part of the talk here? Uh I I was just wondering in the example when or when you were discussing how you the agent would often use GP instead of graph.

所以你们不得不把它重新设计成这样一个搜索工具。呃，你们之前有没有尝试过把图谱提炼成某种文件结构？可能会损失一些信息，但这样 GP 就能自然地在其中搜索，你们也就完全不必另行定义工具了。So you had to retool it to be this search tool. Uh was there ever any uh uh like attempt to take the graph and distill it into a file structure maybe like with some loss but a file structure that then GP would naturally search over so you didn't have to define any separate tool at all.

主持人：是的。今天不行。嗯，只是因为时间有点紧，不过我觉得这绝对是个合理的做法。我的意思是，我觉得这就是你知道的企业级 LLM wiki，里面有各种各样的东西，你光靠 Markdown 语料库就能获得很多不同的好处。但我觉得这确实没错，而且我认为你仍然能从中获得很多类似的好处。不过我不太确定，像 Markdown 里的那个 Ele wiki 之类的，对吧？我还是觉得你的问题在于，你需要真正的边，以及它们之间有据可查的来源关系，对吧？而这些连接会非常难处理，因为即便你用 Grain 这样的东西，对吧？它确实会把所有内容都整理成 Markdown，但它也会在上面构建一个图谱，方便你查看时间线、各种连接之类的，对吧？Host: Yeah. Not today. um just because uh there was a bit of a time crunch but I think that's definitely a reasonable approach. I mean I think that's what you know that's corporate llm wiki you know with all these different things where like you can get a lot of these different benefits just from the markdown corpus but like I and I think that that's that's definitely true and I think that you would still get a lot of the same like benefit out of it but I don't like in the markdown like the ele wiki that kind of stuff right I still think that that your problem is going to be like you want real like edges and provenence between them right and like those connection connections are where it's going to be really hard because even when you do something like Grain or something like that, right? It yeah, it builds all the stuff in markdown, but it does build a graph on top of it for you to see the timeline and the connections and stuff, right?

所以，我觉得从某种程度上来说，你知道，如果你的任务需要理解这些不同事物之间的关系，那么从某种程度上来说，你会想要某种图。尽管我确实认为，像那样的方法也能取得非常相似的成功。是啊。嗯，你觉得像这样的配置还缺少什么，才能让非技术人员、非数据工程师的人把它当作浏览代码库的指南，发起一个 PR？你试过这样做吗？或者说，在让大家这样做之前，你会再加些什么吗？So, I think at some level, you know, if your task is based in needing to understand the relationship between these different things, then at some level you're you will want some kind of graph. Even though I do think that you can get a very similar level of of of of success out of out of an approach like that. Yeah. Um, do you think there's anything missing from a setup like this for letting non-technical, non-data engineer folks like kick off a PR with this as like the guide through a codebase? Have you tried that or like is there a something you'd add before you empower people to do it that way?

主持人：哦，比如把它当作一种工具？Host: Oh, like for this as a as a tool?

嘉宾：是的。Guest: Yeah.

主持人：哦，是的。我觉得，嗯，这是个好问题。是的。我觉得最简单的部分，大概就是先把它作为 MCP 接入你已经在用的工具。这样你就会有一个代码图谱，作为你现有 Claude Code 的补充；然后你可以让 Claude Code 调用这个图谱并执行其他操作，再编写技能，让它能够正确地使用这些功能。我觉得这是最好的做法。嗯，不过我想，如果你的技术水平较低，或者完全不懂技术，那么目标应该是：如果你想让这个智能体在实际编程过程中使用这些东西，就把我这里提到的这些提示工程方法整理成技能，并提供给你的编程智能体。Host: Oh, yeah. I think um that's a good question. Yes. I I think the the the easiest part I mean would probably be like starting to hook it in as an MCP to the tools that you're already using. So you would have you know this code graph in addition to your your cloud code and you would ask your cloud code to call the graph and do those other things and then you would write skills in order for it to actually use that that that properly. I think that would be the the best approach there. Um, but I would think that yeah, if you're less technical or non-technical, I I would think that, you know, the the the goal would be that you would if you want to engineer the agent to be using these things in the context of actual coding, you would promote these prompt engineering things I'm talking about here into skills and give those to your coding agent.

嘉宾：是啊。这样一来，这个 PR 第一次交到工程师手上时，就已经有了一个还算不错的初稿，因为整个过程受到了更具确定性的引导。Guest: Yeah. Like that PR lands at the engineer for the first time with like a halfdeent first pass as a result of being more deterministically guided by this.

主持人：是的。对。而且我觉得确定性部分很有帮助，因为就像我们刚才讨论的，关于后面那个问题，你可以增量地只对比发生变化的内容，对吧？以确定性的方式分析，然后非常非常快地得出结论，了解正在发生什么，以及从上次以来发生了哪些变化，而不是等着智能体去读取所有内容、查看所有变化，然后再把同样的……反馈给你。Host: Yes. Yeah. And and I think that the the deterministic part is helpful because you can like like we were talking about before with for the question in the back like you can incrementally just diff over what's changed, right? Analyze that deterministically and get a really really fast conclusion back about what's what's happening and what's been changing since the last time as opposed to waiting for the agent to go and read everything and look at all the changes and and give you back that that same

调用图、目录和数据基准测试Call graphs, catalogs, and data benchmarks

差异。diff.

嘉宾：后面那儿有一个。我想……Guest: That's one back there. I think

主持人：嗯，你有没有尝试过或考虑过不同的分辨率级别？比如，我在之前的一些讨论中想到的一件事是，你们会做下一个 token 预测，然后会有我想大家提到的那个前缀，它就像……Host: um did you uh experiment or think through like different uh levels of resolution like one thing I was thinking of um in some of the earlier talks is like you do next token prediction and you will have the prefix I think people said which is just like

嘉宾：你知道，这一切都处在这样的层面上：这里有代码库中所有内容组成的完整字符串，你也能看到符号，但符号所处的粒度是一种特定的粒度。代码库中还有更高层次的粒度，这些层次已然存在。让这些内容对智能体而言可搜索、可访问，或许会带来一些收益。我想知道你有没有考虑过这一点，或者对如何着手有什么想法。Guest: you know it's all at the level of like this is the whole string of everything in the codebase here you have the symbols but like the symbols is a certain granularity, there would be higher levels of granularity that are imminent in the codebase and making those things searchable and accessible to an agent, you know, might have payoffs and I was wondering if you thought about that or if you have thoughts on how you'd approach that.

主持人：这是个非常棒的问题。我确实想过这个问题。嗯，我实际上会在演讲最后、在演讲结束时谈到这件事。嗯，这里肯定还有一些后续工作可以做。嗯，如果你真的想构建这个东西，你可以用类似 SCIP 的工具，实际追踪代码库内部的调用图。你可以看到什么在调用什么，以及它们之间是如何交互的，因为你可以追踪符号，以及它们如何在代码库中流转。但如果你想实际追踪什么在调用什么，以及它们之间是如何关联起来的，那就是你说的更高层次的问题，你需要一种稍微增强的方法，再增加一层确定性解析。嗯，你可以用……有个 Rust 的工具，或者 SEIP。Host: It's a fantastic question. I did think about this. There are uh and I will actually talk about it at the end of the at the end of the the presentation. Um there was definitely some follow-ups I would do here. Um if you were actually trying to build this um you would uh uh use something like SCIP to actually trace the call graphs inside of the codebase. You can actually see what is calling what and how that interacts because you can track um like symbols and how they move across the codebase. But if you want to actually track, you know, what is calling what and how it's all connected, that is the higher level that you're talking about and you need a you need a slightly augmented approach, another rung of of of deterministic parsing. And you can do that um with uh there's like a Rust or SEIP.

嗯，对于 TypeScript 或 Python 之类的语言，也有类似的 SEP 输出。我本来也打算把这些都做出来，但时间不够了。不过，回答你的问题：是的，我们肯定还能从给定的代码库中提取出更多结构。这也算是 Type Def 尝试做的事情：找出各种不同的方法，尽可能从现有数据中提取更多上下文。嗯，我有个小问题：你会怎么描述数据上下文层这个概念与数据目录这类东西的结合方式？比如，数据目录会帮助构建一个图谱吗？在企业环境中，同时拥有两者是不是有些重复？Um there's similar SEP outputs for like TypeScript or or Python. I was going to do all of that too, but I I ran out of time. But to answer your question, yes, there's uh there's definitely more structure we can extract from a given codebase. And that's sort of what we try to do at Type Def is find all the different ways that we can pull as much context as possible out of the data that we have. Um I have a quick question for how would you say like this concept of the data context layer um combines with like something like a data catalog like is it the data catalog would help feed a graph are they kind of redundant to both have in an enterprise

嘉宾：我不认为它们是冗余的，我觉得它们回答的是不同的问题。嗯，你的数据目录是一种对数据仓库中具体有哪些内容的固定认知，我觉得这和上下文图试图提取的信息是不同的需求，对吧？我觉得它们可以共存：你会从目录中读取信息来构建上下文图。嗯，你要做的基本上就是这样：假设我有一个 dbt 项目，里面有一堆模型，这些模型会在 Snowflake 中构建表和视图，对吧？所以我们可能会说，我们要导入这个 dbt 项目，把所有 SQL 都拉取过来，进行解析，把所有内容关联起来，然后我们会说，哦，好吧，这个模型对应着这张表。让我去查询目录，获取这张表的信息，然后也把这些信息拉进图中。Guest: I don't think they're redundant I think they answer different questions um your data catalog is a a sort of fixed understanding of exactly what is present in your data warehouse which I think is a kind of a different ask than what the context graph is trying to extract right um I think they they coexist you read information from the catalog to form the the context graph um so what you'll try to do is basically say okay I have my say my dbt project that has you know a bunch of models in it those models build out uh uh tables and views in Snowflake, right? So we may say we're going to ingest that DBT project. We're going to pull in all that SQL. We're going to parse it, link everything together, and then we're going to say, oh, okay, this model is corresponding to this table. Let me go query the catalog, get information about that table, pull it into the graph as well.

所以我们可以说，好，这个模型正在生成这张物理表，而这张物理表包含这些列，这些列又有各自的特征，而所有这些信息都来自 catalog，对吧？因此，那个数据目录、信息架构、数据画像信息，所有这些对我们所做的事情都至关重要，而且都在 catalog 里。所以我会说，它们更多是共存关系，而不是替代关系。So we can say okay this model is uh producing this physical table and this physical table has these columns and these columns have these these profiles and that information all comes from the catalog right so that catalog of information the information schema the profiling information all that stuff that's essential to what we do and that's all in the catalog so I would say they coexist more than it's a replacement

主持人：[清了清嗓子]Host: [clears throat]

嘉宾：试着把话题拉回到数据工程，以及评估这个概念。你是否也会用类似的方式来思考，比如提出一系列数据问题，然后看看使用或不使用这个工具时，这些问题会有什么表现？Guest: to try to bring this back to like data engineering and this concept of like eval. Do you think about the same way in terms of like asking a set of like data questions and then seeing how those react uh to using the tool or not using the tool?

主持人：是的。Host: Yes.

嘉宾：对。对。没错。哦，你还有更多要说的吗？抱歉。Guest: Yeah. Yeah. Precisely. Oh, so do you have more? Sorry.

主持人：抱歉。Host: Sorry.

嘉宾：哦，我不知道你是不是说完了。嗯，是的，当然。嗯，正如我刚才所说，我们确实会尽量找到尽可能真实的数据语料库，然后再寻找多种不同类型的场景，对吧？比如最基本的修复 bug 场景：我们会把一个引入 bug 的 diff 注入代码库，然后让智能体尝试修复这个 bug。接着，你就可以测试它有没有查看正确的位置、检查正确的内容；以及它修复 bug 时，有没有在上游或下游修复，而不是打个补丁了事，对吧？它确实找到了正确的模型，并修复了正确的问题。你还可以把这些测试设计得更简单或更难，对吧？比如，你可以修改提示词，说：“哦，我觉得模型 XYZ 有问题。”Guest: Oh, I didn't know if you were done. Um yes absolutely. Um we do use uh like I was saying we basically try to find like as as real a data corpus as we can and then we try to find multiple different kinds of scenarios right so you have your basic fix a bug um where the scenario is we'll inject a diff into the codebase that introduces a bug right and then you'll ask the agent to go and try and fix this bug um and then you can you can test you know that it looked at the right places that it checked the right things that when it fixed the bug, it didn't fix it upstream or fix it downstream, right? By putting a band-aid on it, right? It actually found the right model and fixed the right issue. And you can make those tests easier or harder, right? Like you can you can change the prompt language to say, "Oh, I think there's a problem with model XYZ."

或者对于更难的例子，你可以说：“我的……我的仪表盘 Y 出现了这个问题，这个数字翻倍了。我不知道为什么。”所以要强迫智能体从这个问题依赖的内容入手，沿着依赖树向上追溯，弄清楚问题真正的根因来自哪里，然后再断言它确实会着手修复这个问题，而不是放任不管。嗯，这才是真正困难的地方，而问题在于智能体总会想办法作弊。嗯，所以在你的评估框架中，必须非常谨慎地加固防范。嗯，比如说，如果你告诉智能体：“我的测试用例是，智能体会检出一个分支，然后尝试修复问题。”智能体就会毫不犹豫地作弊，使用 git 切回 main，然后试图从那里弄清楚你是如何在这个分支中引入问题的。Or you can say for the harder example, my down my dashboard Y is having this issue where this number is doubled. I don't know why. So forcing the agent to work backward from what that depends on upward through the tree to understand like where the actual root cause the issue is coming from and then asserting that it actually goes in and fixes that problem instead of leaving it be. Um so that's what's really hard and the problem is that the agents are always going to cheat. Um so in your eval harness you have to be really careful about hardening it. Um like for example your agent if you if you if you say that oh my test case is the agent will check out a branch and then it will go and try to fix the problem. The agent will very happily cheat and use git to go back to main and try to figure out how you introduce the problem in this branch from from there.

嗯，它会很轻易地找到那个写着“这是个测试”的分支名，然后就会想：“哦，这是个测试。哦，我可以在这上面作弊。”嗯，所以你得非常小心地强化你的测试框架，确保它能真正代表你希望智能体执行的任务。嗯，因为这始终会是个潜在风险。Um and it'll go and very happily find the branch name that says this is a test and be like oh this is a test. Oh I can cheat on this. Um, so you have to be very careful about hardening your your your harness to actually uh be representative of the tasks that uh you want your agent to do. Um, because that's always going to be the danger there.

主持人：对于这些评测，你发现人们大多会从头开始编写，还是……Host: For these evals, do you find people like encoding them from scratch for the most part or like

来宾：有……有一些现成的框架吗？嗯，ADE Bench 就是很不错的一个。嗯，它可能是目前这类数据工程基准测试的行业标准。他们采用了非常类似的方法：通过 diff 在代码仓库中构造问题，然后让智能体读取数据仓库和代码中的数据，并尝试修复问题。嗯，所以这类框架确实已经存在。大家正在把它们搭建起来，我觉得这正变得越来越普遍。我认为更难的部分始终是围绕它搭建配套环境：对于代码，你只需要有代码库；但你还需要一个正在运行的数据库，需要确保其中填充了正确的、有问题的数据，而且这些数据在内部保持一致。这样你就不会让智能体一直兜圈子，试图弄明白：“哦，哦，为什么这些数字是 2024 年的？”Guest: there are there are some existing frameworks for this? Um, ADE bench is is a great one. Um, it's probably the current industry standard for this kind of like data engineering benchmarks. They have a very similar approach um for uh uh creating problems in repos using diffs and then having the agent go and read data from the warehouse and read data from the code and try to fix the problem. Um so those do exist. People are wiring them up and I think it's becoming more and more of a thing. I think the harder part is always the scaffolding around it because for code you just have your codebase but you need a running database. You need it to be you know populated with the right buggy data. you need that data to be internally consistent. So you don't want the agent to be spinning around trying to figure out, oh, oh, why are the numbers from 2024?

我们在样本集中只生成了 2024 年的数据。所以，嗯，我觉得你经常会碰到这类问题。嗯，这也是为什么人们试图在 AD 中将其标准化。我觉得难点在于，建立这样一个其他人也能使用的基准，而且它确实要足够大……It's we only generated 2024 data in our sample set. So, um, I think you run into those kind of issues a lot. Um, which is why people are trying to standardize it in AD. I think the hard part is building out a benchmark like that that other people can use that actually has a big enough

阅读追踪记录，并质疑记分牌Read traces and challenge the scoreboard

数据集。这将是棘手的部分，而且目前还没有人做到这一点。嗯，至少在我看来，还没有做到让我满意。不过，是的，你知道，我们回头看一下，这只是对刚才看到的相同结果的总结。嗯，我们看到，随着能力的提升，随着我们让智能体更多地使用工具，智能体回答同一组问题的能力会更加稳定，也能更高效地回答这些问题。嗯，这也说明了为什么多次运行这些评估非常重要。关键不在于单次准确率，而在于确定性。真正重要的是，你如何确保智能体在使用新工具。你确实应该确保提高工具的采用率，并衡量工具的使用情况，因为这才是帮助你完成最后一段路的关键，对吧？data set. That's going to be the tricky part and that's not something that has been done yet. Uh, at least to me. to my satisfaction. Um but yeah, the uh you know, we get back the the this is just a summarization of the same results we just saw. Um, we see that as we uh increase the amount of ability, as we tell, as we get the agent to use the tools more, um, the agent is able to be more consistent in its ability to answer the same set of questions and is able to answer those same set of questions um, more efficiently. Um, and an example of why it's quite important to uh, run these eval multiple times. It's not about your per shot accuracy, it's about your determinism. Um, and the number that actually matters was how do you actually make sure that your agent is using the new tooling. And so you really want to make sure that you're getting your adoption numbers up and that you're measuring your adoption of the tools because that's really what's going to get you um the right the last part of the way, right?

你知道，一款优雅的工具，如果只有三分之一的时间会被拿来使用，那它就只发挥了三分之一的作用。嗯，你能从更好的工具中获得的每一项能力提升，都受这个数字制约。嗯，所以这就是我们为什么要构建这个评测：不是为了证明这个工具有多好，而是为了证明它没有被闲置。整场演讲最难的部分，基本上就是承认图谱确实非常非常有效。嗯，我做了这些工作，构建了这个上下文图谱，也跑了评测，结果打成了平手。这让人有点沮丧，你会忍不住继续优化、继续构建，想从中挖掘出越来越多的价值——这其实是个陷阱，对吧？我觉得，那种一味想继续构建的本能有时会让人陷入沉没成本，对吧？就像智能体有它自己的反射行为，而我也有自己的本能，觉得它应该怎么做。You know, an elegant tool that gets reached for a third of the time is just a third of a tool. Um, and every capability win that you can get from better tools is gated by this number. Um, so this is why we built the eval, not to prove that the tool is good, but to prove that it's not going unused. So the hard part about this whole talk was basically admitting that graph works really, really well. Um, I did all this work and I built this context graph and I ran the evals and it was a tie and it was like it was kind of frustrating and you kind of want to keep optimizing and building the thing and trying to get more and more out of it and it's kind of a trap, right? That instinct is like to build more I think can sometimes lead to a sunk cost, right? like the agent has its own reflex. I have mine what I think it should be doing.

而且我们是在努力折中，对吧，想办法推动这件事继续往前走，对吧？然后你可能会想，哦，也许只是需要更多测试、更多评估，或者更多别的什么，对吧？最后你就会产生一种错觉，觉得我只需要更严谨、做更多测试，智能体用工具的能力就会变好。但你也许真正该问的是：嘿，为什么 GP 表现这么好？而不是问：它为什么不用我的工具？答案也许是，GP 本来就已经表现得很好了，对吧？所以你也许没法靠跟它较劲来取胜。嗯，或许你得稍微顺着它一点。反正智能体无论如何都会用 GP。呃，所以你要是一直试图在工程上压过 GP，最后只会觉得很挫败。And like we're we're trying to meet in the middle, right, and actually figure out a way to move forward with this, right? And it's like, oh, you think that maybe it's just that you just need more testing or more eval or or or more something, right? And you end up with this this false feeling of I just need more rigor and more tests and then the agent will be better at using the tools. But you really maybe you need to ask, hey, why is GP working so well as opposed to saying, why isn't it using my tools? Well, maybe the answer is that GP actually does work really well already, right? So maybe you're not going to win by fighting it. Um maybe you're going to have to lean into it just a little bit. So the agent is going to GP anyway. Uh so when you keep trying to out engineer GP, um you're just going to have a kind of frustrating time.

嗯，所以如果你能尽可能顺势而为地进行设计，帮助智能体更好地发挥它们本来就想做的事，那会对你很有帮助。嗯，我现在已经做过好几次演讲：我得出了结论，讲完了，然后说：“让我再跑一组评估。”结果我不得不重做整场演讲，因为结论完全变了。嗯，问题在于，我编写了评估，搭建了整套测试框架，但框架搭错了，所以得到的答案虽然大方向上没错——它们确实说图方法胜过了 GP——但我想再深入看看。深入调查后你会发现，哦，我们做的那个工具会返回文件路径，但返回的并不是正确的路径。所以有时候它们调用搜索时，会找到错误的路径，接着就沿着错误的路径去尝试读取这些文件。Um, so if you can design as with the grain as much as possible, help the agents lean into what they want to do anyway, uh, it will help you out quite a bit. Um, I've now done multiple talks where I I I had a conclusion, finished the talk, I said, "Let me rerun one set of evals and then I had to redo the whole talk because the conclusions were completely different." Um, what had happened was I wrote my evals and I built this whole harness and I built it wrong and I got answers that were directionally correct. They said that graph beat GP, but I wanted to look into it a little deeper. And when you look into it a little deeper, you'll find that oh, that tool that we made, it returns the path to the file, but it's not like the right path. So sometimes when they call search, it would find the wrong path and it would go down a wrong path to trying to read these files.

所以你确实得进去，把所有不同的 trace 都读一遍，对吧？进入 Logfire，或者去你发送 trace 的地方，确保你真的在读它们。因为你不应该只是看一眼自己的记分板就算了；如果只看记分板，你可能会得到一些方向上看起来正确、或者对你来说很不错的结果，但它们可能掩盖了底层更严重的问题，以及架构中更大的隐患。嗯，所以解决办法绝不是单纯把数字调高一点，或者让它看起来更好看。办法是构建一个更严格的评估。呃，读 trace，找出问题所在，强化测试框架，然后重新运行。永远不要盲目相信自己的记分板。嗯，对，这些就是我想到的其他一些看法。So you actually have to go in and you read all of your different traces, right? Go into logfire, make sure or go into wherever you're you're you're you're sending your traces in, right? and make sure that you're actually reading them because you're not supposed to just if you just look at your scoreboard, you might get answers that look directionally correct or or that look great for you, but they may be burying underlying issues that are actually much larger and and a much bigger problem inside of your inside of your architecture. Um, so the fix is never just to make the number higher or nicer. It's to build a harder eval. Uh, read the traces, find the artifact, harden the harness, and then rerun. Never trust your own scoreboard. Um and yeah, these are just some of the other thoughts I had.

你知道，模型只会变得越来越好。所以，我们可以继续相信，模型的表现差异会越来越小，也会越来越能够完成我们希望它们完成的工作。因此，简单的部分可能其实是构建上下文、构建工具。真正困难的部分将是构建评估循环、完善监控和可观测性，并确保你的智能体确实在使用你希望它使用的工具完成你希望它完成的工作。嗯，所以……The you know, the the models are only getting better. Um so, you know, you can you we can continue to believe that the models are going to have less variance and more drift towards being able to uh do the work that we want them to do. So, the easy part may actually be building the context, building the tools. The hard part is really going to be building out your eval loop, building out your monitoring observability, and making sure that your agent is actually doing the work that you want it to do with the tools you want to be using. Um, and so

拓展到 Rust 之外，并提出一些小问题Extend beyond Rust and ask small questions

所有东西都在那里，代码、循环，呃，都在仓库里，谁要是想抽空自己玩玩都可以。嗯，对，这就是我之前说的。你知道，我确实耍了个小聪明，用了 Rust。嗯，Rust 编译器已经运行过一次了，对吧？这是作业。呃，我选了可以免费使用的语言，因为这堂课讲的是，你可以在一个相对简单的问题实例上，构建一个设计精良的版本。呃，一般来说，如果你能让东西在一个稍微简单些的问题上跑起来，那么之后把它扩展到更大的问题上也会容易一些。嗯，所以现在我们可以说，哇，我们已经用 Rust 做出了这个非常简单的东西。我想用 Python 也做一个。而 Python 就是更难，对吧？all the stuff is there, the the score, the loop, uh, it's all in the repo if anyone wants to play with that um on their own time. And uh, yeah, so this is what I was talking about before. You know, we did I did cheat by using Rust. Um, the Rust compiler already did run one, right? It's homework. uh picked the language that was free because the lesson was you can build a well-engineered version of something uh on a relatively easy instance of a problem. Uh and in general, if you can get things to work on a slightly easier problem, it's slightly easier to go and scale that back up to a larger issue. Um so we now can say, oh wow, we have this really easy thing that we did for Rust. I want to do for Python. And Python is just harder, right?

就像，你看这些代码行，其实也很难真正找到什么，对吧？如果你用 grep 查找 client，就会陷入五万个 client 的汪洋大海里。你可以用静态分析来加载这些东西之类的，但你知道，这会很难处理。所以这就是代码图真正发挥作用的地方，对吧？像这种代码库比较大、语言结构没那么规整的情况，代码图甚至比 grep 更有效。因为在 Rust 里，你知道，我想看看是什么实现了这个 trait。好，很简单，我只要搜索 `imple`，对吧？真的很容易。但在 Python 里，这就难得多了，对吧？所以这时候，你就确实需要构建一个更健壮的东西。like you can't really if you look at this these lines of code you're not going to get anywhere right if you GP client you're going to get stuck in a land of 50,000 clients you can use static analysis to to load this stuff and things like that but you know you're going to have like a a difficult time and so this is where a code graph really earned its keep right in a case like this where the code bases are larger and the the languages are are less well structured a code graph is actually even more effective than grap in that case Because for Rust, you know, I want to see what implements this trait. Okay, cool. I just search imple, right? Like it's really easy. But for Python, that becomes a lot harder, right? So that's where um you really would need to build something out that's more robust.

嗯，不过图的结构和工具的结构不一定非得改变，对吧？改变的是我们摄取和加载数据的方式。如果把这部分做成模块化的，那么无论你想支持多少种语言或多少个项目，都可以基于它来扩展，对吧？这就是 TypeDef 运作方式的底层结构：我们尽可能多地构建各种不同的数据摄取流程，从许多不同的地方提取数据，然后把它们整合到一起。嗯，这方面还有另一个 notebook。我现在就不讲了，因为我们已经在这里待了一会儿了。不过你们可以自行看看，想深入了解的话也可以试着学习一下。嗯，我刚才讲的最后一件事基本上就是意义层级。Um but the structure of the graph and the structure of the tools doesn't necessarily have to change, right? It's the structure of the way that we ingest and load the data. And if you make that modular, then you can build that out with for as many languages or as many projects as you want, right? So that's kind of the underlying structure for like how type def works is we try to build out as many of these different injest processes as we can to pull data from many different places and then sort of synthesize it all into one. Um there is another notebook for this. I'm not going to go through it now because we've we've all been here for a little while and um but feel free to go through it yourselves and uh try to learn more if you want. Um, last thing I was talking about was uh basically the meaning rung.

嗯，结构告诉你什么与什么相连，而含义则大致告诉你为什么。呃，代码里有注释、文档字符串以及其他所有信息。把所有这些信息都整合进来，有助于我们理解编码人员编写代码时的意图，同时也能了解当前实际存在的代码结构。所以，嗯，我们基本上会想加入这样的小型处理步骤，来概括代码中的信息，嗯，并且逐行理解代码。这样一来，搜索就不仅能回答代码的位置，还能回答代码的作用。嗯，构建这类东西时，重要的是，你所构建的是一种可审计的语义分析，它会叠加在确定性分析之上。Um, the structure is what's telling you what's connecting to what, but the meaning is telling you sort of why. Uh, so there's comments in code and dock strings and all the other information. And folding all of that information in helps us understand the intent that the coders had when they wrote the code in addition to the actual structure of the code that currently exists. So um we would basically want to add in small uh passes like this to say basically summarize the information from the code um and uh understand it at a at a rowbyrow basis so that search can now answer what the code does in addition to um its location. Um, and the important part when you're building something like that, um, you're building this sort of auditable semantic analysis that's going to go on top of your deterministic analysis.

嗯，重要的是提出许多小问题，而不是一个大问题。解决办法不是写一个特别……你知道的。所以我们用 SQL 做分析时，不会只写一个特别特别好的提示词，把 SQL 扔给模型，然后说：“嘿，呃，这是一个有 10 个字段的繁琐模型，尽可能从 SQL 中提取所有信息给我，行吗？”这样行不通。嗯，模型会产生幻觉，会编造信息，也不会尽可能多地收集信息。与其一次让模型回答关于一段代码的 10 个问题，不如针对代码中 10 个较小的片段，分别提出 10 个不同的问题，每个问题都对应一个具体问题。对吧？先用确定性的方法解析结构，然后提出许多聚焦的小问题，发挥模型的长处。Um, the important part is to ask many small questions and not one big question. The answer isn't writing, you know, one really. So when we do analysis in SQL, right, we don't just write one really really good prompt and throw your SQL at it and say, "Hey, uh, here's a pedantic model with 10 fields like give me all the information you can out of the SQL, right? It's not going to work." Um, a model is going to hallucinate. It's going to invent information. It's going to not collect as many as much information as it can. You want to basically instead of asking the model 10 things about a chunk of code at once, you ask the question, you ask the model 10 different questions about 10 different smaller chunks of the code that answer specific questions. Right? You parse the structure deterministically and then ask many small focused questions that play to the model strengths.

对吧？与其让模型查看整段 SQL，然后告诉你所有列分别是什么意思，不如先解析 SQL，然后告诉它：“嘿，这是一个列。这是它的类型。这是它的注释。”你知道的，它是度量值还是维度，还是别的什么。这样一来，你就可以让智能体进行分类和总结，而这正是它擅长的，对吧？如果你想让它更重视某件事，嗯，就给它更多针对这个单一重点问题的提示输入。不要试图塞进更多问题，而是给它更多上下文，让它能够回答那个单一问题。所以，要在我们的结构上执行小范围、有依据、可审计的语义操作。嗯，这就是我们的 fenic 库的用途，也是我们在 typed 中使用它的方式。Right? Instead of asking the model to look at an entire block of SQL and tell you what all the columns mean, you parse the SQL and you say, "Hey, here's one column. Here's its type. Here's the comments." you know, is this a measure, is this a dimension, is this blah blah blah, then at that point, you can ask the agent to do classification and summarization, and those are the things that it's good at, right? And if you want to weigh it more, um, you give it a more you give it more, um, prompt input to that single focus question. You don't try to cram more questions in. You give it more context for it to be able to answer that single question with. So, small, grounded, auditable, semantic operations over our structure. Um, that's what our library fenic is for, and that's what we use it for as part of typed.

嗯，所以最后一个阶梯，也就是阶梯的最后一级，基本上就是我们把所有这些东西都整合进来了，也知道它们是如何相互关联的。现在，这实际上意味着什么？现在，我们可以利用最后一层中的 LLM，开始提取其中一些语义层面的理解，弄清楚这些东西究竟意味着什么。嗯，是的，正如我之前所说，这里还能做的事情要多得多。Um, so that last ladder, the last rung in the ladder, which is basically we have all this stuff layered in, we know how everything links together. Now, what does it actually mean? Now, we can start pulling some of that semantic understanding of what things actually mean by using the LLMs in the last layer. Um, yes, so as I was talking about before, there's a lot more that you can do here.

你实际上想构建的是一个代码库图谱分析代理。你可以利用调用图来理解是谁在调用这里的内容，以及哪些地方引用了它，对吧？它还能读取代码内部和外部的注释，并整合代码之外的说明文档、GitHub issues、Linear issues、工单之类的内容。这样，你就能把它构建成真正更有用、也更有依据的东西。做法是从小处着手，从能做的最简单的事情开始：如何从这个代码库中提取结构？然后再进一步思考，如何利用这些结构，叠加上语义理解，从而构建一个有用的基础层，让代理能够在这个空间里真正开展工作并进行修改。这大致就是最终目标。 总之，非常感谢大家。这就是我的演讲《代理会 grep》。我是来自 Typed Defev 的 Brandon Calendar。这里再放一次工作坊的链接，还有我的 LinkedIn。you actually want to build a coding uh codebase graph analysis agent um you can use you can leverage call graphs to understand who's calling this and who references this right and you know it can read the comments inside and outside of the code and it can fold in pros from outside of the code it can fold in GitHub issues and and and your your your linear issues and your ticket and stuff so that's how you build this into something that's actually more useful and and and grounded is you start small you start with the smallest possible thing you can do which is how do extract structure out of this codebase and then you move on to how do I take that structure and layer into it the semantic understanding that's really going to help um build out a substrate that's useful for agents to actually work inside of this space and make changes um and that's sort of what the uh where the end goal is there so anyway thanks so much guys uh this has been my talk the agent will grip anyway um I'm Brandon calendar from typed defev here is the link again for the workshop and my LinkedIn.

那么，呃，谢谢大家。大家有什么问题或其他事情，我都可以回答。非常感谢各位。So, uh, thanks. I can take any questions or or any other things, but thanks so much everybody.

边界、索引与稳定的评估快照Boundaries, indexing, and stable eval snapshots

还有问题吗？我可以把麦克风传一圈，进行最后一轮提问。你们有没有尝试过这种跨代码库的工作流？比如涉及微服务，或者需要跨越 HTTP 边界的情况？效果如何？Any any questions? I can pass around the mic for one last round. Have you tried this kind of workflow like across repositories where things might be either microservices or like going across HTTP boundaries um with any success?

主持人：嗯，不算特别是吧。我觉得这会派上用场的地方，是在单个代码仓库中。我们现在会在规模更大的数据架构产品中大规模地做这件事。所以我觉得这是可行的。嗯，基本上，难点显然在于弄清楚不同事物之间的边界。因此，你基本上需要编写边界分类器，试着理解……我本来想用 Python 和 Rust 的例子来说明，比如通过 maturin，这样你也可以有一个类似的例子，但我没时间做。基本上，你需要想办法判断 Rust 调用的代码何时在调用 Python，或者反过来，对吧？这样才能构建它们之间的链接层。嗯，所以没错，你可以做到。我只是觉得这需要多做不少工作。Host: Uh not particularly no. I think where where this would where this would come into play is on a single repository. Now we do that at scale for our like larger data architecture product. So I think it can work. um you basically just need to the hard part there is obviously figuring out the boundaries between the different things. So you basically need to write boundary classifiers that basically try to understand um I was trying to do an example with Python and Rust like with maturin so you could have a similar example of that but I didn't have time to do that but you basically would need to write some way of understanding when rust called code is calling Python or vice versa right to try to like build out that linking layer between them. Um, so yes, you could do it. I just think it would it would require quite a bit more work.

主持人：还有其他问题吗？Host: Any other questions?

客人：抱歉。Guest: Sorry.

主持人：我明白了。Host: I got you.

嘉宾：Cursor 做了很多二级层面的事情。比如，你会看到 Cursor 在代码库中搜索时，通常会使用语义搜索，也就是向量嵌入搜索，这种方式确实非常有效。我觉得，各种代码 RAG 产品之所以会采用语义检索，就是有原因的，而且这确实很棒。只是，这种方法能帮你找到代码中所有你想让智能体检查的位置，但你还是得让智能体逐一检查这些位置，对吧？这种方法的目标是：如果我只想用一次操作就能做到这一点——比如我知道自己正在看这个函数，我想像在 IDE 里那样点击它，查看所有调用位置，但这一切都在我的智能体中完成，不需要智能体把所有相关位置都抓取出来。Guest: Cursor does a lot of the um level two stuff. So like when you see like cursor doing searching through your codebase, it's usually doing semantic search like vector embedding search um and that's that's that that does work really well. Like I think that there's a reason why there's all these like code rag products that do um that do semantic or retrieval, right? And I think that that's that's really great. It's just that you know that approach does get you the ability to find all the places in the code that you want to then examine with the agent. But you still have to go and examine all those places with the agent. Right? The goal of this approach is if I want to be able to in one shot say I know that I'm looking at this function right here, right? I basically want to be able to do the IDE click on the thing to see all its call sites but within my agent without my agent needing to grab everywhere.

这样说得通吗？还有其他问题吗？好，这边有一个。稍等一下。That make sense? Any other questions? All right, one over here. One sec.

主持人：我想，构建这个东西的一半乐趣就在于，呃，我也说不好，你最后会得到某种图的向量表示，可以把它调取出来。Host: I imagine half the fun of building this is that you end up with uh I don't know, sort of a a vector of the graph that you can recall.

嘉宾：是的。嗯，我真的很好奇，当你开始认真考虑如何扩展时，缓存和存储会是什么情况，因为要引导智能体，确保它第一次调用就做对至关重要。Guest: Yeah. Um, I'm really curious about caching and storage as you really start thinking about scaling this given that uh making sure the agent gets it right on the first call is so key in steering.

主持人：嗯，这么说确实有道理。我觉得，呃，随着你开始扩展你的图数据库，你会发现，呃，事情很快就会变得非常棘手。你说得完全正确。嗯，我觉得存储方面没那么令人担忧，因为，没错，这些图确实往往会变得有些庞大，尤其是随着它们不断增长，也更难进行垃圾回收。不过，嗯，我相比存储，更担心的是实际运行图分析、处理所有这些不同事物所需的计算量。嗯，所以我觉得，随着你尝试扩展它，正如我之前所说，你需要重点关注的主要是增量化：弄清楚如何在不重新生成所有内容的情况下重建这个东西，我认为这可能是最重要的部分。为了在大型数据架构中以更大的规模构建这个东西，我们在 Typed Death 就一直在为此费心，因为你肯定不想再去查询数据仓库 500 次，对吧？你也不想再检查一遍、重新编译所有 SQL 代码，对吧？Host: Yeah, that makes sense. I think um as you as you start to scale out your um your graph database, you're going to start finding that it's uh it can get very hairy very fast. And you're you're absolutely correct. Um, so I think storage is less of a concern, I think, because like these like yes, these graphs do tend to uh get a little sprawly, especially as they grow and they're harder to kind of garbage collect for, but they do um I'm less concerned about storage and more concerned about like the amount of compute that it takes to actually run like graph analysis across all these different things. Um, so I think the as you try to scale it up, I think the main thing you need to focus on like I was saying earlier is the incrementalism like figuring out how to rebuild this thing without regenerating all of it I think is probably the most important part and that's something we fight with at typed death for trying to build this you know at a larger scale across a large data architecture because you don't want to have to go and query your warehouse you know 500 times again right you don't want to have to go check you know compile and all the SQL code again right?

你希望能够在内容不断到来的过程中持续更新它。所以我觉得，解决这个问题最重要的方式，就是从一开始就把增量索引作为一等公民来设计。嗯，只有这样，你才能逐步走到需要处理其他扩展性问题的阶段。我认为，你得先处理信息重新同步的问题，远早于需要面对真正的扩展性问题——比如如何存储这些信息，或者如何对所有信息进行计算。是的。我发现，对我们来说，构建并维护黄金数据集始终是最大的问题。You want to be able to update the stuff as it's coming in. So, I think the most important way to get around that is design for incremental indexing first as a first class citizen. Um, that's kind of the only way you'll be able to get to a point where you have to deal with the other scaling problems. I I think you have to deal with the problem of reyncing the information long before you have to deal with the actual scaling problem of how to store it or how to compute across it all. Yeah. I found that for us the building of the golden data set and maintaining it is always the biggest problem.

嘉宾：是的，我百分之百同意你的看法。我……这确实是个我没有特别好的答案的问题。我不这么认为，因为我……我觉得……嗯，其实我对此有两个答案。其中一个是，黄金数据集不一定……如果我真的在构建这个东西，并且在某些内容上测试它，我会选定这个代码库的一个特定快照，对吧？这样我就可以说这是我的快照，而且每次运行评估时，我都有一组确定不变的黄金问题，对吧？这算是有点回避你的问题，对吧？但如果我真的在设计这个东西，我就会这么做，对吧？也就是针对某个快照来设计它。这样我的评估问题就是确定不变的，不会随着时间推移而变化，对吧？因为理想情况下，对吧，你的智能体会越来越擅长回答同一批问题。Guest: Yeah, I I agree with you 100% to that and I I that's that is one I don't have a great answer for. I don't think because I I think I think so. I have two answers to that actually. One of them is that like the the gold data set isn't necessarily if I was if I was actually building this and I was testing it on something, I would pick a specific snapshot of that codebase, right? So I could say that this is my snapshot and that when I run my evals, I have a deterministically set of my gold questions, right? That's that's kind of a copout to your question, right? But that's what I would do if I was actually designing this, right? Is I would I was I would design this against a snapshot. So my eval questions are deterministic and aren't changing over time, right? Because the in an ideal world, right, your agent is going to get better and better at answering the same questions.

但根据不断变化的语料库来更改问题，并不能真正帮你评估智能体有没有进步，对吧？评估智能体有没有进步，最好的办法是让它们再做一遍完全相同的任务，对吧？所以你的回答大概是：不会，因为如果是为了做评测，你的标准答案就应该基于固定的语料库保持不变。这样说得通吗？如果你是在做线上测试，那当然，你需要想出新的方法来针对你的代码库进行测试。但我觉得这和你那套固定的评测集是两回事。这样说得通吗？But changing the questions based on changing the corpus doesn't actually help you evaluate if your agents are getting better. Right? The best way to evaluate if your agents are getting better is have them do the exact same task again. Right? So it's sort of your answer is you don't because you if for the purpose of evals your gold should be constant against a constant corpus. Does that make sense? Now if you're testing live yes like you need to be able to figure out new ways of testing it against your codebase but I think that and like your fixed eval set are different things. Does that make sense?

主持人：是啊。不过就像……Host: Yeah. But like

嘉宾：对，对，不过我是说，比如说，如果我们用代码库这个例子，假设你有一个比较复杂的 bug，比如一个函数调用了另一个函数，而其中一个函数里有 bug，或者像你说的，有一个 if 语句。然后你有一个 eval，用来测试：如果引入了这个 bug，agent 是否会找到这个函数并把它修好。但之后你做了重构，这个函数实际上已经不再使用了。于是 eval 就失效了。Guest: Yeah. Yeah, but I mean like if you so if we go with the codebase example and let's say you have a kind of like a complicated bug that is like a function that is calling a different function and there's like a in one of those functions there's like a bug or there's like a if like you said and then you have an eval that tests that if this bug was introduced then the agent went to this and this function and fixed it but then you do a refactoring and then this function is actually not even used anymore. So then the eval breaks.

主持人：哦，我明白你的意思了。那样的话，我觉得总体上还是一样的。我的意思是，我在构建评测时，会说这个评测由几部分组成：我的代码所在的某个特定提交，对吧？我想引入的、用来看看智能体能否修复问题的特定差异，以及根据那个提示产生的一系列更改，对吧？这些应该始终保持不变。我明白你的意思，因为代码确实会变化，而且之后可能不再需要修复那个问题了，但你测试的不是这个，对吧？你不是在测试智能体应对新代码的能力，而是在测试智能体能否根据你编写评测时所处的情境做出那项更改，对吧？我觉得那个任务并没有变。Host: Oh, I see what you're saying. So in that case, I I think it's still I think man is still the same because what what I'm saying is that I when I'm building my eval, I would say my eval is a combination of a specific commit that like my code is at, right? The specific diff that I want to introduce to see if the agent can fix it and then the resultant set of changes from that prompt, right? that should always stay constant. Like I I I see what you're saying because like yeah, the code's going to change and it's not going to need to fix that issue anymore, but that's not what you're testing, right? You're not testing the agents ability to react to the new code. You're testing the agents ability to make that change based on that circumstance that it was in when you wrote the eval, right? Like that task I don't think has changed.

即使它好像已经不再相关了，它仍然在测试某些有用的东西。这样说得通吗？even if like it's not relevant anymore, it's still testing something that's useful. Does that make sense?

本体论与更广泛的数据上下文愿景Ontologies and the broader data context vision

问题。Question.

嘉宾：讲得真好。我很喜欢。嗯，这让我想起了本体论，嗯。Guest: Great talk. Loved it. Um this kind of reminds me of ontology um

主持人：课程，而这实际上正是 Palunteer 靠它赚得盆满钵满的东西。Host: classes and this is what Palunteer is making a killing out of actually.

主持人：是的。那么，嗯，一旦你构建了本体和依赖关系，我们现在能用这些更高层次的结构与 LLM 对话吗？Host: Yeah. So um once you build the ontology and the dependency um can we talk with the LLM in these higher constructs now?

嘉宾：是的。Guest: Yes.

主持人：使用这些类。Host: With these classes.

嘉宾：是的。Guest: Yes.

主持人：所以，这基本上就是你此时此刻暗示的意思吗？Host: And is this is essentially what you were kind of insinuating at this point in time.

嘉宾：是的。所以这基本上就是我们的产品，对吧？这个概念就是，我们构建上下文图，在它之上构建本体层，尝试围绕你的数据架构来组织所有信息：各项内容指向哪里、存放在哪里、分别意味着什么。然后，我们让智能体访问这套工具，以便它能够回答问题，并在你的数据架构中进行更改。不好意思。好吧，这个问题可能有点天真，不过回到你们正在构建的产品，也就是你们在这里展示的东西。你们展示的很多内容，和 Neo4j 在早些时候的一场研讨会上展示的非常相似。Guest: Yes. So that that's that's basically our product, right? Is is is that concept of we we build um we build the context graph, we build the ontology layer on top of it where we try to organize all the information around your data architecture, where things are pointed, where things live, what things mean. Uh and then we give the agent access to that tool set to be able to answer questions and and make changes within your data architecture. Excuse me. All right. So, this may be a little naive, but so going back to kind of the product you guys are building, what you demonstrate here. So, a lot of stuff you showed here is very similar to what Neo4j showed at an earlier workshop,

主持人：这就像，好吧，这是一个知识图谱。你作为具备领域专业知识的开发者，会在这个领域构建一个知识图谱，然后把它提供给一个智能体，让那个智能体查询 Cipher，以获取一些语义信息。我负责的，我姑且称之为一个庞大的数据环境，其中有很多第三方软件即服务，还有其他一些内部系统。我的团队负责为这些系统汇集大量信息。其中有些系统高度可配置，比如 Salesforce。你把信息导入进来，放进数据仓库，等等，最后另一端就有了商业智能和交互功能。我的目标是，如何让团队里这支规模不大、能力不错的队伍，只需在一个地方定义一次所有这些事物之间的语义关系，并且做好版本控制？这样我就既能为 AI 智能体提供上下文和工具，让它们主要用于分析工作，偶尔也能写回数据；还能为必须使用这些系统的人提供文档，无论是用于入职培训，还是集中放在一个地方——我还没想好该怎么做才合适。Host: which is like, okay, here's a knowledge graph. You as the developer with domain expertise are going to build a knowledge graph in this area and they're going to provide it out to an agent and then make that agent query cipher to get some semantic information. I'm responsible for I'll call it a sprawling data state where there's a lot of you know thirdparty software as a service. We've got other things that are internal. My team is responsible for you know bringing in a lot of information for those things. Some of them are highly configurable like a salesforce. You bring the information in you put in the data warehouse etc etc and eventually you have business intelligence on the other side and also interaction. So the goal is how can I get my pretty small team of pretty good people to define the semantic relationships between all of this stuff in one place one time with good version control so that then I can both provide context and tools to AI agents for mostly analytic work but also write back occasionally and documentation for human beings that have to use these systems whether that's onboarding or in one place I haven't figured out the right way to do that yet.

嘉宾：我们聊聊吧。Guest: Let's chat.

主持人：所以，你是、你是要……你会把那个给我吗？因为你也说过，比如说你不打算做数据目录，Host: So, are are you gonna are you going to give me that? Because you also said like you're not going to do the data catalog,

主持人：对吗？Host: right?

嘉宾：不，我们不会……嗯，事情是这样的，我想，我想，我的意思是，我们不会局限于数据目录，因为我们读取的不只是你的 Snowflake 目录，也不只是你的数据仓库信息。我们的目标是读取你的 Salesforce、你的 Fiverr、你的 Airbyte、你的各种系统，以及你拥有的所有数据传输机制，比如 Airflow，或者其他任何东西。把所有这些都整合到一个统一理解的上下文树——呃，上下文图里，这样我们就能真正遍历它。然后，我们的目标是从中提炼出你刚才提到的那些东西。语义层的主要问题是，没人愿意维护语义层，对吧？它算是个不太讨喜的词，因为你一旦构建好了，它确实很酷、很炫、很性感，但之后每次添加模型，你都得不断更新它，这简直是噩梦。Guest: No, we're not gonna Well, the thing is I think I think what what I what I mean to say is we're not going to be limited to the data catalog because we're not going to be reading just your Snowflake catalog, just your, you know, just your warehouse information. The goal is to be reading from your Salesforce, from your from your from your fiverr, from your your um your air bite, from your whatever, from all the transfer mechanisms that you have, your airflow, your whatever. Incorporate all of that into a single understood context tree uh context graph that we can actually traverse. And then from there, the goal is to actually um distill out of that the things that you were talking about. The the main problem with semantic layers is no one wants to maintain a semantic layer, right? It's kind of a dirty word because once you build one, it's all really cool and and sick and sexy, but then you have to actually keep updating it every time you add models and it's it's a nightmare.

所以，typed defe 的目标就是……你真的只是……这太棒了。谢谢你，你的解释让我们非常清楚地了解了我们希望产品能做到什么。正如你所说，它应该能够跨越数据架构中的所有不同层级进行推理，并基于一张统一的图谱，追踪数据如何从 Salesforce 一路流到 BI 层，对吧？然后，当你的智能体——当你向智能体提问，比如“嘿，那个……我的 AR 是从哪里来的？”它就能回答：“哦，没错，实际上它最初来自这里的 Stripe 数据，然后在这里与 Salesforce 潜在客户数据关联起来，用来确定下面这个东西——也就是实际计算数据的地方。”So, the goal of of typed defe is you really just it was great. Thank you for like gave a great understanding of exactly what we want the product to be able to do, which is exactly what you said is to be able to reason across all those different layers that you have in your data architecture. build off a single graph that you can actually trace how data is flowing from Salesforce all the way down to your BI layer, right? And then when your agent asks when you ask your agent a question like hey um where is my like where is AR being coming from it's able to say oh yeah great that originates up here in the Stripe data actually and then it gets joined into your Salesforce lead data down here um to like figure out you know this this thing down here which is where the actual data is being calculated.

是的，像这样的数据血缘非常有用，不过对我们来说，你知道，这在一定程度上是给负责这项开发的数据团队用的。很好。但我希望最终业务部门也能访问这些信息，对吧？这样他们查看报告时，就能发现：“这不对啊。搞什么？”然后他们就不用跑来找我们团队冲我发火，而是可以说：“哦，这实际上归这个系统里的这些人负责，具体是这条记录；我得去找这个人沟通。”Yeah, like that that lineage is very useful, but and for us it's, you know, partially for the data team that's doing this development. Great. But I want that to eventually be accessible to the business, right? So they can look at their report and we're like, that's not right. What the hell? And then rather than coming to my team and to yell at me, they can say, "Oh, well that's actually owned by these people on this system and this record and like this is the person I have to talk to."

主持人：那么Host: So

嘉宾：把所有内容放在一起。Guest: keep it all together.

主持人：我百分之百同意你的看法。我觉得，我们想做的一件事是，LLM 在较高层面上让我们能够充当一种翻译层，而这件事以前未必能做到。这一层位于业务人员、会计师、分析师，以及数据分析师、工程师之类的人之间，对吧？我们想说的是：好吧，业务分析师说，嘿，收入仪表板上的那个数字错了，对吧？那他们要修正这个问题，就得去某个地方提交工单，工单被放进某个系统里。有人接手、分诊，试着弄清楚这东西归谁负责，对吧？然后把工单转给对方。接手的人还得跟最初的人沟通。所以，真的挺疯狂的，对吧？我之前在 Salesforce 工作过，所以对这类事情多少有些了解。Host: Agree 100% with you. And I think that what we're trying to like one thing that the LLM sort of let us do at a high level that wasn't necessarily possible for is be that kind of translation layer that sits between the business folks the the accountants the analysts right the and then the data analyst the the engineers or whatever things like that right because what we're saying is what we want to say is okay that business analyst says hey that number in that revenue dashboard is wrong right so the process for them to fix that would be Okay, I need to go file a ticket somewhere that gets sat in some system. Someone picks it up, triages it, tries to figure out who owns that thing, right? Sends it over to them. They had to talk to the original. So I it's crazy, right? I worked at Salesforce for this, so I I get a little bit of that on a high level.

嗯，所以类型缺陷的目标是让业务分析师能够进去说：“嘿，我的仪表板坏了，对吧？”他们可以输入仪表板的名称。我们可以在他们当前所在的会话中调出仪表板，然后一路追溯并说：“哦，是的，我想我知道问题出在哪儿了。很可能是因为……我往回追溯了好几层，发现这里出了问题。我能识别出这个模型归上面的这个团队所有。我可以直接帮你提交工单，并且已经把解决方案信息记录在工单里，因为我已经能看到根本原因了：我往回查了三层，发现这里的 select 写错了，对吧？我可以把这个问题追溯到上周合入的一项变更。哦，原来这就是这里出问题的原因，对吧？”Um, so the goal with type defeaf, right, is for able that that that business analyst to be able to go in and say, "Hey, my dashboard is broken, right?" They can put in the name of the dashboard. We can pull up the dashboard, you know, in that session where they are. we can trace back and say, "Oh, yes, I think I see what's the problem here. It's probably that, you know, I'm multiple layers back, this is happening. I can detect that this team owns this this model up here. I can go ahead and file that ticket for you with the resolution information already encaptured in it because I already can see what the root cause is because I looked back three layers and saw that this select was wrong, right? I can trace that back to a change that landed last week, you know, oh, that's why this broke here, right?

哦，而且我们能看到这是什么时候发生的，因为这边有一次 Airflow 运行，实际上填充了那些数据，对吧？我们想要实现的就是对一切如何运作有一个统一的视角，从运行时到数据血缘，并让智能体能够回答这类问题，介于不同层之间，让整个流程更高效。因为如果不是让这张工单一直搁着，在五个不同团队之间来回流转，而是能立即弄清楚问题是什么，并马上将它分派给合适的团队，那么即使是在一个理想的世界里——我们的产品无处不在，而且运行完美——那个团队也可以直接说：“没错，这个方案听起来不错，打个勾，然后就去实施。”对吧？这就是我们的愿景，对吧？And oh, we can see when that happened because there was an airflow run over here that actually populated that data, right? That's what we're trying to get to is that unified vision of how everything works from runtime through lineage and being able for the agent to actually answer those kind of questions and sit between those different layers and make that process more efficient because if if instead of that ticket sitting around and bouncing between five different teams, if it's immediately understood understood what the problem is and that gets routed to the right team right away, that team can even just in an ideal world where our product is ubiquitous and works perfectly, that team can just say, "Yeah, that solution sounds good check plus and then it goes and implements it. Right? So that's that's the dream, right?

所以，嗯，这大概就是我们想要实现的目标，而且我们确实已经取得了一些进展。很乐意聊聊。So but that that's kind of what we're trying to to get to and we're we are definitely part of the way there. So be happy to chat.

嘉宾：好。还有最后的问题吗？很好。如果大家想在 LinkedIn 上或之后任何时候联系我们，我们都会在。我们很乐意聊聊上下文图谱。非常感谢大家的到来，回头见。Guest: All right. Any final questions? Cool. We'll be around if you guys want to connect with us on LinkedIn or anytime. We love chatting about context graphs generally. So thank you guys so much for coming through and we'll catch you guys later.

主持人：谢谢大家。Host: Thanks everyone.