超越 RAG：Unblocked 如何构建节省 token 的关系型上下文引擎

简要说明Brief Description

超越 RAG：一种降低令牌消耗的关系型上下文引擎 — UnblockedBeyond RAG: A Relational Context Engine That Cuts Token Burn — Unblocked

目录Table of Contents

• 简介
  Intro
• 议程
  Agenda
• 演示：有上下文和无上下文的智能体
  Demo: agents with and without context
• 成本减半
  Half the cost
• 追踪模型回归问题
  Tracing a model regression
• 开始构建吧
  Let's build
• 向量搜索无法回答的问题
  Questions vector search can't answer
• 难点：搭建脚手架
  The hard part: scaffolding
• 文档查询引擎
  The document query engine
• 自动发现架构
  Automatic schema discovery
• 身份解析
  Identity resolution
• 验证查询
  Validating queries
• 重试
  Retries
• 全文搜索
  Full-text search
• 上下文引擎模拟器
  Context engine simulator

简介Intro

我们开始了吗？好。大家能听到我吗？这样就可以了，很好。好。那么，我们要快速过一遍，因为我们只有 20 分钟。我是 Unblocked 的 Peter。在 Unblocked，我们打造的是一款上下文引擎。我会给大家展示几样东西。这次演讲分为两部分：先介绍什么是上下文引擎，以及它如何为你的组织创造价值；然后我们会实际动手构建上下文引擎的一个组件。我还会介绍几个开源项目，大家听完之后可以去试试。Are we up? Okay. Can you hear me? Just like this just fine, everyone. Good. Okay. Um, so we're we're going to be doing a bit of a speedun here because uh uh we've only got 20 minutes, but um I'm I'm Peter from Unblocked. And at Unblocked, we build a context engine. So I'm going to show you a couple of things. The talk's going to be split into two parts. Um I'm going to show you what a context engine is and how it can add value to your organization. And then we're going to actually go and build a component of a context engine. And I'm going to show you a couple of open source projects that you can uh go and play with after this.

议程Agenda

这就是议程。什么是上下文引擎？它如何创造价值？然后我们会动手构建。嗯，我们先把时间拨回几年，回到 AI 智能体出现之前。那时候，你就是上下文层。想象一下，你是某个组织的新员工，正努力了解当前的情况。通常，你得查看大量代码，在各种文档系统里四处翻找，只为找到完成工作所需的信息。更不用说，组织在处理各种事件的过程中，随着时间积累下来的共同经验，以及由此留下的种种教训。嗯，上下文引擎所做的，就是汇集所有这些组织知识。原本散落各处的上下文被输入上下文引擎，经过处理后，输出的就是有依据的上下文——也就是说，各部分上下文彼此关联。So there's the agenda. What's a context engine? How does it deliver value? And then we'll build. Um so let let's just hop back in time maybe a few years to the times before AI agents. And in those days um you were the context layer. Imagine that you're a new employee to an organization and you're just trying to get to grips with what's what's happening. What you usually end up having to do is look at the look at a whole bunch of code, rip around documentation systems just trying to find the information that you need to get your work done. Um, and not to mention like all of the the sort of shared history that's built up over time through an organization dealing with incidents and the battle scars that come with it. Um, so what a context engine does is it takes all of that institutional knowledge and it pulls it in and what you get the scattered context that goes into the context engine and out the other end you get grounded context which means that the context is all interrelated to each other.

它是针对具体意图的，是个性化的，而且会考虑权限。所以我来给你具体展示一下这是什么意思。现在假设我要进入这里……It's intent specific. It's personalized and it's permissions aware. And so I'm going to show you exactly what I mean by that. Now let's imagine that I'm going to go into here into

演示：有上下文和无上下文的智能体Demo: agents with and without context

Claude 和我将向你展示几个我经历过的有趣会话。嗯，我在这里让 Claude 在没有接入上下文引擎的情况下，为我们某一层中的一个名为 source mark engine 的组件制定优化计划。我想大家都很熟悉 Claude 开始新任务时的表现。它就像一名新员工，对你的代码库或组织一无所知。所以它做的第一件事就是在你的代码库里四处摸索，试着弄清楚情况。你可以在这里看到这一点。你知道，它最终会弄明白——或者至少会很自信地表现得像是已经弄明白了。嗯，然后它会给我一些建议。现在，我来给你们看看把上下文引擎接入后端之后会是什么样子。Claude and I'm going to show you a couple of interesting sessions that I had. So um I asked Claude here, this is without a context engine attached to create a plan to optimize a component in our layer called the source mark engine. And uh I'm sure you're all familiar with what Claude does when it starts a new task. It's like a new employee. It doesn't understand anything about your your codebase or your organization. So the first thing it does is it reps around your codebase and and tries to figure out what's going on. And so you can kind of see that here. And you know it figures it out eventually or at least you know it confidently projects that it's figured it out. Um, and so it provides kind of like a a recommendation for me there. But now I'm going to show you what it looks like when you attach a context engine to the back end.

所以现在它的做法是调用 unblock 上下文引擎，而 unblock 会立即为这项任务提供一份非常全面、但又针对性很强的上下文资料包。你可以看到，它引用的不仅有源代码，还有讨论架构的 Notion 文档、拉取请求，以及关于优化这个特定组件的 Slack 对话等等。最终，你会得到这样一份内容紧凑、附带所有这些链接的上下文资料包。这样一来，如果 Claude 想要回头获取更多信息，就有了这些很好的切入点。不过，真正有意思的是这一点：我在这上面花了 1.29 美元，耗时大约 1 分 30 秒。这是使用上下文引擎的情况；而不使用它的话，成本最高要到 260 美元，耗时 3 分钟。So now what it does is it calls the unblock context engine and unblock now delivers a very comprehensive but taskspecific context bundle for this task. And what you can see is that it it's referencing not only the source code but also you know notion documents that talk about the architecture um pull requests slack conversations that were had about optimizing this particular component um and the the the net out is that you get this really tight context bundle with all these links. So now uh if if um you know Claude wants to go back and fetch more information, it has all these great jump off points. But the really interesting thing is is this. So here I spent um you know a dollar 29 on this and it took about a minute and 30 seconds. Um that was with the context engine but without I'm spending up to you know 260 and 3 minutes.

所以你看到的基本上是So what you see essentially is

成本减半Half the cost

成本降低了 50%，这相当惊人。嗯，现在我想给你们看另一个有意思的演示。嗯，这是我同事前几周和 Unblocked 进行的一次有意思的对话。哎呀，抱歉，我把这个调出来。嗯，这次对话挺有意思的。我同事……我们也有一款代码审查产品，Richie 就负责这款产品。他发现代码审查产品发现的问题数量开始减少了。于是他就问了 Unblocked，Unblocked 能够调取这些信息，嗯，然后你知道，把范围缩小到非常……a 50% reduction in cost which is pretty incredible. Um now I want to show you another interesting demo um which is uh an interesting session that my colleague Whoops sorry bring this out here. It's an interesting um session that my colleague had with unblock the other week. So what he we also have a code review product and Richie works on that and he discovered that the number of issues that the code review product was surfacing was was starting to drop. So he just asked unblocked and unblocked was able to pull this information um and and you know narrow it down very

追踪模型回归问题Tracing a model regression

很快就找到了一个可能的原因：我们从 Opus 48 或 Opus 46 切换到了 Opus 48，而背后的行为特征，你知道，导致所有这些问题都消失了。于是他做的就是让 Onblock 直接修复，现在 Onlocked 能够在后台直接创建一个 PR，而且它掌握了所有这些上下文，也知道该怎么做。但真正酷的是，你可以看到它生成了这段描述，这简直令人惊叹：它理解了最初导致回归的那个 PR，然后找到了一段讨论此事的 Slack 对话，并从中提取了一些数据来帮助确定修复方案，这简直太不可思议了。好了，我要回到……我想我把我的主题演示文稿弄丢了。quickly to uh a likely cause which was that we switched from Opus 48 or Opus 46 to Opus 48 and the behavior characteristics behind the scenes um you know caused all those issues to drop and uh so what he did is he asked on block to just fix it and now on locked um was able to just create a PR behind the scenes and it has all that context and it knows what to do. But the really cool part is you can see that it created this description and it's it's just remarkable like it it um understood the PR that caused the regression in the first place and then it found a Slack conversation where the conver where where this was being discussed and pulled on some of the data here to help inform um the fix which which is just nuts. So, I'm going to go back to wh I think I lost my keynote presentation here.

我们看看。好了。所以，你们在代理方面也有同样的问题，对吧？成本。哦，抱歉。这不对……这是错误的主题演讲。Let's see. There we go. So, you have the same problem with agents, right? Cost. Oh, sorry. This is the wrong It's the wrong keynote.

让我们开始构建吧。Let's build

好了。那么我想我们现在要做的，就是直接进入构建环节，因为今天时间相当紧。所以我先简单总结一下。你的上下文分散在各处：PR、评审、评论、议题和工单、文档、对话里。所有这些内容都可以通过语义查询，但它们也有结构化的一面，而 RAG 很擅长处理内容这一部分。它可以处理诸如内容相关的问题，找到代码、修改代码的 PR，以及正在讨论这件事的 Slack 讨论串。如果把这些东西放进一个智能体循环里，它能做的事情会非常非常多。你可以给智能体循环提供工具，让它接下来进行文件搜索，这会非常强大；但它无法处理其他一些类型的结构化问题，比如 PR……There we go. So now I think what we're going to do is we're just going to get right into the building part because we're we're quite short on time today. Um and so I'm just going to summarize this. You've got context scattered all over the place. You've got context in PRs, reviews, comments, issues and tickets, docs, conversations. Um and and all of that is uh semantically queryable, but it's also structured and rag nails the content part. Um, it it can be it can handle things like content questions, find the code, the PR that changed it, the Slack thread that's debating it, and if you wrap that stuff in an agent loop, it can go really really far. You give an agent loop tools to do file search after this and it it's very powerful but it doesn't handle some of the other type of uh structural questions like PRs

向量搜索无法回答的问题Questions vector search can't answer

我上周合并了什么、谁审查付款最多、未关闭 PR 的数量等等。这些都是查询，不是语义搜索。所以我想再给你们看一下，回到 Richie 在这里举的例子：他通过提出一个基于查询的问题来开启对话，以此解除阻塞，因为其中涉及时间因素。“能给我看一张从 1 月 1 日开始按周统计的比率图表吗？”好。要做到这一点，不能只从向量存储中提取这些信息，因为这些信息并不是以那种方式编码的。所以必须通过某种查询引擎来完成。检索本质上有两部分：语义检索可以处理这类基于语义的问题；而今天，我们要研究的是结构化的这一部分。that I merged last week who reviewed payments the most open PRs etc. Um, and these are queries. They're not semantic searches. So, I want to just show you again, um, going back to Richie's example here that he started the the conversation with unblock by asking it a query based question because there's a temporal element. Can you show me a chart of ratio weekly since January 1st? Okay. So, in order to do this, it's not going to be able to just pull this information from a vector store because that's not where that information is encoded. So, it's it's going to have to do that um through some sort of query engine. And so, you know, retrieval has two halves essentially. We've got semantic retrieval and that can handle those types of semantic based questions. But today, we're going to work on this the structured part.

我们将把结构化查询或自然语言查询转换为结构化查询。但实际生成查询其实是We're going to turn structured queries or natural language queries into structured queries. But generating the queries is actually

难点：搭建脚手架The hard part: scaffolding

简单的部分。难的是围绕它搭建的支撑体系。因此，你需要做一些事情，比如理解 schema，这样才能把它传给 LLM，让它生成一个约束严密的查询。嗯，你还需要能够映射身份。如果我问“Rashene 有没有未合并的 PR”，它不会知道 Rashene 是谁，因为他在 GitHub 上的身份可能完全不同。所以我们必须想办法进行映射。然后，当你从 LLM 得到查询时，它是不可信的。它可能包含危险操作，也可能越过租户边界。因此，你必须有办法验证查询是否正确且安全。最后，嗯，即使做了验证，也可能仍然存在错误。嗯，实际对数据库执行查询时，也可能会出现错误。the easy part. The hard part is the scaffolding that sits around it. So you need to do things like understand the schema so you can pass it to the LLM so it generates the a tightly bound query. Uh you need to also be able to map identities. If I ask a question like open PR is by Rashene, um it's not going to know who Rashene is because his identity in GitHub might be completely different. So we have to be able to map that somehow. And then when you get the query out of the LLM, it's untrusted. It could have uh operations that are dangerous. It could cross tenant boundaries. And so you have to have some way of validating that the query is is uh correct and safe. And then finally, uh, even though you do that validation, there could be errors. Um, and when you actually do the query against the database, there could be errors there as well.

这样一来，如果 LLM 陷入这种情况，我们实际上可以让它尝试自我纠正。这就是我们在这里的目标。我们想要进入，呃，进入这个状态。我们希望能够说类似“由机器发起开放式 PR”这样的话，然后基于此得到这类查询。好，稍等一下。呃，这是我们要基于它进行构建的开源代码库。接下来我就直接来看它实际是什么样子。And so we can actually get the LLM to try to correct itself if it gets into that situation. So this is what we're aiming for here. We want to go to uh to this. We want to be able to say something like open PR is by machine and then we want to get this kind of query on the back of that. So, just take a sec here. U this is the open- source repo that we're going to be building against. And I'm going to jump right into what that looks like actually.

文档查询引擎The document query engine

所以，这就是这个仓库。它叫文档查询引擎。我想给你看看我们第一次启动它时是什么样子。所以，我可以说，比如说，PR 是 Peter 在 ground zero 做的。哦，其实我们还没运行它。那我先退回去。我们先运行它吧。现在我可以提问了。刚启动时，它还没有接入任何东西，什么也不会生成。所以，我们接下来会逐步把它搭建起来。好。我们首先要讲的是So, this is the repo here. It's called the document query engine. And I want to show you what it looks like when we just boot it up the first time. So, I can say, you know, PR is by Peter at ground zero. Oh, actually we're actually not running it. So, I'll go back. Let's run it first. Now I can ask my question. So, when you start it's going to have it's not going to be wired up at all. It's going to produce nothing. So, we're going to just gradually build this thing up here. Okay. First thing we're going to talk about is

自动模式发现Automatic schema discovery

自动化模式发现。这完全是动态的，是无模式的。并不是所有数据库都是无模式的，但也可以应用同样的原理。你可以在运行时动态发现模式。嗯，这种技术的做法是从文档存储中抽取大量文档样本，然后据此推导出一个模式。我想给你们展示其中一个棘手的部分，也就是推断枚举值。嗯，这样做的结果是，你可以把 LLM 当作雷神之锤来用，而且它确实能做得相当不错；但实际上，只用传统的过程式方法也能做得非常出色。所以，对于这类看起来像是非确定性的工作，别总是动不动就求助于 LLM。这类事情其实相当确定。automatic schema discovery. This is completely dynamic. is schemalist. Not and not all databases are schemalist, but the same kind of principle can apply. You can dynamically discover the schema at runtime. Um and so what what the technique involves is sampling a whole bunch of documents in the document store and then deriving a schema from that. And uh you know I just want to show you like one tricky part of that which is inferring enums. Um now the the the outcome from this is that you can use an LLM as a Thor hammer um and it can do a pretty good job but actually you get uh really really far just using traditional procedural methods. So you can you know don't always fall back to an LLM to do this kind of what looks like non-deterministic work. This is actually quite deterministic this kind of thing.

现在我们已经发现了模式，我来给你们看看它在我们的演示中是什么样子。哎呀。首先，我们得升级到下一个阶段，然后运行。现在我们已经从 GitHub 集合中发现了模式，这里只有拉取请求。好的，所以这里不仅有模式，还有一些有助于提升查询性能的索引。下一步是身份解析，也就是我之前提到的，把用户映射到真实身份。这个问题同样可以通过简单的程序化技巧来解决。所以，实际上我们做的大致就是这样。在底层，当我们获取用户名或……So now that we've discovered schema, I'm just going to show you what that looks like in our demo. Whoops. First we have to level up to the next and then run. So now we've discovered the schema from our um GitHub collection and this is just pull requests. Okay, so it has not only the schema here but also um indexes that will help improve the performance of queries. The next step is identity resolutions that thing I talked about about mapping you know users to uh real identities and um again this is this can be solved just using simple procedural tricks. So um that that's actually more or less how we do it. Uh and underneath the hooded unblocks when we take usernames or

身份解析Identity resolution

来自不同系统的身份信息，并将它们相互映射。你可以进行这类模糊匹配，最后如果需要，还可以把一组名称传给 LLM，让它充当裁决者。 最后，在真正展示那些很酷的东西之前，还有最后一步：合成。粗略来说，这其实是最简单的一步。我们只需获取之前动态解析出的 schema，再获取当前日期和时间，也许还要加上一些提问用户的信息，这样就能轻松解析出“我的 PR”之类的问题，而不只是“Peter 的 PR”。然后把这些信息传给 LLM，让它为我们合成一个查询。结果大概会是这样。我们来看这个。现在我可以回到引擎，再试着运行一次这个查询。identities from different systems and map them across um you can do this kind of fuzzy matching and then in the end if you have to you can pass a list of names to an LLM and have it act as a tiebreaker. And finally, uh, the last step before we can actually show the cool stuff is this. It's the synthesis part. And roughly speaking, this is actually the easiest part. We just take the schema that we dynamically resolve from earlier. We take the current date and time, uh, maybe some information about the user that's asking the question, so it can easily resolve questions like my PRs and not just Peter's PRs. Um and then we pass it to the LLM and ask it to synthesize a query for us. And the outcome uh will look something like this. Let's get to this one. So now I can go back to my engine and try to run that query again.

现在它会为我提取文档并生成这个查询。And it's going to pull the documents for me now and generate this query.

验证查询Validating queries

接下来我刚才谈到的步骤是验证。这实际上是最重要的事情。嗯，你的查询可能包含非常危险的运算符，比如 function 运算符，它会在 DB 服务器上执行原始 JavaScript，这显然不太理想。嗯，不过你可能还想用它做其他事情。比如，你可能有一些隐藏字段，不希望它们暴露给 LM，也不希望 LM 根据这些字段推导并执行查询。举例来说，如果你有一些包含租户 ID 的元数据字段，并且希望将每个查询限制在特定租户范围内，那么最好确保 LM 完全不接触这些字段，然后在幕后对查询加以限制。你可以在底层注入这些字段。大致就是这样。Now the next step I I was talking about was was validation. This is actually the most important thing. Um your your uh query could contain really dangerous operators like the function operator for example which would execute raw JavaScript um on the DB server which is definitely not ideal. Um, but there could be other things that you want to do with this. Like for example, you might have hidden fields that you don't want uh to be surfaced either to the LM or for the LM to to derive and execute. If you have metadata fields, for example, that contain um tenant IDs and you want to isolate each query to a specific tenant, you're better off making sure that the LM doesn't interact with those fields at all and then just constraining it behind the scenes. you can inject those fields underneath. And that's kind of what that looks like.

所以，我们再快速展示一下这样做的效果，以及它看起来是什么样子。我们跳到验证步骤。哎呀，再运行一次。现在，如果我问这样一个问题：“标题中带有 authentication 的 PR”，这可能会触发某种正则表达式操作。看，果然如此。好。所以，呃，我们的验证引擎提示说：“抱歉，不允许使用正则表达式”，于是把你挡了回来，现在我们不能这么做了。所以验证挺有用的。嗯，但我们该怎么处理这个错误呢？呃，我们确实收到了一个错误。但我们可以换一种做法：把这个错误传回给 LLM，让 LLM 自行修正。这就是它的原理。从技术角度来说，就是这样。它看起来就是一个带有某种最大重试次数限制的循环。So again, we're going to just quickly show the payoff for that and what that looks like. Let's go up to our validation step. Oops. Run it again. So if I now ask a question like this PRs with authentication in the title that is probably going to drive some kind of like regular expression thing and there we go. Okay. So uh our validation engine said sorry regular expressions aren't allowed you're getting bounced and now we're not allowed to do that. So validation is cool. Um but what do we do with that? you know, we got an error. Um, but what we can do instead is just take that error and just loop it back in to the LLM and have the LLM kind of self-correct. And that's kind of what it that's all it is. That's all it looks like from a technical standpoint. It's just a loop with some sort of maximum retry.

所以，如果我再试一次，And so if I try this again,

重试Retries

把重试次数调高，然后运行。现在，如果我再试一次这个查询。哎呀，选错了。我们试试这个。现在你可以看到，它确实尝试了几次。不过最后还是成功了。但是，它把查询限制为使用 match，而遗憾的是，这是精确标题匹配。我们没有标题完全匹配的 PR。所以我们还需要一样东西，那就是全文搜索。bounce up to my retries and run that. Now, if I try that query again. Oops, that's the wrong one. Let's try this one. Now what you can see is that it actually took a couple attempts. Um but it got there in the end. However, um it constrained it down to use match and unfortunately this is an exact title match. We don't have any PRs with that exact title. So we need one more thing which is this full text search.

全文搜索Full-text search

所以现在，如果我们给引擎添加全文搜索，希望最终效果会有点像这样。我们再试一次。太棒了！你看，我们从头到尾都走通了。这是完全开源的，所以你可以看看它是如何实现的，也可以自己动手试试。这就是我们构建出来的东西。它能够有效处理 RAG 的另一半。也就是说，凡是涉及结构化数据的内容，你都可以使用筛选器和聚合，设定时间范围，而且它还能自我纠错。但真正厉害的是，它还可以驱动聊天代理。比如我到这里问一句：“上周我们一直在做哪些与身份验证相关的事情？”现在你就能看到整个过程，它会清楚地展示自己在做什么，以及它生成的所有查询。So now if we add full text search to our engine um hopefully the outcome that we get is something that looks a little bit like this. So let's try it one more time. Awesome. So there you go. We got we went from start to finish there. This is all open source. So you can take a look at how it's implemented and play around with it yourself. And this is what we built. Something that can effectively handle all of the uh other half of rag. So, anything that has to do with uh structural based data, you can have filters, aggregations, it's time scoped, and it selfcorrects. But what's really cool is that it also powers a chat agent. And so if I go over here and I say something like um what authentication related things have we've been working on last week. Now you get this whole thing that will show you exactly what it's doing, all the queries that it creates.

现在它已经获取了数据，并在后台对这些数据进行推理。好了，就是这样。非常酷。你看，这就是面向智能体和人类的结构化、可查询上下文。现在只剩最后一件事。嗯，你可能已经在描述中看到，我们……And now it has the data and it's just kind of reasoning about it behind the scenes. And there we go. Very cool. So there you are. Structured queryable context for agents and humans. Now there's just one last thing. Um on the description you probably saw that we

上下文引擎模拟器Context engine simulator

我们之前承诺过会提供一个上下文引擎模拟器。如果你因为环境中的一些限制，不想马上接入 Unblocked，嗯，我们有一个开源项目，可以让你完全在本地完成这件事。它的思路是，针对每项任务为你创建一个上下文包，然后用同一项任务对这个上下文包进行 A/B 测试，让你看到让智能体随时掌握所有这些上下文所带来的好处。如果你需要，这是相关的 PR 代码。好。最后还有一件事，如果你想要椰子，欢迎来展台找我们。谢谢。were promising a context engine simulator. If you don't want to wire up unblocked uh right away because you've got some constraints in your in your environment, um we have an open source project that will allow you to do this entirely locally yourself. And so the idea is uh it will create a context bundle for you on a taskbytask basis and then take that context bundle and run an AB test against the same task and show you the benefits of having all of that context at your fingertips for your agents. And that's the uh PR code for that if you want that. All right. And one last thing, if you want some coconuts, come by the booth. Thank you.