把智能体记忆蒸馏成真正可用的技能：从记忆图到可治理技能 — Will Lyon，Neo4j

简要说明Brief Description

将智能体记忆转化为真正有效的技能——Will Lyon，Neo4jTurning Agent Memory Into Skills That Work — Will Lyon， Neo4j

目录Table of Contents

• 简介
  Intro
• 失忆循环
  The amnesiac loop
• 回忆并不等同于可付诸行动的知识。
  Recall isn't actionable knowledge
• 短期记忆、长期记忆和推理记忆
  Short-term, long-term and reasoning memory
• 从消息到知识图谱
  From messages to a knowledge graph
• 实体解析与本体
  Entity resolution and ontology
• 推理图
  The reasoning graph
• 决策轨迹
  Decision traces
• 代理间共享内存
  Shared memory across agents
• 从记忆到技能
  From memory to skills
• 智能体技能解析
  Agent skills, explained
• 散文技能的问题
  The problem with prose skills
• 研究：将技能表示为图谱
  Research: skills as graphs
• 基准测试结果
  Benchmark results
• 从记忆中提炼技能
  Distilling skills from memory
• 依据性、覆盖度与连贯性
  Grounding, coverage and coherence
• 技能治理
  Skill governance
• Neo4j 智能体记忆服务
  The Neo4j Agent Memory Service
• 蒸馏流程
  The distillation pipeline
• 创作技能
  Composing skills
• 演示：NAMS 仪表板
  Demo: the NAMS dashboard
• 演示：提炼一项技能
  Demo: distilling a skill
• 演示：患者接诊技能
  Demo: a patient-intake skill
• 资源
  Resources

简介Intro

我们开始吧。我看到有些人想进来了，欢迎大家进来。呃，这场会议的主题是可操作的知识和上下文图谱。我叫 Will，是 Neo Forj 的产品经理。嗯，到目前为止，这次大会上大家讨论了很多关于循环的话题，对吧？比如 React 循环、Ralph 循环。有时候我觉得我们可能正处于……Let's go ahead and get started. I see some folks wanting to to come on in. So, feel free to to come in. Uh, this session is going to be all about actionable knowledge and context graphs. So, my name is Will. I'm a product manager at Neo Forj. Um there's been a lot of discussion in this conference so far about loops, right? The uh React loop, the Ralph loops. Sometimes I feel like we're in maybe

失忆循环The amnesiac loop

就像一个失忆循环，对吧？我们的智能体经历推理和行动阶段，成功完成某项任务，却又有点忘了自己学到的东西。如今的记忆系统大多专注于将文本转换为嵌入，然后在检索时找到最相关的数据，塞进上下文里，对吧？嗯，所以像这样，我们会在语料库中找到相似的数据块，把它们注入上下文窗口，并相信智能体此刻能据此做些有用的事。现在我们面临的挑战是，你知道，召回并不真的像……like a an amnesic loop, right? where our our agents go through this uh reasoning phase acting. They successfully complete some task but then kind of forget uh what they've learned. And memory systems today are largely focused around uh embedding text then at retrieval time sort of finding the most relevant data to to stuff into the context, right? Um so so something like this we find you know similar chunks of data in in our our corpus we inject that into the context window and trust that the agent is able to do something useful with that right now the the challenge that we have is you know recall isn't really like

回忆并不等同于可执行的知识。Recall isn't actionable knowledge

可操作的知识。比如，我们遇到的一个挑战是，我们没有某个事物的规范表示，对吧？假设我们有三种不同的方式来指代 Nwin 医生、Robert N，也就是那位心脏科医生，对吧？具体采用哪种说法，取决于讨论的语境。我们需要为这个实体建立某种规范表示，对吧？所以，智能体需要的知识必须彼此关联、类型明确且可追溯，对吧？不能只是可检索而已。所以，检索只是……的一部分。actionable knowledge so one challenge we run into is like for example we don't have a canonical representation of a thing right so if we have you know three different ways that we refer to Dr. Nwin, Robert N, the cardiologist, right? Like depending on the the context of this discussion, we need to have some canonical representation of the thing, right? So really agents need knowledge that is both connected, typed and traceable, right? Not just retrievable. So retrieval is is uh just a part of the

短期记忆、长期记忆和推理记忆Short-term, long-term and reasoning memory

问题在于，当我们谈到智能体记忆时。NearJ 对智能体记忆的理解是这样的：我们认为它是一个由短期记忆、长期记忆和推理记忆组成的连接图。我们稍后会更详细地看看这一点。不过，请先跟我一起理解这个关于智能体记忆的上下文图的概念，好吗？那么，这里的几个关键组成部分是什么呢？其中之一是从非结构化数据转变为……problem when we're talking about uh agent memory. So at NearJ this is how we think of agent memory. We think of it as a connected graph composed of short-term, long-term and reasoning memory. Uh we'll take a look at uh this in a bit more detail. Uh but bear with me on on this idea of context graphs for agent memory. Right? So what are a few of the the key components here? Well, one is going from unstructured data to a

从消息到知识图谱From messages to a knowledge graph

知识图谱，对吧？通过这个智能体消息处理流程，对吧？用户和助手的消息都会经过某种实体抽取流程，在这个过程中，我们会识别实体是什么，以及它们之间有什么联系。嗯，我们会在一个具有强类型的图中完成这项工作，对吧？我们会用一种关系来描述这些实体之间的关联。knowledge graph, right? Going from the this process of agent messages, right? Both user and assistant messages going through some entity extraction process where we're identifying what are the entities uh and how are they connected. um doing this in a graph where we have strong types, right? We have a a relationship that describes how these entities are related.

实体解析与本体建模Entity resolution and ontology

实体解析阶段是构建这个知识图谱最重要的环节之一，对吧？理解并确保你拥有这个事物的规范表示。嗯，所以当我们谈到 Dr. Win 时，我们知道这是一位医疗服务提供者，也知道他的名字、角色等等。共享本体也是其中一个重要部分，对吧？也就是说，要对你的数据模型以及你所处理的领域进行描述。这是确保你能够从非结构化文本数据构建知识图谱的关键环节之一。对于过去接触过智能体记忆的朋友来说，这应该会让你们觉得有些熟悉。不过，我认为谈到智能体时有一点非常重要，The entity resol resolution phase is one of the most important parts of building up this this knowledge graph, right? Understanding and making sure that you have a canonical representation of the thing. Uh so when we're talking about Dr. win we know this is a provider his name the role that he has uh and so on and a shared ontology is an important piece of this right so having some description of your data model of the domain that you're working with this is one of the key pieces uh making sure that you are successful in going from that unstructured text data to uh to a knowledge graph so for folks that have that have worked with memory uh in agents in the past like this this should look a little familiar. Uh one thing though that I think is really important when we're talking about agent

推理图The reasoning graph

记忆和实际知识，指的就是推理图这个概念，对吧？那么，在我们刚才提到的智能体记忆的三个组成部分中，推理记忆是一个一等公民，对吧？我们已经讨论过短期记忆和长期记忆，它们包括消息和提取出的实体。但智能体实际采取的行动呢？它的推理过程呢？我们要确保把这些也记录下来，因为它们同样是重要的部分。我们要确保存储的是思考过程和事实，而不仅仅是数据，对吧？也就是智能体是如何做出决策的，而这通常……memory and actual knowledge is this idea of the reasoning graph right so um if we saw in our our uh sort of three components of agent memory here reasoning memory is a first class citizen right we've talked about short-term and long-term these are the messages the entities extracted but what about the actual actions that the agent takes what about the the reasoning um we want to make sure that we're capturing that because that is an important piece as well. We want to make sure that we're storing thinking the facts not just uh not just the data right so how an agent decided and this is typically

决策轨迹Decision traces

表示为决策轨迹，对吧？因此，我们代理所做的每一个决策都与基于证据的推理相关联。我们会明确地对策略进行建模，并了解该执行计划中的工具、代理调用的工具以及这些工具调用的结果。我们还希望捕捉诸如消耗了多少 token、耗时多久等信息。这些都是捕捉推理记忆的一部分。这很重要，不仅是为了让单个代理下次执行完全相同的任务时能做得更好，还要考虑这样的系统：其中有数百个或……represented as decision traces right so every decision uh that our agent makes is linked to uh the reasoning grounded in evidence right we have policies um that that we're modeling explicitly and we understand for that execution plan the tools the agent called the results of those tool calls. Uh we also want to capture things like how many tokens did this burn, the the timing that this took and so on. This is all part of uh capturing the reasoning memory. And this is important like not just so that a single agent can get better the next time you do it to do the exact same thing, but rather think about systems where you have hundreds or or

多个代理之间的共享内存Shared memory across agents

成千上万个智能体，它们可以访问一些共享的工具组，对吧？这样我们就能把这些智能体的运行记录、这些决策轨迹持久化到图中，然后与组织里的其他智能体共享，对吧？这样我们就有了一个共享的上下文图，使我们的智能体能够在这块共享记忆中彼此学习。好，这就是记忆部分。这是我们在 Near Forj 构建的一个系统，一种模式。thousands of agents that have some shared grouping of of tools that they have access to, right? So we're able to persist these agent runs, these these decision traces in the graph and then share that with other agents in our organization, right? so that we have one shared context graph uh that enables our agents to essentially learn from uh each other in this shared memory piece. Okay. So that that that's the memory piece. Like this this is um uh a system that we've built at at Near Forj, a pattern

从记忆到技能From memory to skills

很多人都在采用这种方式来处理智能体记忆。不过，可执行知识这个概念又如何呢？我们要如何让智能体采取行动？记忆的一种理解方式是：它能很好地呈现发生过的事情，对吧？我们知道谈到过哪些人、他们之间有什么关联，也有这些决策轨迹。下一步通常就是创建技能，对吧？今天有多少人会让自己的智能体使用技能？大多数人。好。有多少人自己写过技能？好。大多数人。有多少人让智能体替你写技能？对，好。我们这里要讨论的就是如何从这个记忆图谱、这个上下文图谱出发，利用它来创建有依据的……that that lots of folks are are following for working uh with agent memory. But what about this idea of of actionable knowledge, right? How do we enable our agents to to take action? Uh memory. One way to think of memory is is this is a good representation of what happened, right? We know um the people we were talking about, how they're connected. we have the these decision traces. Um the next step is typically creating skills, right? How many people are are using skills with their agents today? Mo most folks. Cool. How many people have have written skills yourself? Cool. Mo most folks. How many folks have had an agent write the skill for you? Right. Cool. Yeah. So that that's what we're we're talking about here is going from this memory graph, this context graph to how can we use that to create like grounded uh

智能体技能详解Agent skills, explained

可执行技能。所以，如果你不熟悉技能，这是 Anthropic 发布的一项开放标准，我想这个开放标准托管在 agentskills.io。基本思路是，我们有一些元数据，对吧？也就是对这个技能用途的描述。然后是渐进式披露，对吧？我还有很多更详细的信息。这些信息通常放在 Markdown 文件或参考资料中，我们可以逐步选择。如果我们沿着技能的某个部分继续深入，就可以检索并将这些数据加载到上下文中。这样就形成了一个可共享的单元，我们可以将技能打包起来，并在不同的智能体之间重复使用。executable skills. So if you're not familiar with uh with a skill, this is um an open standard anthropic um put this out uh agentskills.io IO I think is where the um open standard is hosted. And the basic idea here is that we have some uh some some metadata, right? Some description of like what this skill is about. Uh and then this progressive disclosure, right? So I I have lots of more detailed information. These are often in like markdown files, references that we can um progressively choose. If if we're going down this path of one piece of the skill, we can uh retrieve and load that data uh into context. Right? So this is this then gives us some like sharable unit uh that we can then uh take the skill, package it up uh and reuse that across different agents.

散文技能的问题The problem with prose skills

但技能也面临着一些类似的挑战，和我们之前谈到记忆时遇到的某些问题差不多，对吧？我们通常是在与专业人士合作。所以，我们仍然经常受限于这样一个挑战：弄清楚我们谈论的是否是那个规范版本，并确保我们有可调试的步骤和输出。为了解决我们在技能方面遇到的一些挑战，NeoRaj 研究团队的一些成员做了一些有意思的研究，并发表了一些……But skills have a a similar challenge to some of the issues that we saw uh previously when when we were talking about uh memory, right? is that we're we're typically working with pros. Um, and so we're still often limited in in this uh challenge of understanding is this the canonical thing that we're talking about, making sure that we have like debugable steps and and output. Um and so to address some of these challenges that uh we're seeing with skills uh some folks on the NeoRaj research team um have done some interesting research and published some

研究：将技能表示为图表Research: skills as graphs

致力于开发一种用于学习和治理智能体技能的 APE 图表示。嗯，这是论文中的一张截图。Zach 应该也在现场。如果他不在这里，今天他就在 NearJ 展位。所以，如果你对此感兴趣，一定要和 Zach 聊聊。你可以把 APE 理解为对智能体技能协议的扩展，并加入了一些额外的元数据，使技能现在可以表示为带类型的执行图。对吧？我们将步骤建模为节点，并采用非常严格的模式来规范对这些步骤的描述，以及如何使其可执行。对吧？你可以把它看作一种表示方式：将技能表示为一个拆分成多个步骤的图，每个步骤都可以执行，并且描述方式由模式规范。嗯，这就是 APE 协议，出自……的那个……work on APE uh graph representation for learning and governing agent skills. Um this is a screenshot from the the paper. Zach is uh is here somewhere. If he's not here, he's he's at um the NearJ booth today. So definitely uh chat with Zach if you're interested in this. But you can think of this uh APE is essentially an extension of the agent skills um protocol with some additional metadata that now treats your skills as typed execution graphs. Right? So we have uh steps modeled as nodes. We have uh very um very strict schema that governs how we describe that how these uh steps are actionable. Right? So think of think of this as a way of of representing um a skill as a graph broken up into steps that are executable with a schema governed description. Um that's the the ape protocol and this is from the uh the

基准测试结果Benchmark results

论文中使用的基准测试发现，没错，这确实很重要。当他们将其应用于 SkillsBench 时，发现把该协议应用于人工整理的技能后，成功完成的任务数量显著增加。所以这看起来很有意思。我们如何利用其中一些……其中一些……benchmark that was used in the the paper found that yeah like this actually matters when when they applied this to um skills bench saw like a significant increase in uh task successfully completed when looking at applying uh the a protocol to human uh curated uh skills. So that looks interesting. How can we leverage some of those uh some of those

从记忆中提炼技能Distilling skills from memory

一些想法，还有一些关于技能蒸馏的研究？呃，所以本质上，我们想做的是拿到这个记忆图谱，对吧？这个上下文图谱可能限定在某个工作区，或者某个组织中的某个项目范围内，然后我们想把它蒸馏成一项技能，但不只是一个 Markdown 文件。我们希望它以实际观察到的数据为依据。我们希望它具有确定性，对吧？我们希望图谱中包含定义清晰的流程步骤。我们还希望能够随着时间推移对它进行治理，对吧？如果构成我们技能的底层数据在记忆中发生变化，我们希望能够理解并了解这些变化。ideas, some of that research for skill distillation? Uh so essentially what what we want to do is take this this memory graph, right? This context graph that's maybe scoped to a a workspace or scoped to a a project in an organization and we want to distill that into a skill, but not just like a markdown file. We want this to be grounded in actual data that we've observed. We want this to be like deterministic, right? We want uh well understood procedural steps in our graph. And we want to be able to govern this over time, right? If if the underlying data that makes up our skill changes in memory, we want to be able to understand that and and know about that.

事实依据、覆盖度与连贯性Grounding, coverage and coherence

嗯，grounding（依据）在这里是很重要的一环，对吧？要确保构成我们技能的数据都以我们的记忆系统为依据。但光有这一点还不足以让它有用，对吧？我们还需要考察其他一些启发式指标。比如覆盖度：我们是否确保技能中的步骤和描述都以整个技能为依据？还有连贯性。连贯性很有意思。我们可以通过它来检测，进入技能蒸馏流程的信息片段（子图）是否分散在多个主题中；如果是，我们就可以建议：你可能需要在这里创建多个技能，等等。所以这就是其中一些……Um grounding is is an important piece here, right? making sure that the the data that makes up our skill is grounded in our memory system. But that that's that's not enough for it to be useful, right? There are other heruristics that we need to look at here. So, uh things like coverage, are we making sure that the uh steps and descriptions of our um throughout our skill are grounded across the skill? Coherence. Coherence is interesting. This is a way that we can detect uh maybe if the um information the piece the subgraph going into the skill distillation is split across multiple topics and we can suggest well you may want to create multiple skills here um and so on. So those are are some

技能治理Skill governance

在生成技能之前的一些环节中，嗯，我提到过，技能治理是很重要的一环。我们要确保能够理解：随着技能和数据发生变化，我们是否能够更新技能？借助动态加载，你可以设想一个受治理的技能注册表，让我们能够为任何智能体检索到最新、最及时的技能，并了解该技能是否已经过时，或者发生了偏移，对吧？of the pieces that lead up to generating the skill. Uh skill governance is an important piece that I mentioned. Making sure that we can understand uh as that skill changes uh as the data changes are are we able to update the skill? And with dynamic loading, you can think of uh having sort of like a governed skills registry that allows us to retrieve uh for any agent the the most recent up-to-date skill, understand uh if that skill may have been uh stale or or or drifted, right?

Neo4j 智能体记忆服务The Neo4j Agent Memory Service

不错。所以，我们已经在 NEFJ 智能体记忆服务中实现了这套方案，我们称之为 NAMS。这是 NearJ Labs 项目的一部分，提供智能体记忆即服务。我们还围绕它开发了开源工具，用来实现这些模式。基本上，它的工作方式是：我们有一个作用域限定在工作区内的上下文图，对吧？其中包含三种类型的智能体记忆：短期记忆、长期记忆和推理记忆。我们还有能够执行这一提炼流程的后台工作器。稍后我们会看看它具体是什么样子。然后，还有某种后台整理流程，持续负责治理工作，并在我们的技能变得过时时及时提醒我们。Cool. So, we've implemented this in uh the NEFJ agent memory service or we call NAMS. This is um an agent memory as a service as part of our NearJ labs effort. Um we also have open source tooling around that that implements these patterns. And essentially the way this works is we we have a a workspace scoped context graph, right? That has our three types of agent memory, short-term, long-term reasoning memory. Um we have uh background workers that are capable of going through this distillation process. We'll take a look at at what this looks like in a minute. Um, and then having some background curation process that is still making sure to take care of that governance to surface when uh our skills become stale

蒸馏流程The distillation pipeline

基于我们在记忆系统中的数据，这就是我们用来生成这些内容的整个流程。嗯，我就不详细介绍了。这里我们想指出的一点是，其中大多数步骤都是确定性的。我们实际上只在这里借助 LLM 来综合一些主张，并生成部分用于技能描述的文本。还有一点需要强调：第一步实际上是确定范围，也就是提炼某项技能时的初始范围是什么。它是围绕某个实体、该实体周围的子图，还是某段特定对话，诸如此类。因此，我们可以拆解技能的表示方式，并确保其中的每个组成部分都有底层数据作为依据。based on the data that we have in the memory system. Um, this is the the pipeline that that we go through to generate these. Um, the I won't go through this in in detail. The the one thing that we're trying to point out here is that most of these steps are deterministic. Um we're really only leveraging an LLM here uh for synthesizing some of the claims uh generating some of the the text that we use for some of the skill description. Uh and then the other thing I want to call out here is that the first piece is really identifying the scope like where what is your starting scope for distilling one of these skills. Is it around a certain entity? the subgraph around that is it a specific conversation uh and that sort of thing. And so we can decompose uh the the way we represent the skill and make sure that each of these components is grounded again in underlying data.

这就是我们要关注的部分：如果这项技能本质上变得过时，或者其底层数据发生变化、被删除，或彼此矛盾。And that's the piece that we're looking at for if that skill uh essentially becomes stale. If any of that underlying data changes, is removed or uh becomes uh contradictory.

组合技能Composing skills

嗯，不错。然后，正如我之前提到的，我们也可以组合技能，对吧？这就是连贯性发挥作用的地方。我们会使用社区检测等图算法，对吧？所以，如果我们把来自多个社区的主题混在一起，这可能表明我们需要拆分技能；我们会在技能治理环节发现这一点。不错。我这里还剩几分钟。Um, cool. And then we can also, as I mentioned before, we can also compose skills, right? So, this is where that coherence piece comes in. We use um graph algorithms like community detection, right? So, if we're mixing uh topics from multiple communities, that can be an indication that we need to decompose our skill and we'll catch that in um the skill governance piece. Cool. So, I've got a few minutes left here.

演示：NAMS 仪表板Demo: the NAMS dashboard

来看看它是什么样子。嗯，这是 NAMS，也就是近端智能体记忆服务。目前它是免费的，任何人都可以登录试用。嗯，这就是仪表板的样子。基本思路是，我们有 REST API 和 MCP 工具，可以提供给智能体使用，对应的功能包括摄取、检索，以及处理长期记忆、短期推理记忆。我们会进行实体提取和消歧处理。所以，我可以在这里查看智能体记忆的一些图形表示，也可以遍历其中的内容，等等。我们还可以查看被标记出来的实体。这里一个重要的部分是本体的概念。我们这里使用的是医疗本体，因此处理的数据会涉及就诊记录、医疗服务提供者之类的内容。Let's see what this looks like. Um so this is NAMS the near agent memory service. Um this is this is uh free currently. Anyone can can sign in and and try this out. Um this is what the the dashboard looks like. The the basic idea here is that we have um REST API and MCP tools that we can expose to our agents that map to um ingesting, retrieving, working with long-term, short-term reasoning memory. Um we're going through this entity extraction and resolution process. So I can uh look at some graph representation of my uh agent memory here. I I I can traverse that and and so on. Um we can look at entities that have been flagged. Um one important piece here is this idea of an ontology. So here we're using a healthcare ontology. So we we're going to be working with data about you know encounters, providers, uh that sort of thing.

呃，然后我在这里导入了大量对话。这样我们就能看到已导入的、与医疗保健智能体相关的对话，对吧？现在我们准备好进行提炼了。Uh and then I have ingested a bunch of conversations here. So we can see the conversations that have been um ingested related to a healthcare agent, right? And so now we're ready to distill

演示：提炼一项技能Demo: distilling a skill

一个技能。嗯，我们说过，第一步是确定这个技能的范围。我们可以针对整个工作区来做。不过通常我们并不想这么做。我们可以针对某个特定实体的对话，也就是本体中的一个类来做。我们就针对最近的一次对话来做吧。你会看到，这会启动蒸馏队列，然后获取数据，经过那条七阶段流水线，再将技能构建成图，最后打包好交给我们。嗯，这是其中一个。在它运行的时候，我们先来看看这个东西。这是我们之前运行的一个。它是在……上运行的a skill. Um and we said the the first piece is to decide like what is the the scope of that of that skill. We can do this for you know an entire workspace. That's often not what the case we want. We can do this around um you know a certain entity specific conversations a a class in the ontology. Let's do this around um our most recent conversation. And we're going to see this is going to kick off um in the the distillation queue and go through and fetch the data, go through that sevenstage pipeline and um construct the skill as a graph and then package that up for us. Um here's one. While this is running, let's just take a look at this guy. So, here's one uh that we ran previously. This is uh was run on a

演示：患者接诊技能Demo: a patient-intake skill

患者接诊对话。这里可以看到，我们基本上提取出了构成这项技能的各个步骤，从患者接诊一直到为患者撰写病历。我们还可以看到，这些步骤中的每一步都基于实际的工具调用，以及构成这项技能底层组件的实体。我们把这些内容打包成了一个 skill MD 文件。如果我们下载它，就会发现它还与其他参考资料等内容一起打包了，遵循的是我们在智能体技能中采用的渐进式披露标准。很好。以上就是我们如何将智能体记忆视为这个上下文图谱的一部分的简要介绍，使用的是 nearj。patient intake uh conversation. And you can see here that we've essentially extracted out the steps that make up the skill to run from a patient intake um all the way through uh charting for the patient. But we can see each one of these steps is grounded in the actual tool calls and the entities that constructed the underlying components of the skill. And we packaged this up with skill um MD file. Uh if we downloaded this, we would this would be packaged up with uh other references and and and so on. uh following that progressive disclosure uh standard that we use with agent skills. Cool. So that was a a quick look at kind of how we think about agent memory as part of this context graph uh with nearj

资源Resources

我会留下一些资源。你可以在这里获取幻灯片，这里有幻灯片链接和二维码。我提到的 nearf agent memory service 也列在这里，还有许多关于我们一些开源工具的文档和资源。就这样，我的时间到了，不过我们有一个 NearJ 展位，我也会在那里，NearJ 团队的许多其他成员也会在。到时候见。谢谢大家。and I'll leave up some uh resources. You can c grab the slides here. There's a link uh to the slides and a QR code. Um, the nearf agent memory service that I mentioned um is listed here as well as lots of documentation uh and resources for some of our open source tooling. So that's it. I'm out of time, but we have a NearJ booth uh so I will be there as well as lots of other folks from the NearJ team. So we'll see you there. Thanks folks.