本体驱动的可观测性：在 Netflix 规模下构建端到端知识图谱

文字稿Transcript

普拉萨纳·维贾亚纳坦：我叫普拉萨纳·维贾亚纳坦，是 Netflix 可观测性团队的一员。我负责 Netflix 所有会员应用的可观测性、性能和体验质量。也就是说，从你点击 Netflix 标志到开始观看内容，这整个过程中你在 Netflix 上的体验如何，都是我的工作范畴。我们应该关注哪些指标？应该使用哪些工具？应该为用户提供哪些洞察，才能确保我们了解大家喜欢什么？如果你喜欢看 Netflix，那就欢迎你继续看。如果你不喜欢，可以怪我的同事伦佐。Prasanna Vijayanathan: My name is Prasanna Vijayanathan. I'm part of the observability team at Netflix. I work on observability, performance, and quality of experience across all our member applications. What that means is from the point you click on the Netflix logo until you start watching something, how does your experience feel on Netflix is what I look into. What metrics should we look at? What tools should we have? What insights should we provide to our users to make sure that we understand what you all like? If you do like watching Netflix, you're welcome. If you don't, you can blame it on my colleague, Renzo.

Renzo Sanchez-Silva：我是 Renzo Sanchez-Silva。我在 Netflix 工作快 10 年了，一直在可观测性团队中负责监控和告警，构建了多个告警基础设施，现在则开始涉足 AIOps 领域。我以前也做过一些本体论研究。如今，随着 AI 的发展，一切都在汇聚。Renzo Sanchez-Silva: I'm Renzo Sanchez-Silva. I've been at Netflix for almost 10 years now with the monitoring and alerting team within observability, building several alerting infrastructure, and now getting into AIOps space. I've done also some research back in the day in ontology. With AI, everything is converging.

大规模可观测性——为什么这么难？Observability at Scale - Why is it Hard?

Prasanna Vijayanathan：今天我们要和大家聊聊一个我们一直在着手解决的问题。乍看之下，这似乎是一个可观测性方面的挑战，但实际上这是一个棘手的数据工程问题。在深入讨论之前，我先来铺垫一下背景。为什么大规模场景下的可观测性很难？我们说的规模到底有多大？来做个快问快答。Netflix 推出了直播活动。2024 年底，我们举办了首批大型直播活动之一：杰克·保罗对阵迈克·泰森的拳击赛。你们当中可能有人看过。你们觉得那场活动期间，我们的峰值流量是多少？肯定有数百万。实际上是 6500 万路并发流。也就是说，有 6500 万用户同时点击观看并收看直播。这就是当时的峰值流量。再问一个。你登录 Netflix 时，应用会向我们的基础设施发送请求，由我们的后端进行响应之类的。</numerusformPrasanna Vijayanathan: Today we're going to talk to you about a problem that we've been tackling, which at the surface feels like an observability challenge, but it's really a hard data engineering problem. Before I get into that, let me set the stage up. Why is observability hard at scale? What scale are we even talking about? Let's do a pop-up quiz. Netflix launched live events. End of 2024 was one of our first major big live events, a boxing match between Jake Paul and Mike Tyson. A few of you might have watched it. What do you think was our peak traffic during that event? It's definitely in the millions. It's 65 million concurrent streams. What that means is there are 65 million users clicking watch and watching the stream at the same time. That was our peak traffic. One more. When you log into Netflix, the apps are sending requests into our infrastructure for our backends to respond and stuff.

你觉得我们的应用每天会向服务器发送多少请求？我们每天处理超过 20 亿个请求。最后一个问题。面对这么多请求，我们显然有自己的可观测性系统，用来记录指标和许多其他不同的数据。你觉得我们每秒要处理多少个实时日志事件？每秒超过 3800 万个。这就是我们所说的规模。全球各地的电视、手机、笔记本电脑，以及你能想到的各种客户端平台和设备，数以千计的微服务都在产生数十亿个遥测事件。每一位用户都会接触到我们基础设施中这么多不同的部分，比如 CDN、节点等等。除此之外，我们的整个生态系统中还有数千项 A/B 测试正在运行。要在这样的规模下管理这一切。简直就是一切都在同一时间、同一地点发生。How many requests do you think we handle per day coming from our applications to our servers? We handle more than 2 billion requests a day. Last one. With all those requests happening, we obviously have our own observability systems where we log metrics, a lot of different things. How many real-time logging events do you think we have to handle per second? It's upward of 38 million events per second. That is the scale we're talking about. Numerous client platforms and devices across all the TVs all over the world, mobile phones, laptops, you name it. Thousands of microservices emitting all these billions of telemetry events. Every single member touches so many different parts of our infrastructure, our CDNs, nodes, and so on. On top of all that, we have thousands of experiments running through A/B tests throughout our ecosystem. Managing all of that at scale. It's everything, everywhere, all at once.

端到端可观测性——这是什么？End-to-End Observability - What is It?

即使达到这样的规模，Netflix 的目标依然是每时每刻都为每位用户提供尽可能出色的体验。我们在可观测性方面的策略很简单：我们是否拥有合适的工具、指标和洞察，能够帮助我们做到这一点？策略很简单，但实现起来并不容易。在这样的规模下，将用户体验与系统性能关联起来，是一个本质上截然不同的挑战。这已经不再是工具方面的挑战，而是数据工程方面的挑战。传统上，可观测性系统一直是监控和告警工具，用于观测应用和服务。出了问题，系统就会发出警报。然后，你会查看具体情况，将问题分派给合适的团队，必要时寻找合适的开发人员来修复，部署修复方案，再确认数据是否恢复到预期状态。传统的可观测性系统就是这样运作的。借助端到端可观测性，我们希望重新思考如何实现可观测性，将其从被动的监控体验转变为主动的洞察引擎。Even at that scale, our goal at Netflix is to deliver the best possible experience to every user every single time. Our strategy towards observability is simple. Do we have the right tools, metrics, and insights to enable us to do that? The strategy is simple. It is not easy. At this scale, correlating user experience with system performance is a fundamentally different challenge. It is not a tooling challenge anymore. It is a data engineering challenge. Traditionally, observability systems have been monitoring and alerting tools where you observe your apps and services. Something goes wrong, you get flagged. Then you look at what's going on, triage it to the right teams, look for the right developers to fix it if needed, roll out a fix, and then see if the data comes back to where it is expected to be. That's how traditionally observability systems have worked. With end-to-end observability, we wanted to rethink how we do observability, and change it from a reactive monitoring experience to a proactive insights engine.

在向你展示我们如何实现这一点之前，我想先谈谈我们构想的愿景。想象一下，有这样一个系统，能够在几分钟内自动检测整个技术栈中的问题。这不只是某一项服务或某一个应用，而是覆盖整个技术栈。无论问题发生在哪里，你都能知道正在发生什么。接下来，系统会根据问题对用户造成的影响，自动确定优先级。它还可以自动将问题分派给合适的团队。你不必盯着某个应用和某个错误说：“我想沿着这条线索，在大海捞针般的排查中追踪下去，找出不同的服务，再把它们都拉进来。”那么，我们能否自动定位根本原因？不只是分诊，而是能否真正找出问题的根本原因？或者，更进一步，能否在问题真正开始影响用户之前就预测到它，甚至开始建议或采取纠正措施？Before I show you how we do that, I want to talk about the vision that we created. Imagine a system that can detect issues automatically in a few minutes across the entire stack. It's not just one service or app, but across your stack. Wherever an issue happens, you know that this is what is happening. Then you go into automatically prioritizing those issues based on how it's impacting our users. You also can triage it to the right team automatically. You don't have to look at one app and one error and say, I think I'm going to follow this needle through this haystack and pick out different services and loop them in. Then, can we automatically root cause? Not just triage, but can we actually identify the root cause of the issue? Or, even better, can we predict issues even before they would actually start impacting our users and maybe even start suggesting or taking corrective actions?

这有可能实现吗？我们希望做到这一点。构建这个系统，就能让我们无缝地追踪从用户设备、经过我们的网络和网关服务，一直到后端依赖项和服务的深层链路。所有环节彼此连接，全部实时运行。用户在 Netflix 上的体验不会受到任何干扰。每次观看 Netflix，你都会乐在其中。每一次，你的体验都会无比出色。这就是我们的愿景。Is this possible? We wanted to do this. Building the system would give us seamless visibility from the user's device through our networks into our gateway services into the deep trenches of our backend dependencies, services. All connected, all in real time. Users would have no disruptions to their experience on Netflix. Every time you watch Netflix, you will love it. Your experience is going to be awesome, every single time. This is our vision.

听起来很棒，但这与现实相去甚远。如今的情况大不一样。我来带你回顾一下最近一次调查的时间线，大概是过去一年左右发生的事。大约在 3:30，我们的一个客户平台触发了一条告警。通常，值班人员会看到告警，然后想：“我先等一会儿，看看情况会不会持续。说不定会发生什么奇迹，它自己就消失了。”有时确实会，有时不会。几个小时后，他们才真正开始查看：“我觉得这确实是个问题。我得去查清楚，顺着这条线索追查，看看是不是还有其他服务受到影响。”一般来说，通常确实是这样。随后，问题会被分派给合适的团队。结果发现，那个团队当天早在 1:30 左右就已经在排查这个问题了，大家早就开始处理了。Sounds great. That's far from reality. It's very different today. I'll walk you through a timeline of a recent investigation, probably within the last year or so. There was an alert that got triggered on one of our client platforms, I'd say around 3:30. Typically, on-calls look at it and they're like, let me wait for a while. Let me see if this is going to sustain. Maybe something happens magically and it'll go away on its own. Sometimes it does. Sometimes it doesn't. A couple of hours later, they're actually looking at it, "I think it's a real issue. I need to go figure out, follow this needle and see if there's another service that's impacting." In the average case, usually it does. Then it's triaged to the right team. Then it turns out that team was already debugging this issue from like 1:30 that day. People are already on it.

现在有多个团队参与其中。这些团队之间需要进行更多调试。调试开始后，他们会找出问题的根因，顺着线索追踪到真正有问题的服务。在这个案例中，事实证明有个服务认为自己没那么重要，于是削减了自身容量，导致了反向传播错误。那时大约是 6:30，距离客户端应用实际发现真实用户受到影响已经过去了三个小时。我们推出了修复方案，并在一小时内解决了问题。 这是一个平均水平的案例。听起来不算太糟，也就四个小时，还可以。但考虑一下为此付出的代价：期间有九个团队收到告警，三十多名工程师以不同方式参与了整个调试过程。此外，还有另外三个相关事件也被标记出来。各个团队都陷入了恐慌。 这通常就是这样的情况，而且这只是一个平均案例，甚至算不上糟糕的例子。把这四个小时以及投入的所有资源加在一起，就会让人觉得非常浪费。这正是你想通过端到端可观测性解决的问题。Then, multiple teams are involved now. There's a lot more debugging happening between all these different teams. Then once the debugging starts, they root cause the actual issue, follow the needle down to the actual offending service. In this case, it turned out there was some service that thought they're not as important and cut down their capacities. That caused backward propagation errors. That's around like 6:30, three hours since the client application actually noticed real user impact. We rolled out a fix. It was resolved within the hour. This is an average case. It doesn't sound too bad. It's like four hours. Not too bad. Then when you consider what it took, during this time, there are nine teams that were paged. More than 30 engineers were involved in the entire debugging process in different capacities. Then three other related incidents that were also flagged. There's panic everywhere across teams. This is typically, and this is an average case. This is not even a bad example. Then when you put together the four hours and all the resources that get involved, it just feels wasteful. This is what you want to solve with end-to-end observability.

我们研究了原因：为什么会这样？我们总结出了四个主要主题。第一，数据源过于分散。日志、指标、事件和追踪数据大致存储在不同的数据存储中，形式各异。它们彼此孤立，无法互通，也没有统一标准，结构各不相同，诸如此类。这会导致告警彼此割裂、缺乏上下文：每个团队都会在自己的孤岛中选择对自己最有效的数据源，然后设置告警。告警触发时，各团队针对同一个问题，却从不同角度进行分析。他们本该彼此沟通，但实际并非如此。这让团队在问题分诊和实际排查故障时变得更加复杂。归根结底，我们实际上无法检测问题对用户造成的影响。如果你负责的是距离用户有五跳、十跳的下游服务，你几乎永远无法了解自己的系统和服务如何影响最终用户。We looked at why. Why does this happen? We came up with four main themes that existed. One, you've got too many siloed data sources. We got roughly logs, metrics, events, and traces that are stored in different forms in different datastores. They are in silos. They don't talk to each other. They're not standardized. Structures are different, and so on. That leads to disconnected and non-contextual alerting, which means each team follows a data source that works well for them in their silo and then set up alerts. Then when alerts trigger, they are looking at the same problem in one way compared to another team that's looking at the same problem from a different perspective. They should be talking to each other, but that doesn't happen. What that leads to is complexity in how teams triage and actually troubleshoot the issue. Ultimately, what that means is we actually don't detect what is the impact for our users. If you're a downstream service, 5 hops, 10 hops away from the user, you almost never understand how your system and service is affecting the end user.

互联互通——弥合差距，打破孤岛Connectedness - Bridging Gaps, Breaking Silos

鉴于我们对端到端可观测性的愿景，我们该如何解决这个问题，从现实状态迈向愿景？用一个词来回答，就是互联互通。我们当时就是这么想的。这就是贯穿始终的共同主题。我们希望数据、告警、调试系统、工作流和根因分析彼此互通。如果我们将其作为端到端可观测性的核心主题，就不需要让 10 种不同的数据源催生出 6 种不同的工具。我们希望构建一个集成式可观测性层，用一种抽象来呈现系统可观测性的方方面面。这意味着，我们的仪表盘、告警、调试工作流和事件响应工具都使用同一种语言。这样一来，针对系统中相同问题和相同症状的重复调试与分诊工作就会减少；你也就可以沿着线索一路追踪，从用户设备一直追溯到真正的根因。Given the vision that we saw, that we have for end-to-end observability, how do we solve this and move from the state of reality to the vision? The one-word answer for that is connectedness. That's what we thought. The common theme is that. We want connected data, connected alerting, connected debugging systems, workflows, and root cause analysis. If we create that theme for end-to-end observability, we don't need 10 different data sources giving rise to 6 different tools. We want to create an integrated observability layer. One abstraction that can tell you everything about our system's observability. What that means, then, is our dashboards, alerts, debugging workflows, and incident response tools, they all speak the same language. Then duplicating efforts in debugging and triaging, the same issues, and the same symptoms across the system is reduced, which means you can follow a breadcrumb trail, and then get into the same path from the user's device to the actual root cause.

你可以从不同的线索顺藤摸瓜，但最终要找到同一条主线，确定问题的根本原因。这样可以缩短问题解决所需的时间，最终提高诊断的准确性，让用户受到的干扰更少，体验更好。这一切的核心始于数据，也就是互联的数据。用户、设备和应用，以及服务和基础设施，都会产生大致分为四类的数据：指标、事件、日志和追踪。我们将它们统一起来，放入同一个层，即 MELT 层。统一所有这些数据源之后，问题也就迎刃而解了。重要的是，即使这些数据都集中在一起，我们也必须确保它们准确且可采取行动。You can follow the breadcrumb from different trails, but get on the same main path of root causing the issue. That leads to reducing the time it takes to resolve issues and ultimately will improve the accuracy in diagnostics, and give our users a less disrupted and better experience. The core piece of all this starts with the data, connected data. We got our users, our devices and applications, and our services and infrastructure, all emitting these broadly four different categories of data: metrics, events, logs, and traces. We unify them and put them in one layer, the MELT layer. Once we unify all these sources, we solve the problem. It's important that even if all of this exists together, we need them to be accurate and actionable.

让我再带你了解另一个场景。假设我们已经有了这一层。假设我们发现了一个回归问题：LOLOMO TTR。LOLOMO 是 Netflix 上电影列表的列表的指标。TTR 则是这个电影列表的列表的某种渲染时间。你发现这里出现了回归。有一个缓慢的泄漏，还有一个明显的回归。开发人员注意到了这一点，于是向我们的统一数据层询问：发生了什么？现在所有可观测性数据都存放在同一层里，所以告诉我发生了什么。然后，指标系统会告诉你：是的，我发现 LOLOMO TTR 正在回归。我还发现，另外 5 个不同服务中的 12 项指标与此相关，并且在同一时间发生变化。它们也在回归。我们找到了一些相关性，这很好。接着，事件数据会告诉你，在这次回归开始之前，这些不同应用和服务中还发生了 47 个事件。是的，我可以在图表上看到。Let me walk you through another scenario. Let's say we have this layer already. Let's say there is a regression we see, LOLOMO TTR. LOLOMO is the metrics of the list-of-list-of-movies that you see on Netflix. TTR, some rendering time for that list-of-list-of-movies. You see a regression there. There's a slow leak, and there's a big regression. Developer notices this and they ask our unified data layer, what happened? Now all the observability data lives in one layer, so tell me what happened. Then the metrics system would tell you, yes, I see LOLOMO TTR regressing. I also see 12 other metrics across 5 different services correlating, happening at the same time. They are also regressing. We find some correlations. That's good. Then events could tell you there are 47 events across all these different apps and services that also happened right before this regression started. Yes, I can see it on a graph.

我可以看到所有这些事件以及指标。然后日志会告诉我，我觉得当时这 7 个服务中的某一个出现了连接池耗尽错误，而且错误数量突然激增。也许问题就在这里。同时，其他所有这些服务也都在同一时间出现了超时错误。接着，链路追踪可能会告诉你另一个故事。关于我们在链路追踪方面所做的工作，我会另开一场专门的分享。简单来说，一条追踪记录会展示某个特定请求和响应从用户到所有处理过它的服务所经过的路径。你可以沿着这条路径进行追踪。追踪的难点在于，除非你确切知道要找哪条追踪记录，否则很难用它准确地排查问题。这甚至不是大海捞针，而是像蒙着眼睛在黑暗中寻找森林里的一片树叶，实在太难了。I can see all these events along with the metrics. Then logs tells me, I think there is a connection pool exhaustion error that spiked up at this time in one of those 7 services. Maybe that's what it is. There are also all these timeout errors in all these other services at the same time too. Then traces can tell you a different story. If I'm going to talk about the work we've done with traces, that will be a different session on its own. To give you an idea, a trace gives you a path for a particular request and response from the user to all the services that touched it. You can trace that path. The challenge with tracing is, unless you know exactly what trace you're looking for, it is super hard to use it to debug an issue accurately. It's not even a needle in a haystack. It's like looking for a leaf in a forest in the dark and you're blindfolded. It's hard.

本体——关系，而不只是连接Ontology - Relationships, Not Just Connections

我们真正缺少的，不只是连通性，还有这个拼图中所有不同部分之间的关系。我们确实需要了解这些组件之间的连接，但也想知道它们如何相互作用、彼此之间又有什么关联。我们称之为可观测性本体。Renzo 将带你了解我们如何构建这些关系，以及如何在日常工作中加以应用。What we are really missing is not just connectedness, but the relationships between all these different pieces of this puzzle. We need to know the connections from these components, yes, but we also want to know how they interact with each other and how they are related to each other. This we call the observability ontology. Renzo will walk you through how we build these relationships and how we're putting this to use every day.

Renzo Sanchez-Silva：什么是本体？它只是一种对类型、属性和关系的形式化规范。它是我们编码含义的方式。它的基本构造就是所谓的三元组。三元组由主语、谓语和宾语组成，每个三元组表达一个事实。比如，一个人是员工；一项服务很重要，或者属于 0 级服务。它也可以表达属性。这就是我们编码知识的方式。其实，这非常基础，甚至可以说是原始的。就像说“泰山爱简”。“妈妈爱爸爸。”婴儿脑中也有微型本体。我们只是开始在脑海中收集这些符号三元组，并据此提出命题、进行推理等等。本体是一种编码知识的方式。我们使用 W3C 标准，包括 RDF、OWL（Web 本体语言）、SPARQL（查询语言）和 SHACL（形状约束语言）。借助这些标准，我们可以声明某项服务可以在 AWS 中终止；但还有一项约束是，如果该服务的容量超过或低于 10%，就不能终止。Renzo Sanchez-Silva: What is an ontology? It's just a formal specification of types, properties, and relationships. It's how we encode meaning. The basic construct of it is what you call the triple. The triple is a subject, a predicate, and an object, one fact per triple. It can be, a person is an employee. A service is important, or tier-0, for example. It can be properties too. This is the way we encode knowledge. It's very basic, very primordial, actually. It's like saying Tarzan loves Jane. Mommy loves Daddy. Babies have mini ontologies in their head. We just start collecting all these symbolic triples in our heads, and we start making propositions and reasoning and so forth. Ontologies are a way to encode knowledge. We're using W3C standards, RDF, OWL, web ontology language, SPARQL, the query language, SHACL is shapes constraint language, where we can say a service can be terminated in AWS, but then another constraint is if the service has more or less than 10% of capacity, you cannot terminate.

你可以对所有这些内容进行编码。你可以描述你的服务和整个基础设施。比如，我们有 API 网关，它属于应用类型，由某个团队所有，部署在某个区域，等等。我们可以描述基础设施中的每一个部分，而且每个部分也都有自己的命名空间。我们通过描述事件，开始构建这个小型本体。我们已经确定了几个命名空间，正在构建一个运维本体。再回到 Prasanna 的愿景，我们的目标是开始找出这张端到端的可观测性图谱，呈现基础设施中所有相互关联的事物。我们就称之为运维本体。比如，我们可以确定这些命名空间，并在三元组中正式引用它们。You can encode all these things. You can describe your services, the whole infrastructure. For example, we have API gateway. It's of type application. It's owned by a team. It is deployed in a region, and so forth. We can go and describe every single piece of the infrastructure, and every piece there is namespaced as well. We started building this baby ontology by describing incidents. We have identified a few namespaces. We're building an operational ontology. The goal, again, going back to Prasanna's vision, is to start finding this end-to-end observability graph of all the things connected in our infra. Let's call that the operational ontology. For instance, we identify these namespaces, and they can be referenced formally in the triples.

问题在于，如何理顺这种运维混乱：事故散落在 Slack、PagerDuty、MELT 层、指标、追踪、日志和事件中。我们所有人都聚到同一个频道里，里面有数百条消息，涉及许多服务，大家彼此交流。在找到根因之前，场面一片混乱，而等到问题修复时，往往已经过去了好几个小时。我们想做的是观察这一切，并开始实时地对其进行编码，捕捉其中的结构，以及我们如何用机器可读的类型来保存这些信息。一条消息显然可以是英文的。接着有人说：“这次部署之后，API 网关的延迟飙升了。”诸如此类，然后 Jane 正在回滚。之后就可以把这些内容描述成三元组。这就是这样做的思路。手动完成这件事很难。事实上，在过去还没有 AI 的时候，你需要一位本体工程师一直做标注：这是那个，这是一个人，这是一个服务，等等。The problem is how to bring order to this operational chaos, in incidents scattered across Slack, PagerDuty, the MELT layer, metrics, traces, logs, and events. We all gather in the channel together. There are hundreds of messages, many services, people interacting. It's crazy until we find root cause, and then we fix the issue many hours later. What we want to do is observe that, start encoding in real time. Capturing the structures and how we preserve this in machine-readable types. A message can be, obviously, in English. Then someone says, API gateway has latency spiked after this deployment and so forth, and then Jane is rolling back. Then you can describe that in triples. That's the idea of doing this. Doing this by hand is hard. In fact, back in the day when there was no AI, you needed to have an ontologist who was always annotating, this is that, this is a person, and this is a service, and so forth.

借助 AI，我们可以快得多地完成这件事。为此，我们创建了一套称为“采集流水线”的流程。事件发生在 Slack 上，那里才是真正开展工作的地方。对于每个频道，我们都会转换其中的每条消息，并将每条消息发送给一系列增强器。我们会调用可观测性 API，并在此基础上使用 LLM 来补充一些分类信息，然后将这些信息关联起来。我们把结果保存到图数据库中，并将其称为 QuipuDB。标成绿色的增强器非常确定。我们想要识别——例如，这是 incident.io 上的内容，我们用它作为事件管理软件——它对应的 ID。比如，我们会提取已触发的 PagerDuty 告警。我们还使用各种非常确定的小型算法来分类，因为例如我们能识别 URL，所以可以提取 ID。我们也会提取例如讨论串或消息中所反映的情感倾向。With AI, we can do this much faster. For this, we created what we call a harvest pipeline. The incidents occur in Slack. That's where the action happens. For each channel, we transform every message, and we send every message to a bunch of enrichers. We call observability APIs. We use LLMs on top to find some extra classification, we correlate that. We save that in our graph database, we call it QuipuDB. The enrichers in green are very deterministic. We want to identify — for example, this is on incident.io, we use them as our incident management software — it's of this ID. We have the PagerDuty alert that was fired, for example, we extract that. We have various very deterministic, small algorithms that classify things because we know URLs, for example, we extract IDs. We extract, for example, also the sentiment of what is happening in the thread or the message.

蓝色部分表示告警以及正在发生的时间线。人们会分享指向我们仪表板的链接，我们会对这些内容进行编码，并知道它们的含义。有人上传快照或屏幕截图时，我们会将其保存下来。这是一张与这张图表、这个指标相关的屏幕截图。 然后，在这一流程的最后，我们也会使用 LLM。此时，我们已经识别出所有这些基本模式或实体，并开始寻找新的关系、概念或定义。你可以看到，LLM 能发现一些很有价值的内容，比如这是金丝雀分析、部署失败之类的。它会从目前收集到的内容中开始发现概念。如果发现了新概念，我们就会尝试提出新的定义。这在本体论中称为开放世界假设：你可以在一定程度上自由地添加关系，而我们也有办法对其进行控制。这是一个受到严格控制的流程，按顺序进行。有时如果某些环节失败了，我们就会暂时搁置。例如，如果我们无法获取某些数据，就会先把它留在那里。然后，我们会将其与主要的元数据来源进行核对。比如，软件目录中包含了我们所有服务的定义。我们会识别每项服务，了解它们属于哪种类型，等等。整个过程受到严格控制。In blue, we have alerts and the timeline that is occurring. People are sharing links to our dashboard. We encode that. We know what that means. Someone uploads a snapshot or a screenshot, we save that. It's a screenshot related to this graph, this metric. Then we use also LLMs at the end of this process where we have identified all these basic patterns or entities, and start finding new relationships or concepts, or new definitions. The LLM, you can see great things like this has canary analysis, deploy failed, and things like that. It starts finding concepts from what has been currently harvested. If it's new, we'll try to propose a new definition. This is what it's called in ontology, the open world assumption, where you are somewhat free to add relationships and we have ways to control that. It's a very controlled process. This is sequential. Sometimes if something fails, we just leave it. For example, we cannot grab some data, for example, you would leave it there. Then we check against our main metadata sources. For example, the software catalog is where we have the definition of all our services. We identify every service, and we know which type they are, and so forth. It's very controlled.

目标是从非确定性中创造确定性。非确定性首先来自渠道中的混乱，这是其中一个来源。接下来，我们有一系列规则来提取数据。最后，锦上添花的是 LLM 登场并指出，这可能是一种新的关系，这里存在新的关联，诸如此类。这里的悖论在于，我们使用随机性工具来生成可通过 SPARQL 查询的确定性图谱。本体就是这座桥梁，也是我们保存所有数据的地方。我们确定了五个层次。这是一个引导过程。确定性提取是用绿色标出的那些步骤。例如，我们有一系列正则表达式，用来按 ID 和服务器提取 Jenkins 作业。这是一种实体。这些都是非常成熟的模式。然后是目录验证，也就是将服务编入目录，或检查该服务是否是在这个区域部署的有效服务，等等。The goal is to create determinism from non-determinism. The non-determinism starts with the chaos in the channel, for example. That is the first source of non-determinism. Then we have a bunch of rules where we extract the data. Then, at the very end, the icing on the cake is the LLM coming and saying, this is probably a new relationship. There's a new connection here, and so forth. The paradox here is that we use stochastic tools to produce deterministic SPARQL queryable graph. The ontology is the bridge. It's where we save all the data. We identify five layers. This is a bootstrapping process. Deterministic extraction is those steps in green. We have a bunch of regexes, for example, that extract what a Jenkins job is by ID, by server. That's one entity. Those are very well-known patterns. Then the catalog validation is cataloging or checking against the service is a valid service deployed in this region and so forth.

我们使用自己的 API 和可观测性 API 丰富数据，然后将其交给 LLM 进行分类并查找新的谓词。我们还有最后一个步骤，称为概率关联：在整个对话中进行首次搜索。我们会找出这样的链路：事件始于应用程序 A，影响了应用程序 B；随后原始应用程序被回滚，问题也因此得到解决。这条链路也会被识别为一个实体，也就是一种模式，并保存下来，以便将来我们可以寻找相似之处，将这起事件与未来发生的事件进行比对。 此外，所有内容都受到约束。我们有类别和一些分类体系，它们都有明确的约束。当 LLM 开始工作时，它需要将内容归入某个类别，例如判断这段对话属于事件、问题、讨论还是决策。它只能从这四个值中选择。我们不会允许它凭空编造新类别；除非它告知我们有这种情况，之后我们再看看如何处理。所有内容都经过类型化，例如，thread 123 是相关项，true 则是布尔值。We enrich with our APIs, observability APIs, and then we pass the LLM to classify and find new predicates. We also have a final step, what we call the probabilistic correlation, where we do the first search in all the conversation. We find chains where the incident started with this one application that affected application B, the original application was rolled back, and then the issue was fixed. That chain is identified also as an entity, a pattern that will be saved so that in the future, maybe we will find similarities, this incident with something in the future. Also, everything is constrained. We have categories, some taxonomies, and they are all constrained. When the LLM comes in, it wants to classify some category, say whether this conversation is an incident, question, discussion, or decision. It can only choose from those four values. We don't let it hallucinate new things unless maybe it tells us that, and then we'll see how we can manage that. Everything is typed through, for example, thread 123 is related, true is a Boolean.

我们所描述的是一个自举过程，也就是我们所说的知识飞轮。每次转动时，比如从一个线程开始，我们就开始识别实体。随着更多数据不断涌入，我们通过观察、丰富、推断和适应，沿着这个飞轮运转。我们观察 MELT 数据，借助所有这些丰富器来补充信息，推断可能的新模式，或应用已有模式，并通过适应来了解我们能获取哪些知识。一个事件中的因果链，会成为下一个事件的已知模式。观察、丰富、推断，这就是主要流程。在每次迭代中，我们都会编码知识。每个周期都会教会系统本体如何将这一模式演化为记忆。我们还有一个过程：数据转化为信息，再将信息编码为知识。第四个阶段是智慧，这里的智慧是指我们能够找到根本原因。我们今天可以将其编码，也能在未来做到。The process we're describing is a bootstrapping process, what we call the knowledge flywheel, where every rotation, say a thread starts. We start identifying entities. More data comes in, we go around this flywheel by observing, enriching, inferring, and adapting. We observe the MELT data. We enrich with all these enrichers. We infer possibly new patterns or old patterns just to apply them, and we adapt to see what knowledge we can gather. The causal chain from one incident becomes a known pattern for the next. Observe, enrich, infer, that's the main process. In every iteration, we encode knowledge. Each cycle teaches the system, how the ontology will evolve this schema into memory. We have a process also where data is converting information, that information is encoded in knowledge. The fourth is wisdom, wisdom in the sense that we can find root cause. We can encode that today, but also in the future.

我们现在还没有在这里进行预测，只是在持续地描述。先描述。这正是婴儿会做的事。爸爸爱妈妈，他知道这是事实，但之后还会发生更多事情。一切都可以通过 SPARQL 查询。SPARQL 的语法很奇怪，但你会慢慢习惯。大型语言模型很喜欢它。我们不在乎。有时我甚至都不读它。再说一次，本体会累积不断发生的事情的结构。模式会变成可查询的事实。We're not predicting here yet. We're just describing always. Describing first. That's what a baby does. Daddy loves mommy, he knows that is a fact, but then more things will happen later. Everything is queryable via SPARQL. SPARQL is a weird syntax, but you get used to it. LLMs love it. We don't care. I don't even read it sometimes. Again, the ontology accumulates the structure of what keeps happening. Patterns become queryable facts.

我们用 AI 来做这件事并不奇怪。我们把 Claude 当作协同开发者。每当收到一个新的讨论串，或者事故中出现新消息时，我们就会运行这个采集器。它会创建一个 git worktree，因为我们不想和其他分支发生冲突——毕竟同时发生的事故很多，也可能马上又有新一轮迭代；之前的采集任务可能还在处理某些内容，尚未完成。它会创建一个 git worktree 或分支。然后我们也会用 Claude 来进行采集。我们不只是运行一个脚本，而是让 Claude 配合采集脚本运行，以提取数据，因为其中有一个 LLM 步骤，用来发现新的模式。同时，我们还会用它来查找这个脚本本身可能存在的漏洞，让它能够自行修复。接着，假设整个循环结束了，我们最终会发现，仅这起事故的一系列迭代就产生了 15 个分支，然后人工介入。It's not surprising that we use AI with this. We use Claude as our co-developer. When we get a new thread or a new message in an incident, we run this harvester. It creates a git worktree because we don't want to conflict with other branches, because there are many incidents happening, or maybe there's another iteration coming, the previous harvest can be working still on something and then it doesn't finish yet. It creates a git worktree or a branch. Then we use Claude also to harvest. We are not just running a script. We are running Claude with the harvesting script to extract the data because there's the LLM step that finds new patterns. At the same time, we use it to find maybe bugs in this very same script, so that it actually fixes itself. Then, say, that whole loop ends, and then we ended up with 15 branches only for this series of iterations, for one incident, and then human in the loop comes in.

我或者 Prasanna 都可以进来，说：“我们有这么多分支。”然后我们让 Claude 挑选提交，看看发生了什么。它会发现一些很有意思的东西，也会发现新的事实。有时它会说，这五个分支做的几乎是同一件事，它们发现了这个模式。另一个分支发现自己运行在 Sonnet 4.5 上，于是决定把自己升级到 4.6，这很酷。同时它也能自我修复。你说：“好，合并吧。”然后继续推进。这是一个元循环。飞轮产生知识，第二个飞轮改进这套机制，目前还要靠我们参与。我们的想法是让它不再依赖人类，并能自行合并。然后我们继续在上面迭代，层层叠加许多飞轮，形成某种金字塔。直到它完全自主，然后我们再转向其他领域或命名空间。这就是我们在 Slack 里设想的方式，因为 Slack 用起来很方便。Either me or Prasanna can come in and say, we got all these branches, and then we ask Claude to cherry pick and see what happened. It finds very interesting things. It finds new facts. Some, it says, these five branches did almost the same. They found this pattern. This other branch realized it was running on Sonnet 4.5, and it decided to upgrade itself to 4.6, which is cool. It's self-healing at the same time. You say, ok, merge. Then you keep going. It's a meta loop. The flywheel produces the knowledge, and the second one improves the machinery, and uses us for now. The idea is actually to make that non-dependent on humans and merge by itself. Then we continue iterating on top, so many flywheels on top, kind of a pyramid. Until it's completely autonomous, and then we move on to other domains or namespaces. This is how we see that in Slack, because Slack is convenient.

它看起来是这样的。例如，每次迭代时，我们都会找到几个频道。这次找到的恰好是一个平台运维频道、一个指标/分析工程频道；它之前已经发现过所有这些分类。我从未告诉它这些频道是什么，但我接受了它的判断。我看了看，也抽查了一下。这是一个告警管理频道，不是事件频道。这些都是通用频道。我们也在监控所有 Slack 频道。因为记住，我们的想法不只是从一个小型本体开始，再扩展到基础设施中更大的部分并对其进行描述。这只是针对通用频道的。这就是一次迭代结束时的情况。例如，在这次运行中，它找到了一些 Turtle。Turtle 是我们用本体语言序列化 RDF 的方式。它发现正则表达式模式存在一些问题，而且 URI 没有被正确保存为有效的 Turtle。This is how it looks. For example, at each iteration, we find a few channels, and then, this happened to be a platform operations channel, a metrics/analytics engineering, all these classifications are something that it has found before. I never told it what it was, but I accepted it. I looked and spot checked. This is an alert management channel. It's not incident channels. These are general channels. We're also watching all the Slack channels as well. Because, remember, the idea is not only start with a small ontology and then go up into bigger parts of the infrastructure to describe them. This is just for general channels. This is what happens at the end of one iteration. For example, in this run it found some Turtle. Turtle is the way we serialize RDF in ontology language. It found some issues with the regex patterns, and the URIs were not saving correctly to valid Turtle.

它发现了一个 diff 之类的东西。这时我可以继续说：“你可以合并这个”，然后它就会去执行。所有测试都通过了。合并已完成且成功。让我来创建一份摘要。这意味着，到了下一轮迭代时，这段代码已经在主分支中了。下一轮迭代里，有人会在频道中添加一条新消息，或者有新的发现。有人找到了一个根本原因，实际上会运行这段新代码。下一个启动的 git worktree 就会已经合并了这个修复。It found a diff and so forth. That's what I can continue and say, you can merge this, and it goes and does it. All tests passed. The merge is complete and successful. Let me create a summary. That means for the next iteration, this code is already in the main branch. The next iteration, someone is adding a new message in the channel or a new discovery. Someone found a root cause that will actually run this new code. The next git worktree that spins up, it will already merge this fix.

我刚才提到的提取这些链的部分，是我们关联引擎的一部分。它会找出所有这些链。比如，某次提交采用了重试逻辑，并随某个版本部署。三分钟后出现了连接超时错误，触发了一条告警，随后创建了事件。然后，LLM 会给出自己的置信度，但我们会保存这条链。一次事件中可能会出现很多条链。想象一下，把所有这些链放在一起，未来我们就会积累大量数据，用于微调和相似性分析。智能体实际上是熵减器。因为正如我所说，我们接收到的是充满噪声的输入。最后，我们会输出三元组，并建立新的关系；与此同时，我们也在寻找更好的数据提取方式，因为智能体循环正在不断运行。比如，我们会进行 SME（领域专家）检测，判断某个人是否是领域专家，并找出其专长主题。The part that extracts these chains that I talked about, it's part of our correlation engine. It finds all these chains. For example, commit happened with retry logic. It was deployed with this version. Three minutes later there was an error, connection timeout, an alert fired, and the incident was started. Then the LLM comes up with its own confidence, but we save this chain. Many chains can happen in one incident. Imagine all these chains together, we'll have a lot of data in the future for fine-tuning and similarity. Agents are actually entropy reducers. Because, like I said, we receive noisy input. We spit out triples at the end, with new relationships, but at the same time, we find better ways of extracting data, because the agentic loop is happening. We do things, for example, like SME, Subject Matter Expert detection. This person is actually a Subject Matter Expert, and we find the topic too.

这些三元组不断累积，最终就能告诉我们，某个人是服务系统、Kafka 等方面的专家。我们也有以 Turtle 格式表示的服务拓扑，这些拓扑早已存在，因为我们从零开始，利用 eBPF、日志和追踪数据等许多来源构建了它们。我们创建了一个图谱，其中包含数千项服务依赖关系，记录了服务之间的调用关系。这只是其中一部分。 这套约定是针对增强器的。我们有一个类，它是一个接口，可以在其中看到增强部分；你可以向图谱添加三元组，等等。我们有很多这样的增强器，随着更多数据不断涌入，还会有更多增强器加入。Claude 实际上会告诉我们需要构建一个新的增强器，因为我们发现了某种新事物，而我无法从以往的历史记录中了解它，等等。代码里没有，本体里也没有。我们需要创建它。实体中的每个 URI 都是稳定的。我们有域和 ID，始终可以通过命名空间引用它们。The accumulation of those triples will tell us eventually, this person is an expert in the service systems, or in Kafka and so forth. We also have these service topologies on a Turtle that already exists, because we build it from the ground up from many sources, using eBPF and logs and tracing. We created this graph of all our thousands of dependencies of services, which service calls each other. That's part of it. The contract is for the enrichers, we have a class, which is an interface where you can see the enrichment part, you can add triples to the graph and so forth. We have many of these, and more enrichers come as more data come in. Claude will actually tell us we need to build a new enricher because we're seeing this new thing that I don't know about from previous history and so forth. It's not in the code. It's not in the ontology. We need to create. Every URI in the entities are stable. We have a domain and IDs. We can always reference them by namespace.

我们从图谱进入知识库。每次采集器运行时，都会将 Turtle 文件保存到 S3。我们会创建一个 Parquet 文件。我们的图数据库会持续轮询这个数据流。实际上，这是一个 CDC（变更数据捕获）数据流，会将那些三元组加载到内存中。这样我们就能实时查询，因为此时数据已经在库里了。比如，我们已经描述了这起事件。其他人可以通过 Claude Code 会话、Cursor 或其他工具，查询事件 123 的情况。我们已经有这些数据了。LLM 会发起 SPARQL 查询并开始提问，然后也许会将它与过去的另一起事件关联起来，等等。你还可以使用其他 MCP 工具，诸如此类。我们也会保存 Blob 对象。比如有人上传图谱时，该文件会保存在 S3 中，而且我们知道它与某个指标有关。From a graph, we go into a knowledge base. The process again, at every run of the harvester, we save the Turtle files in S3. We create a Parquet file. We have our graph database that is constantly polling that stream. It's actually a CDC, change data capture stream that loads those triples in memory. We can query in real time, because then we have the data. We have described the incident, for example. Someone else from a Claude Code session, or Cursor, or whatever can be querying what's going on with incident 123. We already have that data. The LLM will go make SPARQL queries and start asking questions, and then maybe correlate it to another incident in the past, and so forth. You use other MCP tools and so forth. We also save Blobs. For example, when someone uploads a graph, that file will be saved in S3, and we know it's related to a metric.

这就是所用的查询。所有内容都在其中描述。实际的图，也就是我所说的仪表板图表，有一张截图。截图已保存，字节数据存储在 S3 中。那个 Blob 也是一个实体。一切都相互关联。我们可以恢复所有这些数据。也许你想做取证分析，把数据找回来，准确查明事情发生时的情况。本体是混乱与理解之间的一份契约。我们的随机引擎受本体所奠定的强类型系统约束，并会生成可供查询的确定性三元组。我们就是这样从事件本体、几个命名空间，以及许多智能体和信号起步的。This is the query used. All that is described. The actual graph, the graph I mean the dashboard graph, it has a screenshot. The screenshot is saved, the bytes in S3. That Blob is also an entity. Everything is linked. We can recover all that data. Maybe you want to do forensic analysis and bring it back and get what happened exactly when things occurred. The ontology is a contract between chaos and understanding. Our stochastic engines are constrained to this very strong type system grounded on the ontology, and they emit deterministic triples that can be queryable. That's how we started with the incident ontology, a few namespaces, many agents and signals.

这是我们每次运行都会生成的产物之一。它只是一个保存在 S3 中的 HTML 文件。我们用它来调试，因为我刚才在检查：它提取的内容对不对？我知道其中应该有 Jira 问题、Slack 用户、文件附件之类的东西。每次运行时，我有时会发现它没有找到文件附件。我需要一些产物、一些目标，因为我一直在看代码。我只是在查看这些结果，因为我知道里面应该有什么。如果缺少了，我们就得继续修复。最后，借助这个产物，我可以进行检查。它只是一个事件时间线。所有内容都经过类型化并相互关联。这是一起涉及许多关系的重大事件。拓扑结构和服务依赖关系也都编码在其中。任何人都可以询问其中涉及哪些团队。This is one of the artifacts that we produce on every run. This is just an HTML file saved in S3. We use this to debug, because I was just looking, is it extracting the right things? I know there has to be a Jira issue, or Slack users or file attachments and all those things. Every run, sometimes I realize, it's not finding file attachments. I need to have some artifacts, some goal, because I'm looking at the code all the time. I'm just looking at these results because I know what has to be in there. If it's not there, we need to keep fixing. Finally, by using this artifact, I could inspect. This is just a timeline of the incident. Everything is typed and linked. This is a big incident with many relationships. The topology, the service dependency is also encoded in there. Anyone can ask any questions, which teams are involved.

这里谁的情绪低落？谁压力特别大，诸如此类？一切都是相互关联的。想象一下，所有这些事件都汇聚到一个大型图谱中，这样我们就能同时查询它们。真正的挑战在于让所有这些数据持续实时流动，这就是我们所说的“时间旅行”。目前，我们只保存了几周的数据。所有这些数据都在 S3 中，所以最终我们可以构建一个非常庞大的数据库，随时进行查询。Who has low sentiment here? Who's very stressed, and all that? Everything is linked. Imagine all these incidents gluing up in the big graph so we can query them at the same time. The challenge is actually to keep all this data flowing in time, what we call the time travel. For now, we're just saving a few weeks of data. All this data is in S3, so eventually we can come up with a really big database that we can query at any time.

AutoSRE——将本体论付诸实践AutoSRE - Putting the Ontology to Use

我们如何使用本体？其中一个应用就是我们所说的 AutoSRE。这是我们的 AI SRE 解决方案。这是如今业内的称呼，指借助 AI 提升站点可靠性、辅助 SRE。我们就这么称呼它。AutoSRE 会实时接入指标、日志和追踪数据，也会接入 Slack。它包含拓扑信息，也包含设备信息，并能查看端到端图，也就是设备之间的关系图。它还会接入本体，也就是知识图谱，其中包含作为领域概念的更高层次关系。回到 LOLOMO 事件或问题，AutoSRE 会介入，不只是通知值班人员；它会分析所有 MELT 数据，以及正在运行的所有 A/B 测试版本，然后以比人类快得多的速度进行关联分析。它会找出涉及哪些事件或应用，等等。它也会查找日志。How do we use the ontology? One of the applications is what we call the AutoSRE. It's our AI SRE solution. That's the name now in the industry, for site reliability, aiding SRE with AI. We call it that. What AutoSRE does is taps into the metric, logs, and traces in real time, in also Slack. It has the topology, it has devices as well, and ways of looking at the end-to-end graph, which is the devices. Taps into the ontology, so in the knowledge graph, which is higher creative relationships as domain concepts. Going back to the LOLOMO incident or issue, AutoSRE comes in, not only the on-call, and it analyzes all the MELT data and versions of all the A/B tests that's running, and then it does correlation much faster than humans. It finds what incidents or apps are involved in it, and so forth. It finds logs as well.

它会使用那个。我们有一系列 MCP 和 CLI 工具来查询基础设施，查询范围不仅限于本体，也包括整体数据。它还会查看代码，就像许多现有解决方案一样，因为如果不深入代码，就无法理解系统。它会生成这样的结果：这条记录操作出现了 GraphQL 错误激增，接下来的步骤如下。我们实际上还为此加入了交互功能，用户可以提出后续问题，因为如今很难让 AI 第一次就给出正确答案。你需要经过多轮迭代才能深入探查，因为 LLM 有时会漫无目的地跑偏之类的。It uses that. We have a bunch of MCPs and CLI tools querying the infrastructure, not only in ontology, but also really the data in general. It also looks at code, as many solutions out there do, because you cannot understand systems if you don't go down into the code. This is what it produces. It says, there was a GraphQL error spike for this record operation, and the next steps are these. We have actually added interactivity to this, where users can ask follow-up questions, because it's really hard to get any AI today to give you the right answer on the first go, in the first loop. You need to go through many iterations to dig in, because the LLMs can wander and stuff.

未来会怎样What the Future Holds

未来会给我们带来什么？有三件事。我们希望找到自动化根因分析的方法。我们不想再不停地深挖。只要给出水晶球，它就能告诉我们根因是什么。接下来是真正实现自动修复。比如，如果我们知道这些模式会出现，本体就会告诉我们，在这种情况下，最好的做法就是终止这个实例。好，那就执行。自动修复，甚至不需要人工介入。第三件事是基础设施自愈。这就是我们的梦想。我们甚至不想等到事故发生，也不想收到告警。我们希望不断把处理环节左移，一直左移，直到基础设施能够自我修复。因为等我们不仅描述事故和运维数据，还将本体一路整合到数据血缘，甚至追溯到一部电影是如何创作出来的时候，就能做到这一点。What the future holds for us? There are three things. We want to find automatic root cause analysis. We don't want to keep digging. We can just give the crystal ball, this is the root cause. Next is to actually do auto-remediations. For example, if we know these patterns occur, the ontology, we'll say, in this situation, the best thing to do is just to terminate the instance. Fine. Do it. Auto-remediation. Without human interaction, even. The third one is self-healing infrastructure. That's the dream. We don't even want to get to an incident. We don't even want to get an alert. We want to keep shifting left and left, all the way to the point that the infrastructure is healing itself, because by the time we describe not only our incidents or our operational data, we will merge ontologies all the way to the data lineage to the point where, say, a movie is created.

某个团队在某个时间由某个人运行的流水线中出现了数据错误，这个错误一路传导，最终导致用户服务中的点击数据出错。这就是我们想要构建的端到端愿景。不仅要构建本体，还要将其与其他更大的领域合并，创建一个描述一切的大型企业本体。这又回到了 Prasanna 展示的最初那个端到端闭环：找到这个将所有较小本体连接成一个整体的大循环。There was a data error in a pipeline that was run by this team, by this person at this time, that cascaded into finally bad clicks on the user service. That's the end-to-end dream that we want to create. Not only forming the ontology, but merging into other bigger domains and create this big enterprise ontology that describes everything. That's going back to the original end-to-end wheel that Prasanna showed. Finding this huge cycle of connecting all the smaller ontologies into big.

有人认识《暗黑》里的这个角色吗？他们其实是时间旅行者。有一个秘密组织掌握了操控时间的能力。他们能够前往未来，实际上也掌握了知识。我从这里获得的灵感是，我借用了他们的一句箴言。它说：“Der Anfang ist das Ende und das Ende ist der Anfang。”也就是“开始即是结束，结束即是开始”。这意味着观察、丰富、推断。我只是想用这个类比来结束，呼应我们的节目。Does anyone know this character from "Dark?" They are actually time travelers. There is a cabal that were able to harness time. They were able to go in the future, and actually, also knowledge. The inspiration here is I'm using one of their mantras. It says, "Der Anfang ist das Ende und das Ende ist der Anfang." "The beginning is the end and the end is the beginning." That means observe, enrich, infer. I just wanted to end that with that analogy to our show.

问题与解答Questions and Answers

参与者 1：你们如何处理图中这些事件的去重和消歧？Participant 1: How do you deal with deduplication and disambiguation in the graph on those incidents?

Renzo Sanchez-Silva：目前，我们让 LLM 进入这个开放世界。它会进行查询、创建三元组，并判断“这个关系已经存在”，然后我们让它创建一个。这会成为一个需要合并的分支，但我只是让它通过，它就这样不断向本体中追加内容。我们可能会得到许多相似的概念吧。我们还没检查过这一点，但理想情况下，比如对于这个规模较小的本体，这个过程已经停止了，也就是说它正逐渐趋于稳定。我发现需要合并的 PR 越来越少，诸如此类。它最终确实会趋于稳定。不过它必须经过人工审核。也就是说，必须由本体专家实际查看并仔细审阅。我看到的是，去重过程受到严格控制。这是可行的。Renzo Sanchez-Silva: Right now, we're letting the LLM go into this open world. It does lookup, and it will create the triples, and say, no, this relationship is already there, and we let it create one. It is going to be a branch that has to be merged, but I'm just letting it pass, and it just keeps appending to the ontology. We may get to many similar concepts, maybe. We haven't checked that, but ideally, for example, for this smaller ontology, the process has stopped in the sense that it's becoming stable. I see less PRs that we need to merge, and so forth. It does eventually become stable. It does have to go through a human. Say an ontologist needs to actually see this, look at it closely. What I see is that deduplication is very controlled. This is possible.

普拉桑纳·维贾亚纳坦：未来，我们也认为需要构建一个模式提取层，将模式泛化以帮助去重。我们还没做到这一点。目前我们还能应付。Prasanna Vijayanathan: In the future, we also see the need to have a pattern extraction layer that can generalize patterns to help with deduplication. We're not there yet. We're able to manage.

参与者 2：在这些图中，所有事物都可能开始彼此关联。你们如何微调基础设施，让信号更多、噪声更少，从而确保它输出的内容不是凭空捏造的，也不是仅仅可能相关，而是真正可操作、真实可靠，并且对用户有价值的信号？Participant 2: In these graphs, everything can start relating to everything else. How do you fine-tune your infra to give you more signal over the noise, so that the things that it outputs are actually not something that has been hallucinated or possibly related, that it's actually actionable, real, and a signal for the user.

阿南德：如何消除虚假相关性？有可能做到吗？Anand: How do you remove false correlations, is that possible?

Prasanna Vijayanathan：回到传统的测试系统，我们采用的是瑞士奶酪模型。我们希望尽可能提供准确且可执行的信息，而眼下最适合告诉我们这些的就是用户。正如 Renzo 提到的，它会创建并自我修复，因此这一环节也存在。我们会跟踪不同的质量指标：收集在线日志，再从中提取指标；此外还有评估系统。也就是说，我们会固定输入和输出，然后持续测试模型的表现。接下来还有人在环反馈。所有这些系统都会把反馈送回模型。我们目前还没有广泛开展这项工作，但之后会通过某种形式的强化学习来利用这些信号，让模型不断改进。Prasanna Vijayanathan: Going back to old testing systems, we have a Swiss cheese model. We want to be able to give as much accurate and actionable information as possible, and the best person to tell us that would be the users right now. Like Renzo mentioned, it creates and self-heals itself, so that also exists. We have different quality metrics that we track. We do online logs, and then extract metrics from that, and then we have evals systems. It's like, we freeze our inputs and outputs, and then keep testing how our model is doing. Then we have human-in-the-loop feedback. All of those systems feed back into our model, and we haven't done that extensively yet, but it'll get into some kind of reinforcement learning to use those signals, and improve itself.

参与者 3：规模这一部分让我有点好奇，因为整个图看起来都是用 Turtle 表示的，就像通过 PR 提交代码一样创建的。那么，这只是每秒 3200 万次查询中的一个小子集，而且我们并没有把这些部分关联起来。这是一个抽象层级高得多的结构。对吗？Participant 3: I'm a bit intrigued about the scale piece, because it seems like the whole graph is in Turtle, created by PRs like code. This is a small subset, then, of those 32 million per second queries, and we're not relating those bits. It's a much higher-level construct. Is that correct?

Renzo Sanchez-Silva：这个想法是，对于这种高吞吐量场景，我们只收集元数据，也就是某件事开始发生时留下的指纹。我们发现了这种模式，但不会用 Turtle 获取数据。那样根本行不通。我们会找出正在发生的事情的指纹，然后将其关联到图中的其余部分。同时也要始终控制基数。Renzo Sanchez-Silva: The idea is, for this high-volume throughput, we're just going to be collecting metadata, fingerprints of something has started. We have found this pattern, but we are not going to get the data on Turtle. There's no way to do that. We're going to find the fingerprint of what's happening, and then relate that for the rest of the graph. It's just always controlling cardinality as well.

参与者 4：你们如何判断某件重要的事情没有发生，而不是仅仅发现已经发生的不良事件？Participant 4: How do you detect that something important didn't happen, instead of just picking up something bad that happened?

Renzo Sanchez-Silva：本体实际上具备非常强大的形式推理能力。这与 AI 无关，实际上是基于命题逻辑的推理。它依靠非常复杂的算法进行推理，能够发现不一致、传递关系或矛盾。随着工作的推进，我们也计划利用形式推理来捕捉这些问题。不仅要找出数据中的不一致之处，也要发现新的关联。我们可以通过非 AI 的方式进行形式推理，也可以利用 AI 来辅助这种基础推理。Renzo Sanchez-Silva: The ontologies actually have a very powerful formal reasoning. It's not AI-related. This is actually proportional, logical. Very hard algorithms that actually can reason, can find something that is incongruent or transitive or contradictory. We also plan to use formal reasoning to catch these things as things progress. Not only to find inconsistency in data, but also to find new connections. We can do formal reasoning with non-AI, but we can also use AI to help that base reasoning.

普拉萨纳·维贾亚纳坦：在这个系统中，让人参与其中也很有帮助。开发人员可能会注意到某个系统没发现的问题，或者我们发现了一个实际需要解决的事故或问题。接着我们会查明系统为什么没发现它。事实上，最近我们遇到过这样的情况：有人指出问题并问：“你为什么没告诉我这件事？”之后系统就能自行纠正，并创建一组新的配对。这就是事情的另一面。Prasanna Vijayanathan: A human in the loop also helps a lot in that system, where a developer notices something that was not caught, or we notice an incident or an issue that came up that actually had to be solved. Then we would look at why the system didn't catch it. In fact, recently we had a case where someone flagged it and said, why did you not tell me this? Then it was able to correct itself, and then create a new pair. That's the other side of it.

参与者 5：您能告诉我，您会允许本体多久变更一次吗？这会如何影响知识图谱的重建？Participant 5: Could you tell me how often you would allow your ontology to change? How does that affect your knowledge graph rebuild?

Renzo Sanchez-Silva：我们采用开放世界假设，放手让它发展。至少就事件而言，它似乎很稳定，但我们也会找出与告警之间的关联。这是一个非常小的本体，因为我们想捕捉的只是告警的元数据。比如，告警和信号中包含的关键词或概念都很少。对于其他内容，比如流水线、作业等等，也有办法加以控制。它的规模很小。比较难处理的是事件，因为什么事情都可能发生。概念还在不断增加，但就像我说的，它会逐渐稳定下来。到某个时候，你需要回头检查；我们需要和人一起对其进行精简，这样它就会保持在可控范围内。我们肯定需要让人在回路中。Renzo Sanchez-Silva: We are letting it go with the open world assumption. It seems stable, at least for the incidents, but we're also going to find connections with alerts. That's a very small ontology, because all we want to catch is the metadata of the alerts. There are very few keywords or concepts in the alert, for example, and the signals. Then for other ones, like pipelines and jobs and all that, there's also a way to control that. It's very small. What is harder is the incidents, because anything can happen. The concepts are growing, but like I said, it becomes stable. At some point, you need to go back and look, we need to compress that with humans, and then it will stay controlled. We need human in the loop, for sure.

参与者 6：有计划开源吗？Participant 6: Any plans on open-sourcing?

Prasanna Vijayanathan: Maybe. Not yet.

 

查看更多带文字稿的演示文稿See more presentations with transcripts