快速扩展在线存储以服务超过 10 亿 ChatGPT 用户（第二部分）

所有 OpenAI 产品都需要快速、可靠地访问数据，这由我们的存储平台 Habitat 提供支持。它必须在毫秒内响应来自 40 个地理区域、超过 10 亿用户的请求。我们的基础设施面临着非同寻常的挑战：在持续提升可靠性、性能和产品开发效率的同时，支撑爆发式增长和新的用例。要解决这一挑战，就必须重新思考在线存储平台的设计和运营方式。我们不断权衡取舍，并遵循这样的路径：首先，让平台足够可靠，能够承载关键任务的产品流量；然后，提升速度，以服务全球用户；最后，灵活自如地在超大规模下运营。All OpenAI products require fast, reliable access to data, provided by our storage platform, Habitat. It has to respond in milliseconds to over 1 billion users across 40 geographic regions. Our infrastructure faced an extraordinary challenge: supporting explosive growth and new use cases while continuing to improve reliability, performance, and product developer velocity. Solving that challenge required rethinking how our online storage platform was designed and operated. We constantly evaluated tradeoffs, following the pattern of: first become reliable enough for mission-critical product traffic, then fast enough for global users, and, finally, to deftly operate at massive scale.

这个由两部分组成的系列将介绍我们如何将 Habitat 从一个简单的客户端库和数据库，发展成一个全球分布式存储平台，为超过 500 拍字节的数据提供服务。第一部分重点介绍了 Habitat 如何从 Python 客户端演进为一项服务，也介绍了让我们能够将基于 Python 的服务推向远超许多人认为切实可行范围的各种技术。This two-part series explains how we developed Habitat from a simple client library and database into a globally distributed storage platform serving over 500 petabytes of data. Part I focused on Habitat’s evolution from a Python client to a service. It also covered the techniques that allowed us to push a Python-based service far beyond what many would consider practical.

在这篇文章的第二部分中，我们将介绍如何扩展底层存储组件 Azure Cosmos DB。通过与 Azure 团队紧密合作，我们最终采用了一种可行的多租户方案：共同管理容器，并在全球范围内实现了出色的大规模读取性能。In this post, Part II, we walk through how we scaled the underlying storage component: Azure Cosmos DB. Through a close working partnership with the Azure team, we arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.

随着公司规模扩大，基础设施团队通常会从运行单租户系统转向运行多租户系统。单租户模式下，每个产品界面团队都维护自己的系统，这有助于缩小故障影响范围，明确所有权边界，并隔离工作负载。但随着团队和产品界面不断增多，这种模式的运营成本会变得很高。单租户模式还意味着每个产品团队都要负责容量规划，而且往往会造成大量资源过度配置。As companies scale, infra teams often evolve from running single-tenant systems to multi-tenant ones. Single-tenancy, where every product surface team maintains their own system, reduces blast radius and provides easy ownership boundaries and workload isolation. As teams and product surfaces grow, this becomes expensive to run. Single-tenancy makes every product team responsible for capacity planning and often results in significant over-provisioning.

相反，多租户模式将控制权交由集中的数据库团队，他们在资源配置方面拥有更丰富的知识和经验。但多租户也会带来自己的问题，例如“吵闹的邻居”问题：某个租户占用异常多的资源，从而损害其他租户的利益。Conversely, multi-tenancy assigns control to a centralized database team, with better knowledge and experience for provisioning resources. But multi-tenancy creates its own problems, such as noisy-neighbor issues, where one tenant consumes an unusually large share of the resources to the detriment of the other tenants.

在 Habitat 中，为了设计出能够大规模运行的可靠、高效存储系统，我们不得不权衡这些因素。如今，Habitat 是许多 OpenAI 产品使用的多租户数据库层，但我们的历程颇为曲折：从使用单个多租户 Azure Cosmos DB 容器，到配置单租户后端容器，再到默认恢复使用多租户容器。Habitat 的多租户实践仍处于非常早期的阶段：我们支持大量用例，但通过尽可能利用 Azure Cosmos DB 内置的基础功能，我们得以暂缓在这一领域投入资源。In Habitat, we had to weigh all these tradeoffs to design a reliable, efficient, storage system at scale. Today, Habitat is a multi-tenant database layer for many OpenAI products, but we went through a circular journey from using one multi-tenant Azure Cosmos DB container to provisioning single-tenant backing containers to reverting to a multi-tenant container by default. We’re still very early in our multi-tenancy story on Habitat: we support a large set of use cases, but we’ve been able to defer investment in this area by leveraging Azure Cosmos DB’s built-in primitives wherever possible.

在介绍我们使用 Azure Cosmos DB 的历程之前，先了解一些基本背景：Before walking through our Azure Cosmos DB journey, some basic context:

• Azure Cosmos DB 帐户定义了一组容器和一项复制策略。

  An Azure Cosmos DB account defines a set of containers and a replication policy

• 每个容器都配置了一定的吞吐量，以每秒请求单位 (RU/s) 表示。

  Each container is provisioned for a certain amount of throughput in request units per second (RU/s)

• 每个容器中的数据通常按照指定的逻辑分区键进行分片存储，并且每个逻辑分区键的存储量和吞吐量均有明确限制。

  Data within each container is stored in a typical shard-by-specified-logical-partition-key manner with documented limits on storage and throughput per logical partition key

• Azure Cosmos DB 会将逻辑分区键分片到数量较少的底层物理分区中；当存储量超过阈值，或容器需要更多物理分区来维持预配吞吐量时，会将物理分区拆分为多个分区。

  Azure Cosmos DB handles sharding logical partition keys across fewer backing physical partitions, splitting them into multiple physical partitions when storage exceeds a threshold or when the container requires more physical partitions to sustain the provisioned throughput

最初，我们允许任何客户端负载通过，依靠每个逻辑分区键 10K RU/s 的既定上限来提供背压。刚开始时，我们还在每个帐户中只使用一个默认的 Azure Cosmos DB 容器。这种做法效果尚可，但后来发现存在几个问题：We originally allowed any client load through, relying on the stated 10K RU/s limit per logical partition key to provide back pressure. When we first set out, we also used only one default Azure Cosmos DB container in each account. This worked well enough, but proved problematic for a few reasons:

由于早期 API 设计选择，我们将逻辑分区键的选择直接开放给客户端。因此，不同产品用例中常用的分区键（例如 user_id 或 workspace_id）经常发生冲突。由于这些用例共享 Azure Cosmos DB 对同一逻辑分区键施加的限制，整体设计限制了单个用户可获得的吞吐量。Due to early API choices, we exposed logical partition key selection directly to clients. As a result, we have frequent collisions on commonly used partition keys, such as user_id or workspace_id, across different product use cases. Because these use cases shared Azure Cosmos DB limits on the same logical partition keys, the design constrained overall throughput for a given user.

在迁移到 Habitat 和 Azure Cosmos DB 之前，扩展 Postgres 时我们遇到的一个常见问题是：来自某个极为活跃的组织或热点键的大量请求——也就是某个租户产生的负载远高于其他租户——可能会拖垮整个数据库。我们为缓存这些键、并在缓存未命中时引入租约机制投入了大量精力，以免由此产生的缓存未命中负载压垮底层数据库。One common problem we faced in scaling Postgres before we moved to Habitat and Azure Cosmos DB was the ability for many requests from a very hot organization or hot key—one tenant generating a disproportionately high load—to take out the whole database. Significant collective effort was spent to cache those keys and build leases into cache misses so that the resulting cache miss load does not stomp on the underlying database.

虽然逻辑分区键受文档所述的数据大小和吞吐量限制约束，但实际执行这些限制的是物理分区。这意味着，逻辑分区实际上无法达到 10,000 RU/s，而是要与同一物理分区上的其他所有逻辑分区键共享该物理分区的限制。While documented data size and throughput limits apply to logical partition keys, the physical partition is actually where limits are enforced. In practice, this means logical partitions are unable to reach 10,000 RU/s. Instead, they share the physical partition’s limits with every other logical partition key colocated on that physical partition.

这使得允许的噪声邻居影响了许多无关的逻辑分区键，处理小众产品功能数据的工作负载干扰了核心登录和注册流程。This allowed noisy neighbors to impact many unrelated logical partition keys, with workloads operating on data for niche product features disrupting core login and signup flows.

给定物理分区的吞吐量上限也会变化，具体取决于分配给 Azure Cosmos DB 容器的总吞吐量。容器的预配吞吐量大致均匀地分布在该容器的各个物理分区上。The throughput limit for a given physical partition is also variable, dependent on the total throughput assigned to the Azure Cosmos DB container. The container’s provisioned throughput is distributed roughly evenly across the physical partitions for that container.

随着数据量增长，物理分区会发生拆分，导致分区总数增加，而预置吞吐量总量保持不变。因此，随着数据规模扩大，每个分区的有效吞吐量上限会降低。As data volume grows, physical partitions split, increasing the total partition count while total provisioned throughput remains unchanged. The result is that the effective throughput limit per partition shrinks as data size grows.

对于存储需求最高、类似 Blob 的用例，这会造成一种故障模式：即使请求速率和使用量保持稳定，单是存储量的增长也会降低活跃工作负载的吞吐能力。For our most storage-heavy, blob-like use cases, this creates a failure mode where storage growth alone reduces throughput capacity for active workloads even while request rates and usage remain stable.

使用独立容器有助于部分解决这些分区键限制和“吵闹邻居”问题。通过使用独立容器，每个容器都可以分配到各自独立的物理分区，并且只有该容器自身不同的工作负载才可能彼此成为“吵闹邻居”，或在分区键空间中重叠。缺点是，预配新容器会带来一定开销——每个容器都需要单独进行容量规划——因此，我们只希望在合理的服务边界或用例边界上创建容器。Separate containers help partially solve these partition key limits and noisy neighbors. By using separate containers, each can be assigned its own independent physical partitions and only the varying workloads on that container itself could be noisy neighbors or overlap in partition key space. On the downside, there is non-zero overhead in provisioning new containers—each container requires individualized capacity planning—so we want to create containers only at a service or use case boundary that makes sense.

我们开始针对需要强隔离的最关键新用例，使用独立的 Azure Cosmos DB 容器。再结合一个用于管理弱分区的脚本，使每个容器预配的总吞吐量与其物理分区数量保持同步，我们基本上避免了此前面临的所有问题。We began to use separate Azure Cosmos DB containers for the most critical new use cases requiring strong isolation. Combined with a script for managing weak partitions by synchronizing the total throughput provisioned for each container with its number of physical partitions, we mostly avoided all the problems we were facing.

然而，由于我们一开始使用的是默认容器，许多传统关键用例很遗憾地一直留在那里，未进行数据迁移。为了保护仍依赖共享默认容器中数据的关键服务，我们常建议团队为新服务或重要产品功能创建独立容器。随着规模扩大，默认容器承受的压力越来越大，生产事故也随之增多。为了避免默认容器承受更多负载，我们开始为任何新服务或大型产品功能配置容器。However, since we started with a default container, many legacy critical use cases were unfortunately stuck there without a data migration. To protect critical services that still relied on data in the shared default container, we often advised teams to create separate containers for new services or major product features. As we scaled, pressure on the default container grew and production incidents also grew. We began to provision containers for any new service or large product feature just to protect the default container from more load.

这让我们避免了许多全站范围的故障，将影响范围限制在功能或服务降级层面。然而，配置新容器的开销最终还是给我们带来了负担：This saved us from many site-wide outages, limiting the blast radius to feature or service degradation outages. However, the overhead of provisioning new containers eventually caught up to us:

在旧的共享默认容器中，任何特定的新用例即使从 0 QPS 增长到 100 万 QPS，也不太可能显著改变总的聚合用量。然而，在隔离容器中，一个新用例从 0 增长到 100 万 QPS，其负载可能达到仅由内部自用所形成基线负载的 1,000 倍。In the old shared default container, any particular new use case going from 0 QPS to 1 million QPS was unlikely to significantly shift the total aggregate usage. On an isolated container however, a new use case going from 0 to 1 million QPS might represent 1,000x more load than the baseline attributed to only internal dogfooding.

采用专用容器后，团队需要记得提前预配足够的容器，以满足上线时预期的吞吐量。产品团队经常会将功能发布到生产环境，却忘了提前扩容，结果我们只能在上线当天待在 SEV 事件响应室里，试图在最后一刻扩充容器规模。With dedicated containers, teams now needed to remember to pre-provision containers to the expected level of throughput for their launches. Product teams would often ship their features to production and forget to pre-scale and we’d sit in SEV rooms on launch day attempting to scale up containers last minute.

这是多租户系统混合运行核心工作负载的一项优势，而采用这种容器策略来实现更好的隔离性，就意味着我们会失去这项优势。This is a core workload mixing benefit of multi-tenant systems that we lose in favor of better isolation with this container strategy.

随着时间推移，监控专用容器的使用情况也要求深入了解产品，以便掌握流量何时会发生转移或增长。Over time, monitoring usage of dedicated containers also required intimate product knowledge of when traffic would shift or grow.

我们可以提供一个基本的控制平面脚本，根据增长趋势提高每个容器的预置吞吐量；但由于整体增长并不稳定，要想正确地扩展这些容量，OpenAI 几乎每个产品团队都必须参与容量规划。We can provide a basic control plane script to scale up the provisioned throughput for each container based on growth trends, but without the stable aggregate growth, scaling these correctly enabled essentially every product team at OpenAI to be involved in capacity planning.

由于各服务所属团队在 Azure Cosmos DB 配置方面投入的审慎程度差异很大，这种做法在组织层面难以扩展。This failed to scale organizationally with a wide variance in how much diligence each service’s owning team paid to their Azure Cosmos DB provisioning.

随着组织不断发展，这些运营问题变得难以扩展。由于没有一个统一的中央团队负责规划，许多团队的规划都不够完善。因此，我们最终转而采用多租户解决方案，并与 Azure 团队密切合作，通过共享租户设计解决最初遇到的多租户问题：As the organization grew, those operational problems became difficult to scale. No lone, central team doing planning resulted in many teams planning poorly. As a result, we ended up shifting back towards a multi-tenant solution and worked closely with the Azure team to engineer fixes for our original multi-tenancy issues with a shared-tenancy design:

优先对最热门的键进行限流，可以防止吵闹邻居占用特定物理分区的全部可用吞吐量。这能减轻吵闹邻居效应，即一个热点键可能造成波及整个站点的影响。Throttling the hottest keys first prevents noisy neighbors from consuming all of the available throughput for a given physical partition This reduces the noisy neighbor effect, where one hot key can have a site-wide blast radius.

请求分层 QoS 同样可以防止离线工作负载（例如批量插入或数据提取）占用给定物理分区的全部吞吐量。为这些工作负载的请求分配较低优先级后，它们会首先受到限流，从而避免服务终端用户的高优先级交互式请求受到限流。Request tiering QoS similarly prevents offline workloads, such as bulk inserts or data extracts from utilizing all of the throughput for a given physical partition. By assigning the requests for these workloads a lower priority, they are throttled first, preventing throttling on higher priority, interactive requests that serve end users.

Azure Cosmos DB 当时已经提供无服务器模式，本可以解决我们在预配吞吐量模式下遇到的弱分区问题。但它有一个重大缺点：只能根据存储使用量进行扩展，并将每个物理分区的 RU 使用量限制在 5K RU/s。虽然这能解决我们的弱分区问题，却会阻碍 RU 消耗量大的使用场景扩展。Azure Cosmos DB already had a serverless mode that would’ve fixed our weak partitions problem with the provisioned mode. However, there was a major downside: it scaled based on storage usage only and limited RU usage to 5K RU/s per physical partition. This would solve our weak partitions problem, but prevent us from scaling up RU-heavy use cases.

Azure Cosmos DB 团队为我们打造的新版增强型无服务器模式，使每个物理分区都能扩展至 10K RU/s，并在基于存储使用量拆分的基础上引入了基于 RU 使用量的拆分，让我们不再需要手动监控和扩容容器。A new improved serverless mode Azure Cosmos DB team built for us allowed every physical partition to scale up to 10K RU/s and introduced RU-usage-based splits in addition to storage-usage-based splits allowed us to stop monitoring and scaling up containers manually.

通过这些更改，我们得以重新采用一个共享容器，这显著简化了产品团队进行前期和持续容量规划的需求，同时保留了各工作负载之间的隔离。此后几个月里，尽管产品开发步伐加快，我们也没有遇到任何资源配置问题或“吵闹邻居”事件。With these changes, we’ve been able to shift back to using one shared container which has significantly simplified the need to do any upfront or continued capacity planning for product teams while preserving isolation between workloads. We have not had any provisioning or noisy neighbor incidents in the months since, despite accelerating product development.

当企业发展到 OpenAI 这样的规模时，托管服务就不再那么“托管”了。许多工作负载在云服务商的基础设施中都成了特例。随着 OpenAI 成为 Azure Cosmos DB 的最大用户之一，要想成功运营这项服务，就需要采用不同的模式：共享测试、共享仪表板、直接对接工程团队，并共同负责那些只有在我们这样的规模下才会出现的故障模式。如今，我们最大的 Azure Cosmos DB 账户存储着数十拍字节的数据，每秒处理超过 2000 万次查询。据我们所知，我们是唯一一家将 Azure Cosmos DB 部署到这一规模的公司。When companies grow to OpenAI’s scale, managed services become less “managed.” Many workloads become outliers in the cloud provider’s infrastructure. As OpenAI became one of the largest users of Azure Cosmos DB, operating it successfully required a different model: shared tests, shared dashboards, direct engineering channels, and joint ownership of failure modes that only appear at our scale. Today, our largest Azure Cosmos DB account stores 10s of petabytes of data and serves more than 20M QPS. To our knowledge, we are the only company which has deployed Azure Cosmos DB at this scale.

2024 年底，一次区域性服务中断让我们清楚地意识到这项协作为何不可或缺。当时，我们规模最大的 Azure Cosmos DB 账户之一所在的主要 Azure 区域发生了停电。在随后的故障转移演练中，我们发现，规模最大的账户需要数小时才能故障转移到辅助区域，而无论是我们的团队还是 Azure Cosmos DB 团队，此前都未在我们这样的规模下进行过测试。In late 2024, a regional outage made it clear why this collaboration was necessary. At the time, a power outage struck the primary Azure region for one of our largest Azure Cosmos DB accounts. In the resulting failover exercise, we realized that our largest accounts would take several hours to fail over to a secondary region, something neither our team nor the Azure Cosmos DB team had tested at our scale.

大约一个月后，在将这些账户故障切回原始区域时，我们又引发了一次严重服务中断。大量连接涌向原始区域，压垮了该区域的 Azure Cosmos DB 网关。网关模式下的 Azure Cosmos DB 客户端处理故障转移的方式存在一个特殊问题，导致所有客户端立即与原始区域建立连接并向其发送请求，尽管实际上只有一部分物理分区已切回该区域。About a month later, while failing those accounts back to the original region, we triggered another serious outage. A connection stampede to the original region overwhelmed the original region’s Azure Cosmos DB gateway. A quirk in how the gateway-mode Azure Cosmos DB client handled failovers caused all clients to immediately establish connections and send requests to the original region, even though only a subset of physical partitions had moved back to that region.

每次服务中断期间，OpenAI 和 Azure 都执行了各自的事件管理流程，找出了多个根本原因并部署了修复措施。但这些服务中断暴露出一个更大的系统性问题：我们的团队与 Azure Cosmos DB 团队在运营和测试方式上存在问题。起初，借助托管服务，我们得以快速推进并专注于业务目标。但随着规模扩大，托管服务的托管程度会降低，因此从设计到运营的各个层面，都需要与服务提供商进行深度集成。In each outage, both OpenAI and Azure ran through their incident management processes, with a number of root causes identified and fixes deployed. But these outages were symptoms of a larger systemic issue between the way our team and Azure Cosmos DB team operated and tested. At first, leveraging managed services allowed us to move fast and focus on business goals. At scale, managed services become less managed and require deep integrations with the service provider at all levels from design to operations.

如今，Habitat 和 Azure 工程团队共同监控 Habitat 的每一层，从我们的服务层一直深入到 Azure Cosmos DB 的物理存储层。我们在一个规模与生产环境相当、但并非生产环境的 Azure Cosmos DB 帐户上运行并测试变更。我们共同测试并迭代 SDK，确定 Azure Cosmos DB 帐户的扩展和容量规划最佳实践。我们可以通过 Teams 直接联系彼此的工程团队，甚至还建立了无需经过各级支持渠道、即可直接向对方发送寻呼通知的方式。Today, the Habitat and Azure engineering teams jointly monitor every layer of Habitat from our service layers down into Azure Cosmos DB’s physical storage layers. We operate and test changes against a production-sized but non-production Azure Cosmos DB account. We test and iterate on SDKs together and determine best practices for how we scale and size our Azure Cosmos DB accounts. We have direct engineering access to each other through Teams and have even exposed ways to directly page each other without going through support tiers.

过去一年，双方开展了大量协作工作，以提升 Azure Cosmos DB 控制平面和网关的可扩展性：在网关中增加负载保护机制，并缓存各种 Azure Cosmos DB 元数据，以保护 Azure Cosmos DB 元数据分区；通过连接池和提高连接复用率，减轻我们部署的大量 Python habitat-service 进程带来的压力；在可能的情况下将工作负载共置；并通过使用不同的 VPC，将关键工作负载与离线工作负载隔离开来。A substantial amount of joint work has occurred over the last year to improve Azure Cosmos DB control-plane and gateway scalability, adding load protection and caching of various Azure Cosmos DB metadata in their gateway to protect the Azure Cosmos DB metadata partition, reducing pressure from our fleet of Python habitat-service processes through connection pooling and higher connection reuse, colocating workloads where possible, and isolating critical workloads from offline workloads via different VPCs.

过去一年，这种协同共建对 Azure Cosmos DB 的稳定性至关重要。任何规模足够庞大或足够关键的系统，都需要这样的团队合作：打破职责壁垒，跨越组织边界开展协作。This level of co-engineering has been critical to our Azure Cosmos DB stability in the last year. Any system large or critical enough necessitates this kind of teamwork: breaking down ownership walls and operating across organizational boundaries.

正如第一部分所述，我们的核心理念是：先让它能用，再让它变快，最后让它更便宜。可靠性优先，其次是性能，最后是效率。Habitat 已成为 OpenAI 的关键基础设施，因此首要任务是确保其可靠。随着系统在 2025 年趋于稳定，且 OpenAI 业务拓展至全球各地，延迟成为用户和产品团队面临的下一个瓶颈。As mentioned in Part I, our core philosophy is: make it work, make it fast, make it cheap. Reliability first, then performance, then efficiency. Habitat had become critical infrastructure for OpenAI, so the first priority was making it reliable. As the system stabilized in 2025 and OpenAI expanded globally, latency became the next constraint for users and product teams.

计算能力和存储容量通常位于不同区域，因此跨区域请求不可避免。Compute and storage capacity were often located in different regions, making cross-region requests unavoidable.

延迟会因数据存储位置和请求处理位置的不同而有很大差异。为了让 Habitat 请求具备良好的性能，产品开发者依赖额外的客户端缓存、请求批处理、尽量减少数据库读写，并尽可能将工作移出关键路径。虽然这催生了一些颇有意思的产品设计，使产品在面对数据库问题时更具韧性，但也让产品变得更加复杂，并拖慢了产品迭代：这未能实现 Habitat 的目标——鼓励快速开发产品，并让开发者无需了解数据库。Latency varied widely depending on where the data was stored and where the request was served. To make Habitat requests performant, product developers relied on additional client-side caching, request batching, minimizing database reads and writes, and moving work off the critical path wherever possible. While this led to some interesting product designs that were more resilient to database issues, it also made products more complex and slowed down product iteration: a failure to achieve Habitat’s goal to encourage fast product development and not needing to understand the database.

这一直是我们在 2026 年重点关注的方向。针对读性能优化，我们制定了一套分层策略，因为读请求占 Habitat 请求总量的 90% 以上：This has been a big focus for us in 2026, and we’ve developed a layered strategy for optimizing read performance, which represents over 90% of Habitat’s requests:

我们在全球不同的 Azure 区域部署了多个 Azure Cosmos DB 帐户，以满足用户的数据驻留要求。即使用户没有严格的数据驻留要求，将数据部署在靠近用户的位置也能降低延迟，从而提升用户体验。We have a number of Azure Cosmos DB accounts in different Azure regions around the world for users with data residency requirements. Even for users without hard data residency requirements, colocating data near users also improves their experience through lower latency.

实际上，除了数据驻留用例之外，由于部署位置受物理云容量限制，我们无法大范围实现这一点。由于数据集规模庞大，许多区域无法容纳按其用户流量占比所需的海量存储空间。In practice, we’re not able to achieve this broadly outside of data residency use cases because placement is constrained by physical cloud capacity. Due to the size of our data sets, many regions cannot accommodate the massive storage footprint required for their proportional share of user traffic.

在数据中心容量、电力或冷却能力有限的地区，我们有时会面临直接的取舍：将容量留给同机房部署的存储以降低 Habitat 延迟，还是将容量用于增加服务计算能力，并将存储请求路由到更远的地方。In regions with limited datacenter capacity, power, or cooling, we sometimes face a direct tradeoff: reserve capacity for colocated storage and lower Habitat latency, or use that capacity for additional serving compute and route storage requests farther away.

因此，放置决策往往需要在以下因素之间权衡：A）针对不同产品和用户群体优化延迟；B）原始存储和计算容量。As a result, placement decisions are often a product tradeoff between: A) optimizing latency for different products and user subsets, and B) raw storage and compute capacity.

对于延迟要求最高的工作负载，例如每个请求都需要的身份验证元数据，我们通过使用在多个次要区域中高度复制的 Azure Cosmos DB 帐户，来降低计算资源与 Habitat 之间的 RTT。有些帐户会复制到我们提供流量服务的全部 40 个 Azure 区域；其他帐户则至少会复制到每个大洲的一个区域。For the most latency-critical workloads, like authentication metadata that is needed on every request, we reduce RTT between compute and Habitat by using highly replicated Azure Cosmos DB accounts with many secondary regions. Some accounts replicate to all 40 Azure regions where we serve traffic; others replicate to at least one region per continent.

我们不得不比预期更早在 Habitat 中构建一个透明的数据迁移和路由层，以便在复制拓扑不同的账户之间移动数据集。We’ve had to build a transparent data movement and routing layer into Habitat earlier than desired so that we can move datasets between accounts with different replication topologies.

然而，这种方法最终也受容量限制，因此只能用于少数经过人工挑选的数据集。However, this approach is also ultimately capacity constrained and therefore limited to a few hand-selected datasets.

Habitat 工作负载绝大多数以读取为主，应用会不断获取设置、状态和其他属性；流量中有相当大的比例来自相对较小、经常访问的类型集合。因此，该系统尤其适合采用分布式缓存。缓存能够提升单区域基础设施的性能，而在跨区域运行时，性能提升会进一步叠加，因为缓存可以用来“掩盖”跨区域延迟。Habitat workloads are overwhelmingly read-heavy, with applications constantly fetching settings, state, and other properties; a disproportionate amount of traffic is attributable to a relatively small working set of frequently accessed types. This makes the system particularly well suited for distributed caching. While caching improves performance in single-region infrastructure, the improvements compound when working across regions, where it can be used to “hide” cross-region latency.

我们已在 Habitat 中集成了分布式旁路缓存层。它会将最近访问过的对象和边的副本存储在 Valkey 服务器中，这些服务器部署在我们的基础设施内，靠近处理用户请求的位置。通常，这类缓存会使用 TTL，在命中率和数据陈旧程度之间进行权衡。较短的 TTL 可以减少数据陈旧，但会削弱缓存带来的正面效果，因为条目很快就会过期并从缓存中移除，因而必须更频繁地重新获取。较长的 TTL 可以提高命中率，但会在更长时间内提供陈旧数据，导致用户体验不佳，并使产品行为难以预测。We’ve integrated a distributed lookaside caching layer into Habitat. It stores a copy of recently accessed objects and edges in Valkey servers placed in our infrastructure close to where user requests are processed. Typically, these sorts of caches use a TTL that trade off between hit rate and staleness. Short TTLs reduce staleness but blunt the positive impact of caching, because items expire out of the cache quickly and must be retrieved more frequently. Longer TTLs improve hit rates, but serve stale data for extended periods, leading to poor user experiences and unpredictable product behavior.

另一种方法是在应用程序对同一底层数据执行写操作时，执行显式失效操作。这能显著提高缓存数据的新鲜度，但很难确保正确无误。此外，它仍会受到许多边缘情况的影响，例如部分失败会导致缓存数据过时。多区域部署会加剧这些问题：跨区域网络的可靠性较低，而且缓存更新可能与底层存储的复制过程发生竞争。配合 TTL 使用可以限制数据过时的最长时间，但仍无法避免上文讨论的 TTL 固有权衡。Another approach is to execute explicit invalidation operations when the application is performing write operations on the same underlying data. This dramatically improves the freshness of cached data, but can be hard to get right. It also remains subject to a number of edge cases, like partial failures, which leave the cache stale. Multi-region deployments compound these problems, with less reliable cross-region networks, and the cache racing with the replication of the underlying storage. Complementing this with TTLs bounds the maximum staleness, but still suffers from the same underlying TTL trade-off discussed above.

我们采用了更先进的失效处理方案，借鉴了 Meta 的 TAO 等系统的做法。Habitat 的集成缓存通过 Azure Cosmos DB 的更改源实现变更数据捕获（CDC），在几秒内将失效操作传播到全球各地，从而在保持较低数据陈旧度的同时实现较高的缓存命中率。我们基础设施中的一项服务会读取此更改源，并在底层数据更新时使本地缓存中的相应条目失效。这样，下次读取该数据时就会发生缓存未命中，缓存随后会填入最新版本的数据。We’ve taken a more advanced approach to invalidation, following in the footsteps of systems like Meta’s TAO. Habitat’s integrated cache relies on change data capture (CDC) through Azure Cosmos DB’s change feed to propagate invalidations globally within seconds, achieving high hit rates while keeping staleness low. A service in our infrastructure consumes this change feed and executes invalidations to the local caches when the underlying data is updated. This causes a miss the next time the data is read, and the cache is filled with the latest version of the data.

我们还通过用性能优于原始 Python 的 Rust 重写代码，消除了 Habitat 服务本身的额外开销，从而大幅降低了延迟。We’ve also significantly reduced latency by removing overhead in the Habitat service itself through rewriting the code in Rust, which is more performant than the original Python.

这些改进使我们的全球读取 p50 从约 120 毫秒降至约 10 毫秒，自今年年初开始关注性能以来，提升了 12 倍。Together, these changes have reduced our overall global read p50 from roughly 120 ms to roughly 10 ms, a 12x improvement since we began focusing on performance earlier this year.

虽然缓存大幅降低了中位延迟，但最慢的请求仍需从底层存储中获取未缓存的数据，而且这些数据往往位于遥远的区域。随着更多容量投入使用、存储效率不断提升，我们计划着手解决一个更根本的挑战：从一开始就将数据放置在更靠近访问地点的位置。While caching has dramatically reduced median latency, the slowest requests still require fetching uncached data from underlying storage, often across distant regions. As more capacity comes online and we improve storage efficiency, we plan to address a more fundamental challenge: placing data closer to where it will be accessed from the start.

可靠且可扩展的 CDC（将数据库更改传播到下游系统）不仅是 Habitat 缓存策略的关键组成部分。每秒捕获数百万条更改，这些更改来自数千个容器和数百万个容器分区，经由一百多个 Kafka 主题传输，为最终一致的二级索引、数据迁移流水线、离线分析以及其他实时产品功能提供支持。随着我们的底层基础设施在全球范围内不断扩展，监听行变更也变得日益复杂。如果客户端希望注册一个回调，在其所属类型的行发生更改时运行，那么除了其他事项外，他们还必须先确定哪些 Cosmos 账户和容器存储着自己的数据、各区域中哪个 CDC 服务部署正在监听这些容器，并妥善扩容 CDC 服务工作节点集群、Kafka 及其消费者。我们的 CDC 平台旨在将这些工作对产品开发者隐藏起来，转而提供一个简单的接口，让客户端注册一个希望在数据行变更时调用的函数，其余所有底层连接工作和扩缩容都由我们处理。Reliable and scalable CDC—propagating database changes to downstream systems—is a critical component in more than just Habitat’s caching strategy. Millions of captured changes per second, across thousands of containers and millions of container partitions feed through more than a hundred Kafka topics to also power eventually consistent secondary indexes, data-movement pipelines, offline analytics, and other real-time product features. As our backing infrastructure has grown globally, it’s become increasingly complex to listen to row changes. If a client wants to register a callback to run whenever rows of their type are changed, they must, among other things, first determine which Cosmos accounts and containers contain their data, which CDC service deployment is running in respective regions listening to those containers, and properly scale the CDC service worker fleet, Kafka, and their consumer. Our CDC platform aims to abstract all of this away from product developers and provide instead a simple surface where clients register a function they’d like called on their changed rows and we handle all of the plumbing and scaling.

内部控制平面会持续确保所有必需的容器都在被捕获，选择要读取的 Cosmos 区域，并在多个地理区域的数千个工作进程之间高效分配任务。使用副本区域并以较低优先级读取 feed，可避免后台捕获工作负载影响对延迟敏感的 Habitat 流量。在如此庞大的集群中，工作进程崩溃、容器问题、租约存储错误、配置无效以及目标端故障时有发生。我们强化了 CDC 工作进程的生命周期管理和租约处理，在不同流之间隔离故障，并为配置变更和推出新版本构建了防护机制。我们使用死信队列，避免永久性交付失败造成队头阻塞。发生区域性中断时，我们会将读取器切换到其他 Cosmos 区域以应对。由于规模庞大，我们甚至不得不修改 Azure 的 CDC SDK，通过在内存中批量处理检查点并仅定期刷新，使其检查点实现更高效。An internal control plane continuously ensures all required containers are being captured, selects which Cosmos regions to read from, and efficiently distributes work among thousands of workers across multiple geographic regions. Using replica regions and lower-priority feed reads protect latency-sensitive Habitat traffic from background capture workloads. Worker crashes, container issues, lease-store errors, invalid configuration, and destination failures occur continually across a fleet this large. We hardened CDC worker lifecycle and lease handling, isolated failures between streams, and built safeguards for configuration changes and rollouts. We utilize dead letter queues to prevent head-of-line blocking from permanent delivery failures. We tolerate regional outages by switching readers to alternate Cosmos regions. At our scale, we’ve even had to make changes to Azure’s CDC SDK to make its checkpointing implementation more efficient by batching checkpoints in-memory and only periodically flushing them.

尽管 Habitat 内部对 Azure Cosmos DB 的使用方式很复杂，但它的价值主张始终是将基础设施细节抽象出来，让产品开发人员无需关注，从而提升产品开发速度；我们的 CDC 平台也不例外。While Habitat’s internal usage of Azure Cosmos DB is complex, its value proposition has always been to abstract away infrastructure details from product developers to improve product development velocity, and our CDC platform is no different.

一个小型团队通过刻意保持存储 API 精简、持续扩展 Python 服务，并优先解决最紧迫的工程挑战，成功支撑了一个快速发展的存储平台。我们首先致力于保障关键业务流量的可靠性，然后提升全球用户的读取性能，最后提高大规模运营效率。这种做法在推动业务快速增长和拓展新用例的同时，也提升了可靠性、性能和产品开发速度。A small team was able to support a rapidly growing storage platform by keeping the storage API deliberately narrow, continuing to scale the Python service, and prioritizing the most urgent engineering challenges. We focused first on reliability for mission-critical product traffic, then on read performance for global users, and finally on operational efficiency at scale. This approach enabled rapid growth and new use cases while improving reliability, performance, and product development velocity.

为了扩展 Habitat 和对话存储，以服务下一个十亿用户，我们需要解决在更大规模上跨基础设施复制数据的难题。为此，我们一直在构建一套定制的内部 EB 级在线存储系统，专为体量庞大的 Habitat 数据类型进行优化。这套正在开发的系统结合了 Rockset 的 RocksDB Cloud 核心与基于 Raft 的复制机制。它采用与 Azure Cosmos DB 相同的文档模型，但使用存算分离和物理复制来减少所需的闪存容量，代价是峰值 QPS 较低。In order to scale Habitat and conversation storage for the next billion users, we need to address the difficulty of replicating our data across our infrastructure, on an even larger scale. To preempt this, we’ve been building a custom in-house exabyte-scale online storage system, optimized for bulky Habitat data types. This in-development system combines the RocksDB Cloud core from Rockset with Raft-based replication. It implements the same document model as Azure Cosmos DB, but uses disaggregated storage and physical replication to reduce the amount of flash capacity needed, in exchange for a lower peak qps.

如果你有兴趣参与前沿规模的数据基础设施项目，欢迎查看我们的开放职位⁠。If you’re interested in working on these types of data infrastructure efforts at frontier scale, check out our open roles⁠.