在 OpenAI 规模化 Git

过去一年，我们的单体代码库规模不断扩大，承载的需求也以惊人的速度增长。显然，我们需要改造源代码管理系统，以适应 Codex 所支持的并发工作规模。Over the past year, the size and demands on our monorepo have increased at a staggering pace. It was clear that we needed to revamp our source control system to scale to the level of concurrent work that Codex makes possible.

我们构建了 Pando——一个全新的 Git 仓库后端，整个过程没有中断任何工程工作，也没有改变团队使用 Git 的方式。如今，OpenAI 单体代码库中的所有开发工作都由 Pando 提供支持。Pando 旨在实现两个宏大的目标：第一，实现可水平扩展的写入能力，针对同时推送到多个分支进行了优化；第二，最大限度地提升读取吞吐量，让客户端能够以足够快的速度下载数据，充分利用可用的网络带宽。We built Pando, a new backend for Git repositories, without interrupting any engineering work or changing how our team uses Git. Pando now powers all development in OpenAI’s monorepo. Pando is designed to meet two ambitious goals: first, horizontally scalable writes, optimized for concurrent pushes to many branches. And second, maximize read throughput, so clients can download data fast enough to fully saturate their available network bandwidth.

在实现这些目标的同时，我们仍然兼容 Git 协议和客户端，让工程师和代理都能将 Pando 无缝融入现有工作流。在生产环境中，Pando 将推送操作的中位延迟降低了近 5 倍。如今，它每天传输近 1 PB 的 Git 数据。We met these goals while retaining compatibility with the Git protocol and client, enabling both our engineers and agents to seamlessly adopt Pando into their existing workflows. In production, Pando cut median push latency by nearly 5x. It now delivers nearly a petabyte of Git data per day.

在这里，我们将介绍构建新源代码管理系统的动机，以及选择保留与 Git 兼容性的原因。我们会将我们的系统与扩展 Git 的常见方法进行比较，并说明为什么 Pando 能够独特地满足 OpenAI 的可扩展性需求。我们还会讨论接下来的计划，以及我们将如何继续为 Git 做出贡献，以支持智能体规模的工作。Here, we’ll cover our motivations for building a new source-control system, along with why we chose to retain compatibility with Git. We’ll compare our system with common approaches to scaling Git, describing why Pando is uniquely capable of meeting OpenAI’s scalability needs. We also discuss what’s next, and how we’ll continue contributing to Git to support agent-scale work.

Codex 改变了 OpenAI 的开发规模和节奏，使工程师能够并行完成更多工作。这意味着分支和推送更多、合并更多，也意味着开发环境、研究任务和 CI 会执行更多 fetch 和 clone 操作。Codex has changed the scale and pace of development at OpenAI, enabling engineers to do far more work in parallel. That means more branches and pushes, more merges, and more fetches and clones from development environments, research jobs, and CI.

这些需求会叠加。现在，一名工程师可能同时有多个更改准备交由 CI 处理，而每次推送都可能触发多个作业获取同一个提交。These demands compound. One engineer now has several changes ready for CI at once, and each push can set several jobs fetching the same commit.

从去年年底左右开始，支撑我们单体仓库的系统就越来越难以应付。我们遇到了获取和推送缓慢、合并失败等问题，可靠性有时甚至低于 90%。我们遇到的是 Git 本身的极限，还是我们为单体仓库提供服务的方式出了问题？Git 源自一个通过电子邮件交换内核补丁的社区。二十多年来，随着开发者构建软件的方式不断变化，他们也围绕 Git 调整了自己的工具。当我们自己的单体仓库开始让我们感到掣肘时，能够在已有工作的基础上继续构建，对我们来说至关重要。Starting around the end of last year, the systems supporting our monorepo were struggling to keep up. We suffered slow fetches and pushes along with failed merges, with reliability at times falling below 90%. Had we reached a limit of Git itself, or of the way we served our monorepo? Git grew out of a community that exchanged patches to the kernel via email. For more than two decades, developers have adapted their tools around Git as the way they build software has changed. When we started to feel friction in our own monorepo, it mattered to us that we could keep building on that work.

我们找到了一种方法，既保留大家熟悉的工作流程，又重建后端，使其能够并行处理推送到不同分支的操作，并同时向许多读者提供相同的历史记录。以 Git 作为源代码管理工作流程的基础，意味着对客户端的改进也能惠及 OpenAI 之外的用户。我们希望将这些改进从我们自己的 Git 副本（在新窗口中打开）贡献给上游，与更广泛的社区分享。We saw a way to keep that familiar workflow while rebuilding the backend to handle pushes to different branches in parallel and serve the same history to many readers concurrently. Building on Git as the foundation of our source-control workflow meant that improvements to the client could be useful beyond OpenAI. We wanted to contribute those improvements from our copy of Git⁠(opens in a new window) to upstream, sharing them with the broader community.

扩展源代码管理系统需要考虑两个关键维度：大量开发者向不同分支推送代码，以及大量用户获取相同的历史记录。Pando 针对这两方面都进行了优化。Scaling a source control system has two essential dimensions: many writers pushing to different branches, and many readers fetching the same history. Pando optimizes for both.

首先，Pando 会独立协调对每个分支的更新。互不相关的分支可以并行更新，而读取者获取某个分支时，会得到该修订版本所需的历史记录。代价是，查看多个分支的读取者可能会看到它们处于不同的时间点。First, Pando coordinates updates to each branch independently. Updates to unrelated branches can proceed in parallel, and a reader fetching a branch gets the history needed for that revision. The tradeoff is that a reader looking at several branches can see them at different points of time.

其次，Pando 会在多次获取操作之间重复使用 packfile（Git 用于将多个对象压缩在一起的格式），而不是为每个客户端单独生成 packfile。客户端可能会下载一些自己已经拥有的对象，但服务器只需很少的计算量，就能将相同的字节提供给许多客户端。Second, Pando reuses packfiles (Git’s format for compressing multiple objects together) across fetches instead of generating an individual packfile for each client. A client may download some objects it already has, but the server can serve the same bytes to many clients with little computation.

这些选择让我们能够将写入操作扩展到多个分支，并通过缓存提供读取服务。为了了解它们如何协同工作，我们先来看看 Git 如何存储和传输历史记录。These choices let us scale writes across branches and serve reads from caches. To see how they fit together, let’s start with how Git stores and transfers history.

Git 必须能够让仓库的两个部分保持同步：对象和引用。Git must be able to keep two parts of a repository in sync: objects and references.

Git 仓库将文件快照及其关联历史记录为一个对象图。对象可以是 blob、tree、commit 或 tag。Blob 存储单个文件的内容，tree 则将名称与 blob 或其他 tree 关联起来，从而构成目录结构。Commit 指向一个根 tree 和零个或多个父 commit，由此形成仓库的历史记录。A Git repository records snapshots of files and the history connecting them as a graph of objects. An object may be either a blob, tree, commit, or tag. Blobs hold the content of an individual file, and trees associate names with blobs or other trees to form a directory structure. Commits point to a root tree and zero or more parent commits, forming the history of your repository.

对象通过其类型和内容的哈希值进行标识。对象也是不可变的：记录已更改的文件会创建一个新的 blob，而未更改的文件和目录则可以重复使用现有对象。Objects are identified by a hash of their type and contents. Objects are also immutable: recording a changed file creates a new blob, while unchanged files and directories can reuse existing objects.

分支和标签是指向提交的引用（refs）。在分支上提交会创建一个新的提交，并更新该分支的引用，使其指向这个新提交。Branches and tags are references, or “refs”, that point to commits. Committing on a branch creates a new commit and updates the branch’s ref to point to it.

Git 以“松散”或“打包”状态存储对象。松散对象存储在各自的文件中，并通过对象 ID 标识。打包对象则存储在同一个包文件中，从而可以将相似对象存储为彼此的差异。Git stores objects in either a “loose” or “packed” state. Loose objects are stored in individual files identified by their object ID. Packed objects are stored together in a packfile, enabling similar objects to be stored as deltas against one another.

在 Git 中，如果 Git 能从某个提交出发，沿着父提交、树和 blob 的链接找到一个对象，那么该对象就可从该提交到达。执行 fetch 时，客户端会告知服务器它想要哪些提交，以及已经拥有哪些提交。服务器会找出从所需提交可到达的对象，排除它知道客户端已拥有的对象，然后将其余对象打包成 packfile 发送，格式与 Git 在磁盘上使用的格式相同。完整克隆开始时没有任何对象，因此需要获取所克隆历史中所有可到达的对象。In Git, an object is reachable from a commit if Git can find it by following links from that commit through its parents, trees, and blobs. During a fetch, the client tells the server which commits it wants and which it already has. The server finds the objects reachable from the wanted commits, excludes those it knows the client already has, and sends the remainder as a packfile in the same format Git uses on disk. A full clone starts without any objects, so it needs all the objects reachable from the history being cloned.

推送会通过请求服务器更新一个或多个引用，将对象沿相反方向传输。客户端利用服务器公布的引用集合，避免发送服务器已有的对象。因此，所发送的包可能依赖于现有对象：仅接收该包并不足以重建新的历史记录。A push transfers objects in the other direction by asking the server to update one or more refs. The client uses the server’s set of advertised refs to avoid sending objects the server already has. The pack being sent can therefore depend on existing objects: receiving the pack alone is not enough to reconstruct the new history.

复制 Git 仓库意味着要在多台服务器上保存对象和引用。副本在提供某个引用的新值之前，必须能够提供从该值可达的对象。该服务还需要协调相互竞争的引用更新，也就是说，多个推送试图将同一分支从相同的旧提交指向不同的新提交。对仓库中的所有写入操作强制采用单一顺序的系统，可能会导致对互不相关分支的推送彼此等待。Replicating a Git repository means keeping both objects and references available across multiple servers. Before a replica can serve a reference’s new value, it must be able to supply the objects reachable from that value. The service also needs to coordinate competing reference updates, i.e., pushes that try to move the same branch from the same old tip to different new commits. Systems that impose a single order on all writes to a repository can make pushes to unrelated branches wait on one another.

在设计 Pando 时，我们评估了使用 Git 扩展源代码管理规模的传统方法：一种是通过三阶段提交（3PC，一种用于协调多个节点提交的协议）来协调仓库副本之间的引用更新；另一种是通过 Blob 存储中的共享预写日志（WAL，即在应用更改之前记录这些更改的日志）来对更新进行排序。While designing Pando, we evaluated conventional approaches for scaling source control using Git: one that coordinates ref updates across repository replicas using three-phase commit (3PC, a protocol for coordinating a commit across multiple nodes), and one that orders updates through a shared write-ahead log (WAL, a record of changes written before they’re applied) in blob storage.

在我们建模的三阶段提交设计中，Git 服务会在多个副本上保存完整的仓库副本。任何服务器收到 push 时，都会复制传入的 packfile，并将其并行发送给所有副本。要发布此次 push，必须有多数副本接受 ref 更新。负责管理这一过程的服务器称为协调者，它需要与其他每个副本进行多轮通信。In the three-phase commit design we modeled, the Git service keeps a complete copy of the repository on several replicas. When any server receives a push, the incoming packfile is copied and sent to all replicas in parallel. To publish the push, a majority of replicas must accept the ref update. The server managing this is the coordinator, and it requires multiple rounds of communication between the coordinator and each other replica.

要接受引用更新，每个副本都会锁定其引用副本，并检查它是否仍指向客户端预期的提交。然后，协调器向接受更新的副本发送预提交消息。多数副本确认收到预提交消息后，协调器便通知它们执行更新。随后，任何数据已同步的副本都可以从其本地仓库副本中公布新的分支顶端提交。To accept a reference update, each replica locks its copy of the ref and checks that it still points to the commit the client expects. The coordinator then sends a pre-commit message to the replicas that accept. Once a majority acknowledges the pre-commit message, the coordinator tells them to perform the update. Any up-to-date replica can then advertise the new branch tip from its local copy of the repository.

保留完整副本可以在本地处理读取请求，并确保无论客户端选择哪个副本，都能看到一致的所有引用。因此，引用会逐个更新，而且每次推送都需要多次往返通信，以协调各副本之间的引用更新。Keeping complete replicas makes it possible to serve reads locally, and guarantees that clients get a consistent view of all references no matter which replica they pick. As a consequence, references are updated one at a time, and each push requires multiple round trips to coordinate the ref update among replicas.

等待法定人数会使副本磁盘性能和通信延迟成为推送路径上的制约因素。随着写入量增长，副本必须既跟上不断到来的 pack，也应对读取流量。Waiting for a quorum puts replica disk performance and communication latency on the push path. As write volume grows, the replicas must keep up with incoming packs as well as read traffic.

在我们建模的基于 WAL 的设计中，Git 服务使用 blob 存储中的共享清单来确定仓库内各项更新的顺序。服务器收到推送后，会将传入的 packfile 上传到 blob 存储，并在本地仓库副本中准备 ref 更新。要发布这次推送，服务器会在存储于 blob 存储中的共享清单里添加一条记录，记录 ref 更新并指向该 packfile。随后，Git 服务器使用比较并交换操作更新清单。此操作会原子地替换清单，否则就拒绝此次更改。如果期间有其他写入者更新了清单，服务器会读取其间新增的条目，检查自己的 ref 更新是否仍然有效，然后使用已上传的 packfile 重试。In the WAL-based design we modeled, the Git service uses a shared manifest in blob storage to order updates across the repository. When a server receives a push, it uploads the incoming packfile to a blob store and prepares the ref update in its local copy of the repository. To publish the push, it adds an entry to a shared manifest stored in the blob store that records the ref update and points to the packfile. The Git server then updates the manifest using a compare-and-swap operation. This atomically replaces the manifest, or otherwise rejects the change. If another writer has updated it in the meantime, the server reads the intervening entries, checks that its ref update is still valid, and retries using the packfile it already uploaded.

写入者可以并行上传各自的 pack 文件，但即使是推送到互不相关的分支，也需要更新同一个清单。一次成功的比较并交换操作所需的往返时间，限制了仓库发布新清单的频率。低延迟的 Blob 存储可以加快这一步。批处理可以让一次清单更新同时处理多个引用，但代价是需要对更新排队并增加延迟。Writers can upload their packfiles in parallel, but even pushes to unrelated branches require updating the same manifest. The round-trip time for a successful compare-and-swap limits how often the repository can publish a new manifest. A low-latency blob store makes this step fast. Batching lets one manifest update several refs at the same time, at the cost of queueing updates and increasing latency.

Blob 存储可以并行接收 packfile，但对共享清单的更新仍会相互竞争。单纯增加 Blob 存储的容量并不能消除这一瓶颈。每个副本仍需从 Blob 存储读取并索引 pack，然后才能在本地提供服务。如果写入速度超过了这项工作的处理速度，副本就会落后，客户端也需要等待更长时间才能获取新推送的提交。The blob store can receive packfiles in parallel, but updates to the shared manifest still compete with one another. Increasing the blob store’s capacity alone doesn’t resolve that bottleneck. Each replica still needs to read and index packs from the blob store before it can serve them locally. If incoming writes outpace that work, replicas fall behind and clients wait longer to fetch newly pushed commits.

这就引出了 Pando。Pando 是一种用于提供 Git 仓库服务的新架构，旨在扩展到支持许多客户端同时推送。它的工作原理如下：That brings us to Pando. Pando is a new architecture for serving Git repositories with the goal of scaling to many clients pushing simultaneously. Here’s how it works:

在开发 Pando 之前，我们隐约觉得可以在不牺牲 Git 的前提下满足我们的需求，但确实需要对大多数 Git 系统默认视为必需条件的一些事项作出取舍：Prior to developing Pando, we had a hunch that we could meet our requirements without compromising on Git, but we did need to compromise on some of the things that most Git systems assume to be requirements:

• 全仓库一致性：要提供所有引用的统一时间点快照，需要在整个仓库范围内进行协调。在将所有引用更新统一排序的系统中，推送到互不相关的分支也会争用同一个协调点。

  Repository-wide consistency: Providing a consistent, point-in-time snapshot of all refs requires coordination across the repository. In a system that orders all ref updates together, pushes to unrelated branches contend on a single coordination point.

• 每个客户端都需要唯一的 packfile：如果两个客户端的本地历史记录不同，却获取同一个分支，服务器可能需要为每个客户端生成不同的 pack，以避免发送它们已有的对象。生成这些响应会消耗 CPU 和磁盘 I/O，因此难以扩展到大量读取者。

  Unique packfiles for every client: If two clients with different local histories fetch the same branch, the server may need to generate a different pack for each to avoid sending objects they already have. Creating these responses costs CPU and disk I/O, making it difficult to scale to high volumes of readers.

相反，Pando 采用较宽松的一致性模型（因果一致性）来扩展写入能力，并复用 packfiles 来扩展读取能力。Instead, Pando provides a relaxed consistency model (causal consistency) to scale writes, and reuses packfiles to scale reads.

客户端向某个分支推送时，Pando 会将收到的包文件作为不可变 blob 存储在 blob 存储中。每当分支发生变动，Pando 都会在元数据存储中记录一个引用版本，其中包含新的分支头、提供该分支头的包，以及该包所依赖的所有较早的引用版本。blob 存储和元数据存储都支持水平扩展。When a client pushes to a branch, Pando stores the incoming packfile as an immutable blob in a blob store. Each time the branch moves, Pando records a ref version containing the new tip, the pack that supplied it, and any earlier ref versions the pack depends on in a metadata store. Both the blob store and metadata store are horizontally scalable.

从 Pando 获取提交或引用时，需要向客户端重放一组引用版本（packfile），以重建创建所获取引用版本的客户端的状态。这在整个仓库范围内提供了较弱形式的一致性，同时保证对单个引用的更新具有可串行化性。只有在客户端同时向同一个引用推送时，才会发生冲突。读取多个分支的客户端不能假定这些读取结果描述了仓库在同一时刻的状态。Fetching a commit or reference from Pando requires replaying a set of ref versions (packfiles) to the client, reconstructing the state of the client that created the ref version being fetched. This provides a weaker form of consistency across the entire repository, while providing serializability for updates to individual refs. Clients only conflict when pushing to the same ref simultaneously. A client reading several branches cannot assume those reads describe a single point in time across the repository.

这是一种因果一致性形式。为了获取某个引用，读取者会观察生成该引用的写入者的历史记录。在推送过程中，Pando 控制客户端与服务器共同认可的共享历史，确保客户端生成更易于共享的 packfile。This is a form of causal consistency. In order to fetch a reference, the reader observes the history of the writer that produced it. During a push, Pando controls what shared history the client and server agree on, ensuring clients produce more easily shared packfiles.

因此，Pando 不会在 Git 操作期间提供所有引用在某一时点的状态，也不会提供克隆或获取过程中引用更新的顺序。具体来说，假设在完整克隆的协商过程中，分支 foo 和 bar 都发生了更新。在协商期间，一个客户端可能看到 foo 处于版本 1、bar 处于版本 2，而另一个客户端则看到 foo 处于版本 2、bar 处于版本 1。实际上，用户和代理并不关心引用更新的全局顺序，而是依赖于单个引用的更新顺序。As a consequence, Pando doesn’t provide a point-in-time state of all references during Git operations or an ordering of ref updates within a clone or fetch. Concretely, suppose that branches foo and bar are both updated during the negotiation of a full clone. During the negotiation, one client may observe foo at version 1 and bar at version 2, while another client observes foo at version 2 and bar at version 1. In practice, users and agents are agnostic to the global ordering of ref updates, and instead depend on the order of updates to a single reference.

在对这三种一致性模型进行的模拟中，我们配置了 1,000 个客户端，让每个客户端选择一个不同的分支，然后反复推送唯一的提交，并测量总实际耗时以及从推送到获取的延迟。后者是工作负载的一项重要指标，因为它衡量了用户和代理协作，或使用其他依赖版本控制的系统时的往返时间。该模拟涵盖了从普通到高端的虚拟机和 Blob 存储系统。在此模拟中，我们放宽了三阶段提交的仓库级一致性要求。In this simulation of the three consistency models we configure 1,000 clients to each select a distinct branch and then repeatedly push unique commits, measuring total wall clock duration and push to fetch latency. The latter is an important metric for workloads, as it measures round trip time for users and agents to collaborate or to use other systems which rely on version control. The simulation models hardware ranging from commodity to premium virtual machines and blob storage systems. In this simulation, we’ve relaxed repository-wide consistency for three-phase commit.

建模的三阶段提交系统在负载下，随着磁盘趋于饱和，写入延迟会显著增加；与此同时，预写日志系统也因同样的原因出现显著更高的读取延迟，因为服务器难以跟上从日志中读取数据的速度。在我们对单体代码库进行的真实环境测试和这项模拟中，Pando 都能支持数千个客户端并发推送，且在负载下不会影响性能。The modeled three-phase commit system shows significantly greater write latency under load as their disks saturate; meanwhile, write-ahead log systems show significantly greater read latency for the same reason, as servers struggle to keep up with reading from the journal. In both real world testing on our monorepo and in this simulation, Pando supports many thousands of clients able to push concurrently without impacting performance under load.

处理推送时，Pando 会在将传入的 packfile 流式传输到 Blob 存储的同时进行验证，而不会将其解包到本地仓库中。客户端开始获取时，Pando 会选出读取方所需的已存储 pack，然后由读取方下载这些 pack。服务器无需将 packfile 应用到本地仓库，也无需执行开销巨大的解包和打包操作。由于这些服务器主要负责协商和选择 packfile，Pando 服务器可以运行在普通硬件上。On handling a push, Pando validates the incoming packfile while streaming it to blob storage, without unpacking it into a local repository. When a client begins a fetch Pando selects the stored packs the reader needs, and the reader downloads them. No server needs to apply packfiles to a local repository or perform costly unpack and packing operations. Because these servers primarily perform negotiation and packfile selection, Pando servers can run on commodity hardware.

这使 Pando 能够扩展到客户端、客户端与 Pando 之间的网络以及 blob 存储服务各自的容量上限。在容量充足时，处理独立分支的客户端都可以按各自连接所允许的速度传输 pack。为了以这样的速度为更多客户端提供服务，我们可以增加 Pando 服务器，并相应扩展元数据存储和 blob 存储。This allows Pando to scale to the limits of clients, the network between clients and Pando, and the blob storage service. With enough capacity, clients working on independent branches can each transfer packs as fast as their own connection allows. To serve more clients at that rate, we can add Pando servers and scale the metadata store and blob store alongside them.

代理编辑文件并在本地创建提交，然后运行 git push；这会生成一个包含这些更改的 packfile，用于更新分支。Pando 接收该 packfile 并将其流式传输到 blob 存储中。通过单遍解码验证传入的推送后，Pando 使用多键事务在我们的元数据存储中记录一个 ref 版本，并以原子方式更新 ref 指针。这使 Pando 能够提供“读己之写”一致性。整个写入操作只需要 O(1)O(1) 次元数据读取和写入。An agent edits files and creates a commit locally and runs git push, which creates a packfile of those changes to update a branch. Pando receives the pack and streams it to blob storage. After validating the incoming push with a single pass decode, Pando uses a multi-key transaction to record a ref version in our metadata store and atomically update the ref pointer. This enables Pando to provide “read-your-own-write” consistency. This entire write operation requires O(1)O(1) metadata reads and writes.

当另一个客户端获取该分支时，Pando 会根据客户端已有的历史记录选择一组已存储的 pack。这项操作需要 O(log⁡Δ)O(\log \Delta) 次元数据读取，其中 Δ\Delta 是客户端当前拥有的 ref 版本与其所需版本之间的最短距离。服务器会返回 packfile URI，客户端通过名为 Express 的服务下载这些文件；该服务会将 packfile 存储在区域缓存中。When another client fetches the branch, Pando selects a set of stored packs using the history the client already has. This operation requires O(log⁡Δ)O(\log \Delta) metadata reads, Δ\Delta being the shortest distance in ref versions between what the client has and what the client wants. The server returns packfile URIs, and the client downloads them through a service, Express, that stores packfiles in regional caches.

这一设计依赖于 Pando 架构中的三项核心服务：推送、获取和压实。This design relies on the three central services of Pando’s architecture: pushes, fetches, and compaction.

当客户端推送时，传统 Git 服务器就像一个亮出所有底牌的扑克玩家，把它拥有的每个分支都展示给客户端。随后，客户端会收集提交、树和 blob，创建一个打包文件，并排除它知道服务器已经拥有的对象。When a client pushes, a conventional Git server is like a poker player showing all of its cards, revealing to the client every branch it has. The client then gathers the commits, trees, and blobs to create a packfile that excludes objects it knows the server already has.

通告更多 refs 可以让 push 变小。但这也可能让 pack 更难复用，因为另一个客户端可能缺少写入方省略的对象。Advertising more refs can make the push smaller. It can also make the pack harder to reuse, because another client may be missing objects the writer left out.

Pando 采取了不同的方法。它不会向客户端展示全部分支，而只会公布一组选定的分支。这样做有两个原因：Pando takes a different approach. Instead of showing its full hand, it only advertises a select set of branches to the client. This is done for two reasons:

1. 为每个 ref 提供一致视图的成本很高。分支数达到数百万时，仅 ref 广告数据就可能超过 100 MiB。要实现这一点，每台服务器要么维护一份最新映射（这需要大量协调开销），要么执行一个查询量随 ref 数量增长的数据库查询。

   Providing a consistent view of every ref is expensive. With millions of branches, the ref advertisement alone can exceed 100 MiB. Providing this requires each server to either maintain a current mapping, requiring significant coordination overhead, or a database query that scales with the number of refs.

2. 通过只向客户端提供部分 refs，客户端生成的 pack 将仅依赖于服务器明确允许客户端排除的内容。跟踪这些依赖项后，Pando 就能复用客户端在一次推送中生成的 packfile，并在后续获取时将其作为响应提供。

   By providing only a subset of the refs to the client, the client will generate a pack that depends only on what the server explicitly allows the client to exclude. Tracking the dependencies allows Pando to reuse the packfile a client generates in one push by serving it as the response to a subsequent fetch.

Git 服务器公布的引用越多，推送的客户端就有越多可用于排除对象的来源，因此能够生成更小的 pack。要复用这些 packfile，服务器必须能够证明，获取内容的每个客户端都拥有所有依赖对象。否则，packfile 可能会省略客户端所需的对象。As a Git server advertises more refs, clients pushing have more sources from which to exclude objects, enabling them to produce smaller packs. However, reusing these packfiles requires the server to be able to prove the clients fetching have each of the dependencies. Otherwise, the packfile may omit objects the client needs.

另一方面，如果 Git 服务器完全没有通告任何引用，客户端就会生成一个包含仓库完整历史记录的包文件。任何其他客户端都可以使用该包文件，但代价相当高：获取任何分支实际上都等同于完整克隆。On the other hand, if a Git server advertises none at all, the client would generate a packfile containing the repository’s full history. That packfile can be used by any other client, but at the significant cost of making fetching any branch effectively a full clone.

默认情况下，当客户端推送新的功能分支时，Pando 只会将指定的主干分支作为基准进行通告。这样一来，即使仓库中的分支不断增加，通告的大小也能保持不变，但也会带来一些取舍。如果工程师多次向同一个分支推送，或者将一个功能分支叠加在另一个功能分支之上，客户端就必须重新发送主干分支以外的所有内容。By default, when a client pushes a new feature branch, Pando advertises only a designated trunk branch as a base. This keeps the advertisement the same size as the repository gains branches, but comes with tradeoffs. When an engineer either pushes to a single branch many times, or stacks one feature branch on top of another the client has to resend everything outside of the trunk.

这会导致后续的分支推送包越来越大，其中可能包含冗余对象，并占用额外的客户端带宽。在我们的 monorepo 中，以这种方式生成的 packfile 中位数不到 20 KiB，第 99 百分位数不到 3 MiB。对我们来说，这是一个合理的权衡，因为已经拥有所宣告的主干历史的读取方可以通过这一个 packfile 获取该分支。This causes successive branch pushes to be larger, containing potentially-redundant objects, and uses additional client bandwidth. In our monorepo, the median packfile produced this way was under 20 KiB, and the 99th percentile was under 3 MiB. For us, this was a sensible trade-off because a reader that already has the advertised trunk history can fetch the branch with that one packfile.

选择要向客户端通告的 refs 集合后，Pando 会接受此次推送并更新其元数据存储。Pando 将元数据更新拆分为同步和异步两部分。同步元数据更新可确保客户端立即获取到最新更新的提交。此过程会写入一个新的 ref 版本，并将 ref 指针映射更新为指向该版本，同时存储该版本的新 tip、用于协商 packfile 的依赖项，以及指向已推送 packfile 的链接。After choosing the set of refs to advertise to the client, Pando accepts the push and updates its metadata store. Pando splits the metadata update into synchronous and asynchronous portions. The synchronous metadata update ensures clients can immediately fetch the newly updated commit. This writes a new ref version and updates the ref pointer mapping to that new version storing that new tip, the dependencies used to negotiate the packfile, and a link to the packfile that was pushed.

与此同时，异步工作也会记录在同一事务中，例如安排压缩。对于非常大的推送，只有部分提交会以同步方式写入。这样，无论推送规模多大，我们都只需对元数据存储执行固定次数的读取和写入。At the same time, asynchronous work is recorded in the same transaction, such as scheduling compaction. For very large pushes, only a subset of commits are written synchronously. This allows us to require only a constant number of reads and writes to the metadata store, regardless of the size of the push.

重要的是，Pando 服务器可以独立接受对互不相关的 refs 的推送。唯一存在争用的键是 ref 指针，我们通过类似原子比较并交换的操作对其进行保护。我们可以通过增加 Pando 服务器、元数据存储扩展单元或 blob 存储来提高写入吞吐量。Importantly, Pando servers can accept pushes to unrelated refs independently. The only contended key is the ref pointer, which we guard using an atomic compare-and-swap-like operation. We can scale up write throughput by adding Pando servers, metadata store scaling units, or blob stores.

客户端执行 fetch 时，会向服务器提供两组提交 ID，在协议中称为 haves 和 wants。在传统的 Git 服务器中，服务器会计算 haves 可达的对象集合与 wants 可达的对象集合之间的差集，将其压缩后发送给客户端。虽然开源 Git 服务器在本地使用了各种索引来加速这一操作，但计算这些包文件的成本可能很高，而且通常难以缓存。When a client performs a fetch, it provides the server with two sets of commit IDs, known in the protocol as haves and wants. In a conventional Git server, the server computes the set difference among the objects reachable by the haves and those reachable by the wants, compresses them, and sends the result to the client. While the open-source Git server has a variety of indexes locally to accelerate this operation, these packfiles can be expensive to compute and are generally resistant to caching.

相反，Pando 会利用推送时记录的提交映射，查找客户端已有的提交（haves）和所需的提交（wants），并将它们解析为包含这些提交的一组 ref 版本。每个 ref 版本都链接到 Pando 先前收到的一个 packfile。服务器据此确定客户端需要什么，并选择能够涵盖客户端所需对象的 packfile。Instead, Pando uses the commit mappings recorded on pushes to look up the client’s haves and wants, resolving them to a set of ref versions that contain those commits. Each of those ref versions links to a packfile previously received by Pando. The server uses this to solve for what the client needs, selecting the packfiles that cover the set of objects the client needs.

在 clone 或 fetch 操作中，Git 协议只允许在 HTTP 响应中发送一个 packfile。不过，Git 协议中一个鲜为人知的特性 packfile-uri，允许服务器返回链接，以便单独下载其他 pack。Pando 完全依赖这一特性，并在响应中内联传输一个空 packfile。为了支持这一特性，OpenAI 向上游贡献了多项针对 packfile-uri 的改进。The Git protocol only allows a single packfile to be sent in the HTTP response on a clone or fetch. However, an under-appreciated feature of the Git protocol, packfile-uri, allows a server to return links to download additional packs separately. Pando uses this exclusively, and transmits an empty packfile inline. To support this feature, OpenAI contributed a number of improvements to packfile-uri upstream.

OpenAI 系统运行在多个地理区域，因此我们在每个区域都维护着带有本地缓存的 Pando 集群。一个名为 Express 的 pack 交付服务会缓存我们的 blob 存储，另一个独立服务则缓存元数据存储。这些缓存使 Pando 能够完全根据集群中的缓存状态，处理对某个 commit ID 的请求，或对某个 ref 的特殊读取。我们使用一致性哈希将 packfile 请求路由到 Express 服务器，缓存命中率超过 99%。OpenAI systems operate in many geographic regions, so we maintain Pando clusters with local caches in each. A pack delivery service, Express, caches our blob store, and a separate service caches the metadata store. These caches enable Pando to serve requests for a commit ID or a special read of a ref entirely from the cached state in that cluster. We use consistent hashing to route requests for a packfile to Express servers, with cache hit rates of over 99%.

这套缓存系统提供了极高的吞吐量，足以使客户端与 Pando 之间的网络链路达到饱和。通过使用这些 packfile URI 和本地缓存，我们验证了仅使用标准云服务商的虚拟机，克隆和获取操作就能使 10 Gbps 网络链路达到饱和。This system of caches provides extremely high throughput, capable of saturating the network limit between clients and Pando. Using these packfile URIs and local caches, we’ve validated that using only standard cloud provider virtual machines, we see clone and fetch operations saturate 10 Gbps network links.

对于大多数 Git 服务器而言，随着仓库不断增长，重新打包和垃圾回收对于维持性能是必不可少的。这两个步骤都能回收空间，但会消耗大量计算资源和 I/O。For most Git servers, repacking and garbage collection are necessary to maintain the performance of a growing repository. Both of these steps reclaim space, but are compute- and I/O-intensive.

在像 OpenAI 的 monorepo 这样快速变化的代码仓库中，为自上次 fetch 以来发生的每次推送都发送一个单独的 packfile URI 是不可行的。客户端每天都需要下载数万个 packfile，才能跟上主干的最新进度。In a fast-moving repository like OpenAI’s monorepo, it is infeasible to send an individual packfile URI for every push that has occurred since the last fetch. Each day, a client would need to download tens of thousands of packfiles to catch up their trunk.

相反，Pando 会在后台合并 pack，以降低这部分开销，并采用了类似于日志结构合并树（LSM tree）压缩的设计。每次推送都会生成一个 Pando 称为“L0”的 packfile，它代表 Pando 能够传输的最小数据单元。随着时间推移，后台工作进程会将这些 packfile 收集起来，形成覆盖一系列 ref 版本的覆盖层，并从 L0 开始按几何级数扩展。目前，我们的单体仓库最多使用 L4 packfile 覆盖层。Instead, Pando combines packs in the background to reduce this overhead, adopting a design similar to compaction in log-structured merge-trees. Each push produces what Pando refers to as an “L0” packfile, representing the smallest unit of data Pando can transmit. Over time, background workers collect these packfiles into covers spanning a sequence of ref versions, scaling geometrically from L0. Today, our monorepo uses up to L4 packfile covers.

这些覆盖结构有双重作用。当某个覆盖结构积累到足够多的内容后，压缩工作进程就会接管它，并生成一个单独的 packfile。通过重新组织 packfile 中的对象，这样可以提高压缩率；同时，它还能充当引用版本上的跳表，从而提升获取协商的性能。通过同时扫描各层的索引，Pando 可以在对数时间内选出一组要发送给客户端的 packfile。无论这些 packfile 覆盖结构是否经过重新打包，都能让获取规划更加高效，降低 Pando 对吞吐量和延迟的要求。These covers serve a dual purpose. After a cover is sufficiently full, a compaction worker will claim it and produce a single packfile. This improves the compression ratio by reorganizing the objects inside the packfile. But it also improves fetch negotiation performance by acting as skip list over ref versions. By scanning through indexes at each level simultaneously, Pando can select a set of packs to send to a client in logarithmic time. These packfile covers, repacked or not, enable a more efficient solution to fetch planning, reducing both throughput and latency requirements for Pando.

与许多 Git 服务器不同，packfile 会保留在每个层级，不会被丢弃或进行垃圾回收。每个层级的 pack 都会持续用于为客户端提供服务，具体取决于客户端在 fetch 请求中指定的 haves 和 wants。对于将 Pando 用作 API 的客户端，从 packfile 读取任意对象时，从 L0 读取的数据往往比从 L4 少，因为该对象在 L0 中不太可能被压缩。Unlike many Git servers, packfiles are kept at every level and not discarded or garbage collected. Packs at every level are used to serve clients continuously, based on the haves and wants they request in a fetch. And for clients using Pando as an API, reading an arbitrary object from a packfile tends to require reading less data from an L0 than an L4, because it’s less likely to be compressed.

Pando 中的压缩显著减少了发送给客户端的总传输量，同时使 pack 规划的增长幅度相对于客户端落后的程度呈次线性。这对于规模庞大且变化迅速的代码库尤为重要，我们预计这将成为软件开发的新常态。Compaction in Pando significantly reduces both the total transfer size sent to clients and enables pack planning to scale sub-linearly in how far behind the client is. This is especially important for large and quickly changing repositories, which we expect to be a new normal in software development.

我们使用 Codex 构建了 Pando，仅用 10 周就从项目启动推进到投入生产。Pando 的核心由一名工程师设计，由多个智能体团队实现，并由少数几名工程师部署。Pando 于今年 4 月开始承载生产流量，是 OpenAI 使用的一套更大型源代码管理工具系统的一部分。We built Pando with Codex, moving from kick-off to production, in just 10 weeks. The core of Pando was designed by a single engineer, implemented by teams of agents, and deployed by a handful of engineers. Pando began serving production traffic in April of this year, and is one part of a larger system of source control management tools used at OpenAI.

在验证 Pando 的设计时，我们采用了多种方法。我们遵循了构建更安全软件基础的原则。作为一个全新项目，我们选择使用 Rust 实现 Pando，且不依赖任何遗留依赖项。我们编写了一套一致性测试套件，让各个代理能够根据标准 Git 行为测试其更改。我们还开发了确定性模拟测试框架，用于检查系统的不变量和属性，从而帮助我们发现并缓解协商和 pack 规划中的细微竞态条件与回归问题。We employed a wide variety of approaches when validating Pando’s design. We applied our principles for building more secure software foundations⁠. As a greenfield project, we chose to implement Pando in Rust and without legacy dependencies. We wrote a conformance test suite so agents could test their changes against standard Git behavior. We also developed deterministic simulation testing harnesses allowing us to check invariants and properties of the system, enabling us to find and mitigate subtle race conditions and regressions in negotiation and pack planning.

许多模型都为这个系统做出了贡献；随着新模型陆续推出，我们便利用它们提升 Pando 的性能和可靠性。借助 GPT‑6 Astra，智能体编写了 Lean 和 TLA+，将“Pando 理论”形式化，用它来发现并修复实现中的错误，并证明系统的性质。多个持续运行的会话耗费数百小时，不断发现并修复 Pando 中的边界情况。我们还全天候运行智能体，使用具备 Daybreak 能力的模型对 Git 和 Pando 进行红队测试。Many models contributed to this system; as new models became available, we used them to improve Pando’s performance and reliability. Using GPT‑6 Astra, agents wrote Lean and TLA+ to formalize a “theory of Pando” that we used to find and fix bugs in the implementation, and prove properties of the system. Multiple continuous sessions ran for hundreds of hours finding and fixing edge cases in Pando. We also run agents around the clock to red-team Git and Pando using Daybreak-capable models.

我们最初将 Pando 部署为现有源代码管理系统的读写穿透代理。这样一来，我们就能让 Pando 与原有系统保持同步，从而在不中断服务的情况下安全地将流量切换到 Pando。Our initial deployment of Pando was as a read- and write-through proxy to our existing source control system. This allowed us to keep Pando and the previous system in sync, so we could safely move traffic onto Pando without downtime.

作为其缓存设计的一部分，Pando 还可以在代理请求之前对请求进行去重和合并。这样，我们就能在逐步将系统迁移到 Pando 的同时，确保现有工具正常运行。随着我们将更多工作负载迁移到 Pando，这些工作负载的推送会填充 Pando 的缓存，并提高我们原有后端的可靠性。As part of its caching design, Pando can also deduplicate and combine requests before proxying. This allowed us to keep our existing tools healthy as we moved systems to Pando one at a time. As we migrated more workloads to Pando, their pushes populated Pando’s cache and improved the reliability of our previous backend.

为了在使用 Pando 时优化性能，我们开始向开发者机器和 ChatGPT 桌面应用分发公开版 Git。为了将客户端迁移到 Pando，我们使用了一个封装 Git CLI 的内部工具，分阶段迁移客户端。我们还创建了许多其他工具，用于处理变更请求、代码审查，以及供自动化使用的 API。除了 Git 流量之外，我们还迁移了数百个依赖旧版 API、webhook 和仓库下载的集成。In order to optimize performance when using Pando, we began shipping our public distribution of Git to developer machines and within the ChatGPT desktop app. To migrate clients to Pando, we used an internal wrapper around the Git CLI to migrate clients in phases. We created many other tools for handling change requests, code review, and APIs for automations. We migrated hundreds of integrations that depended on legacy APIs, webhooks, and repository downloads alongside Git traffic.

Codex 本身也帮助开发者和团队完成了迁移。我们创建了用于准备现有工作副本的技能，并指导智能体使用新系统。因此，开发者只需让 Codex 迁移他们的开发环境和服务即可。Codex itself helped developers and teams make the move. We created skills for preparing existing checkouts, and for guiding agents to use the new system. As a result, developers could simply ask Codex to migrate their development environments and services.

切换到 Pando 后，合并可靠性得到提升，推送的中位延迟降低了近 80%。Switching to Pando improved merge reliability and reduced the latency of the median push by nearly 80%.

Pando 仅用少数几个 CPU 核心，每天就能传输 PB 级的 Git 数据。Pando 曾在单小时内提供超过 57 TB 的 Git pack 数据，其中 99% 以上的字节来自本地磁盘缓存。Pando can deliver a petabyte of Git data per day with just a handful of CPU cores. Pando has served more than 57 terabytes of Git packs in a single hour, with more than 99% of those bytes from local disk cache.

复用 pack 文件可以让许多客户端下载同一份仓库历史记录，同时几乎不增加服务节点的计算负担。这个示例展示的是单个仓库，但 Pando 的设计可支持更多仓库。随着需求增长，我们可以分别扩展数据传输和 Git 协商能力。Reusing packs lets many clients download the same repository history with little computation on the serving nodes. This example shows a single repository, but Pando is designed to support many more. As demand grows, we can scale data delivery and Git negotiation independently.

基于 Git 协议和客户端构建 Pando，帮助我们发现了漏洞，也找到了提升 Git 自身性能的机会。Building Pando on the Git protocol and client helped us identify bugs and opportunities to improve Git’s own performance.

例如，Pando 会直接向客户端提供可重复使用的数据包，因此下载并索引这些数据包成为提升系统速度和可靠性的核心工作。这项工作暴露了并发下载、中断传输后的清理，以及恢复已完成下载等边缘情况。我们修复了这些问题，在获取数据包时增加了对现有数据包的验证，并将相关改进贡献回上游。For example, Pando serves reusable packs directly to clients, so downloading and indexing those packs became a central part of making the system fast and reliable. That work surfaced edge cases in concurrent downloads, cleanup after interrupted transfers, and resuming downloads that had already completed. We fixed those paths, added validations when accepting an existing pack for a fetch, and contributed that work back upstream.

我们还需要了解时间和带宽都花在了哪里，因此为 packfile-uri 下载添加了追踪，并统计了 pack 生成的字节数，从而更容易判断一次缓慢的获取操作究竟是耗时在生成 pack，还是传输 pack。We also needed to see where time and bandwidth were going, so we added tracing for packfile-uri downloads and byte counts for pack generation, making it easier to tell whether a slow fetch was spending its time generating packs or transferring them.

在 OpenAI 的 Git 构建中（在新窗口中打开），客户端可以并行下载和索引彼此独立的 pack。我们引入了“显式 haves”的概念，让服务器能够单独公布某个分支当前的最新提交，而推送客户端可能会将相关历史排除在其 pack 之外。在 Pando 上，这使客户端能够使用 `--force-with-lease` 检查功能分支是否发生过变动，并仅基于 trunk 构建 pack。任何拥有该 trunk 历史的人随后都可以使用同一个 pack 获取该分支。In OpenAI’s Git build⁠(opens in a new window), the client can download and index independent packs in parallel. We added the concept of “explicit haves” so a server can advertise a branch’s current tip separately from the history a pushing client may leave out of its pack. On Pando, this enables clients to use --force-with-lease to check that a feature branch hasn’t moved and build its pack against trunk alone. Anyone with that trunk history can then fetch the branch using the same pack.

其中一些改进已合并到上游 Git 中，另有许多改进已发布在我们的公共代码仓库中。我们将继续与 Git 社区合作，让这些改进也能惠及 Pando 之外的用户。你可以在我们的公共补丁跟踪器中关注这项工作（在新窗口中打开）。Several of these improvements are already in upstream Git, and many others in our public repository. We’re continuing to work with the Git community to make these improvements useful beyond Pando. You can follow that work in our public patch tracker⁠(opens in a new window).

通过重新审视扩展 Git 仓库的传统思路，Pando 让我们能够扩展单体仓库，同时无需改变工程师和智能体已经在使用的 Git 工作流。在我们的单体仓库中，Pando 提升了并行处理能力和合并可靠性，并缩短了发布代码变更所需的时间。除了单体仓库之外，我们也在探索 Pando 在 OpenAI 内部的其他用途，包括需要大量智能体协同使用同一个 Git 仓库的工作负载。By rethinking conventional wisdom on how to scale Git repositories, Pando enables us to scale our monorepo without changing the Git workflows our engineers and agents already use. In our monorepo, Pando has unlocked a greater degree of parallelism, improved merge reliability, and reduced the time it takes to publish code changes. Beyond our monorepo, we’re also exploring other uses for Pando within OpenAI, including workloads where large numbers of agents need to collaborate on a shared Git repository.

随着工程师和智能体产出更多软件，OpenAI 以外的更多团队也会遇到我们在源代码管理方面碰到的瓶颈。目前，我们正在探索如何与用户共享让 Pando 成为可能的技术。As engineers and agents produce more software, more teams beyond OpenAI will encounter the limits we hit in source control. We’re currently exploring ways to share the technology that makes Pando possible with our users.

Pando 建立在 Git 仓库用户和运营者二十年的经验之上。我们将继续开发 Pando，并与开源 Git 社区合作，分享所有能让更广泛的 Git 用户受益的改进。Pando builds on two decades of experience from users and operators of Git repositories. We’ll continue to develop Pando while working with the open-source Git community to share any improvements that benefit Git users more broadly.