GPU 集群的有效调度

Building a cluster scheduler to prioritize high-impact research while maintaining full occupancy

在 Ai2 的 AI 基础设施团队，我们负责为研究所提供 GPU 计算能力，尤其是面向大型分布式训练工作负载。我们将这项工作视为由四项层层递进的指标构成的金字塔。On the AI Infrastructure team at Ai2, we’re responsible for providing the institute’s GPU compute capacity, specifically targeting large, distributed training workloads. We think about this task as a pyramid of four metrics that build on each other.

基础是可用性：硬件保持健康并随时可投入工作的频率。其上是占用率：可用时间中分配给特定工作负载的比例。接下来是影响力：最有价值的工作负载被选中并获得资源的频率。金字塔的顶端是利用率：在工作负载的整个生命周期内，GPU 容量被使用的比例。The foundation is availability: how often the hardware is healthy and ready for work. Above this is occupancy: the fraction of available time assigned to a specific workload. Next is impact: how often the most valuable workloads are chosen to receive resources. The capstone of the pyramid is utilization: the fraction of GPU capacity used over the lifetime of a workload.

这篇文章探讨如何提升调度决策的影响力。我们最近用一套系统取代了基于优先级的调度器，该系统包含 GPU 时间预算、分层公平份额分配和时间片合同。因此，关于每个研究项目应获得多少 GPU 时间的争论，从逐案处理的运维任务转变为透明的行政预算流程。This post is about improving the impact of our scheduling decisions. We recently replaced a priority-based scheduler with a system including GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract. As a result, we shifted the debate about how much GPU time each research project deserves from a case-by-case operational task to a transparent administrative budgeting process.

过度承诺 Overcommitting

在 Ai2，我们管理着数千块 NVIDIA H100、B200 和 B300 GPU，并将它们组成规模从 88 块到 1024 块 GPU 不等的集群。这些集群专为大规模分布式 AI 模型训练而构建，服务于约 150 名内部研究人员。他们的研究涵盖多个 AI 领域，包括 LLM 和 VLM 训练的完整模型流程、机器人强化学习（RL）仿真，以及面向科学智能体应用场景的后训练。At Ai2, we manage thousands of NVIDIA H100, B200, and B300 GPUs arranged in clusters ranging in size from 88 to 1024 GPUs. These clusters are built for large-scale distributed training of AI models, and they serve a group of about 150 internal researchers whose work covers a diverse set of AI domains, including the full model flow of LLM and VLM training, robotics reinforcement learning (RL) simulation, and post-training for scientific agentic use cases.

和许多实验室一样，我们对 GPU 计算时间的需求远远超过了供给。根据已提交的工作负载，在任何时刻，我们尚未满足的 GPU 请求量都相当于现有 GPU 数量的 2 到 3 倍。换句话说，集群中的每一个可用 GPU 小时，都有 2 到 3 个不同的研究工作负载在争抢。Like many labs, we have demand for GPU time that far exceeds supply. Based on submitted workloads, at any moment in time we have outstanding requests for 2-3x more GPUs than are available. One way to think about this is that every available GPU hour on our cluster has 2-3 different research workloads competing for it. 

过去，我们使用基于优先级的调度器，并允许工作负载选择不被抢占。每个团队都有一个并发 GPU 数量上限，适用于那些受保护、不会被抢占的工作负载。可抢占工作负载则可以在空闲 GPU 上超过这一上限。这种策略带来了可预见的问题。例如，我们发现有人会“占坑”：用户会挂起不执行任何操作的工作负载，以便在需要时连接使用。这是因为研究人员发现，自己无法以足够低的延迟启动调试工作负载，因而难以实时处理问题。我们还观察到优先级不断膨胀，最终 100% 已调度的工作负载都使用了 HIGH 优先级。这意味着低优先级的工作负载完全得不到 GPU 时间。由于是否可抢占是可选的，我们还发现，值班工程师处理工单时，大部分时间都花在协商有序关闭那些运行在已知存在维护问题的主机上的不可抢占工作负载上。Historically, we used a priority-based scheduler, and we allowed workloads to opt out of preemptability. Each team had a limit on concurrent GPUs that could be used by workloads which were protected from preemption. Preemptible workloads could exceed that limit on idle GPUs. This strategy produced predictable pathologies. For example, we observed instances of GPU “squatting” where users would park no-op workloads they could connect to when the need arose. This occurred because researchers found they could not launch debugging workloads with low enough latency to tackle problems in real time. We also observed priority inflation, where eventually 100% of scheduled workloads used HIGH priority. This meant that lower priority levels were starved of GPU time altogether. Since preemptability was optional, we also found that our on-call engineers spent a majority of their ticket response time negotiating the organized shutdown of non-preemptable workloads running on hosts with known maintenance problems.

公地悲剧 Tragedy of the commons

这些问题出现时，我们迟迟未能找出其根本原因。起初，我们为了确保最重要的工作能获得 GPU 时间，着重收紧对优先级设置方式的控制；最终，我们则绕过基于优先级的调度器，明确将 GPU 独占分配给重要项目。尽管一开始我们并未意识到这一点，但我们实际上搭建了一个观察“公地悲剧”的绝佳实验场。个人都在争夺稀缺的共享资源，而他们在追求个人利益最大化的同时，造成了非最优的整体结果，并滥用了底层资源。When these problems emerged, we were slow to identify their root causes. Our initial attempts to ensure the most important work received GPU time were focused on tighter control of how priorities were set, and, ultimately, working around the priority-based scheduler by explicitly assigning GPU monopolies to important projects. While we didn’t recognize it at first, we had built a perfect laboratory for observing the “tragedy of the commons.” Individuals were competing over a scarce, shared resource and, by seeking to maximize individual outcomes, achieving a non-optimal global result and abusing the underlying resource. 

我们并非最早观察到这类互动的人。资源分配是一个引人入胜的研究领域，融合了算法开发、经济学和系统管理。一个核心问题是，用户通常比组织更了解自己任务的价值，但他们可能有动机隐瞒这一价值，或者即使占用资源会损害整体性能，也仍然不愿放手。例如，在 2011 年介绍主导资源公平（Dominant Resource Fairness）的论文中，Ghodsi 等人讲述了这样一个故事：一家搜索公司规定，只有用户能够保证较高的利用率，才会为其任务提供专用机器。他们很快发现，“用户会在代码中加入无限循环，人为抬高利用率。”硬件虽然变了，但导致资源分配复杂化的根本问题依然存在。We were far from the first to observe this kind of interaction. Resource allocation is a fascinating research domain that mixes algorithm development, economics, and system management. A central problem is that users often know the value of their own jobs better than the organization does, but they may have incentives to hide that value or hold on to resources even when doing so hurts total performance. For example, in their 2011 paper introducing Dominant Resource Fairness, Ghodsi et al. recount an anecdote in which a search company provided dedicated machines to jobs only if their users could guarantee high utilization. They soon discovered “users would sprinkle their code with infinite loops to artificially inflate utilization levels.” The hardware changes, but the fundamental problems that make resource allocation complex persist.

预算，而非时间表 Budgets not schedules

解决公地悲剧的经典办法，是将共享资源私有化——所有者会受到激励，尽可能提高其财产的价值。我们将一组 GPU 的独占使用权分配给各团队时，已经是在采用这种做法，只不过粒度太粗。这导致 GPU 因研究工作的季节性而闲置。各团队准备开展实验和训练的时间并不相同，因此，分配独占使用权意味着总会有一些时段没有作业可运行，而另一个团队却只能等待算力。The classic solution to a tragedy of the commons is to privatize the shared resource—owners are incentivized to maximize the value of their property. When we assigned teams monopolies over sets of GPUs, we were already doing a version of this, but it was too coarse. It caused GPUs to sit idle due to the seasonality of research. Teams are ready to run experiments and training at different times, so assigning a monopoly would ensure that there would be times when no jobs were ready to execute, with another team left waiting for capacity. 

我们一直在手动求解一个背包问题，试图将不断变化的研究需求安排进固定不变的日程中。我们既想要所有权带来的激励，也想让 GPU 始终保持满负载运行。We were manually solving a knapsack problem, trying to fit dynamically changing research needs into a static schedule. We wanted the ownership incentive, but we also wanted to maintain full occupancy of the GPUs.

我们决定对所有权模型进行调整。我们没有向各团队直接分配 GPU，而是分配了一定份额的 GPU 使用时间。要预测未来的需求，就必须知道新科学实验的结果，因此无法做到精确预测。然而，如何确定各项研究工作的优先级属于战略问题，可以更轻松地提前讨论并决定。我们没有试图解决调度难题，而是让领导层以投资者的思维来决策：在工作负载尚未出现之前，根据对其潜在影响的判断，决定为每项研究工作投入多少 GPU 时间。随后，调度器便可在确定新到达工作负载的优先级时利用这些信息。We decided to iterate on the ownership model. Instead of issuing teams GPUs, we chose to allocate a portion of GPU time. Predicting demand into the future would require knowing the result of novel science experiments, so it cannot be forecast with precision. Priority across research efforts, however, is a question of strategy, and it can be more easily debated and decided in advance. Instead of trying to solve the scheduling puzzle, we enabled leadership to think like investors. Before the workloads exist, decide how to fund each research effort with GPU time based on their judgment of its likely impact. The scheduler could then use that information when prioritizing arriving workloads.

基于这一点，我们设计了一套分层系统，管理者可以按比例将 GPU 时间分配给其负责的项目和研究人员。如下图所示，这套系统可以将项目战略直接转化为有保障的 GPU 时间份额。项目 A1 知道自己有权使用总容量的 35%，无论其他地方还有多少项目在排队。With this in mind, we devised a hierarchical system where managers could proportionally allocate GPU time to the projects and researchers they were responsible for. As the diagram below illustrates, this translates program strategy directly into a guaranteed share of GPU time. Project A1 knows it has a 35% claim on total capacity, regardless of how many other projects are queuing up elsewhere.

括号中的数值表示分配给叶子项目的集群总容量。Parenthetical values represent the total cluster capacity assigned to a leaf project.

在这个系统中，每次申请 GPU 时间都必须由预算支付，否则就无法免受抢占。在旧系统中，HIGH 优先级不产生任何成本，而且不可抢占意味着团队可以无限期地占满自己的并发 GPU 配额，因此所有人都使用它们。如今，没有什么是免费的，所以任何获取 GPU 时间的伎俩，花的都是受益用户的配额。占着资源不干活的工作负载，只是在白白消耗团队预算。我们的策略是，让钻调度器空子的成本高于坦诚参与预算讨论、争取更大预算的成本。我们一直在不断改进预算审核流程，但关键要求是：研究人员要经常有机会陈述自己所需的 GPU 时间，而决策应由最了解相关权衡的管理者作出。这意味着，研究项目内部的分配决策由首席研究员作出，研究计划内部的决策由首席研究员作出，跨研究计划的决策则由首席项目经理或 CEO 作出。In this system, every request for GPU time must be funded by a budget, or it is not protected from preemption. In the old system, HIGH priority carried no cost and non-preemptibility allowed a team to fill their concurrent GPU limit indefinitely, so everyone used them. Now, nothing is free, so any trick to get GPU time draws from the benefiting user’s allocation. A squatting workload is spending team budget on nothing. Our strategy is to make gaming the scheduler more expensive than honestly engaging in the debate for a larger budget. We are constantly iterating on this budget review process, but the key requirements are that there are frequent opportunities for researchers to advocate for the time they need, and the decisions are made by managers with the most context on the tradeoffs in question. This means allocation decisions within a research project are made by a lead researcher, within a research program by a principal investigator, and across programs by a lead program manager, or by the CEO.

公平份额 Fair-share

配合这款 GPU 时间预算工具，我们构建了一个分层公平共享调度器，用于管理整个项目树中各项分配的实际占用情况。这里的算法并不新鲜——基于时间窗口的分层公平共享，其渊源可追溯到 2009 年的 Hadoop Fair Scheduler；如今，SLURM 的 Fair Tree 和 YARN 的 Fair Scheduler 仍在积极采用同样的方法。对我们而言，新颖之处在于输入数据：这棵树反映了研究项目的组织结构，而权重是由管理者设定的预算，而非静态配额。Paired with this GPU time budgeting tool, we built a hierarchical fair-share scheduler to manage actual occupancy of allocations throughout the program tree. The algorithm here is not new—hierarchical fair-share over a time window is part of a lineage that goes back to the Hadoop Fair Scheduler in 2009, and the same approach is in active use today in SLURM’s Fair Tree and YARN’s Fair Scheduler. What’s new for us are the inputs: the tree mirrors the research program structure, and the weights are budgets set by managers rather than static quotas.

调度器会跟踪一个滑动回溯窗口内的资源占用情况（默认值为 7 天），并将资源利用不足的分配组对应的工作负载排在资源利用过度的分配组之前。这样一来，在一周的时间范围内，只要各组持续提交需求充足的工作负载，我们就可以预期每组都能获得分配给它们的 GPU 时间。The scheduler tracks occupancy over a sliding lookback window (we default to 7 days) and sorts workloads from under-utilized allocations above those from over-utilized allocations. This way, over a week-long time range, we can expect every group to receive their allocated GPU time as long as they are actively submitting workloads with sufficient demand. 

“新调度器让我们感觉算力增加了 30%。使用旧调度器时，如果有些时候我们用不满完整的槽位上限，那部分算力基本上就浪费了。现在使用新调度器，如果出现这种情况，我们之后就可以突破分配上限突发运行，同时仍能看到作业快速排入调度且不会被抢占，本质上就是让我们重新拿回了这部分算力。我们的工作负载经常会突发式增长，因此这为我们带回了相当多的算力。”——Chris Clark“The new scheduler makes it feel like we have an extra 30% compute. In the old scheduler, if we had moments when we didn't need our full slot limit, that compute was basically lost. Now with the new scheduler, if that happens, we can later burst beyond our allocation limit and still see our jobs scheduled quickly and without preemption, essentially letting us reclaim that compute. Our workloads are often bursty, so this gave us a significant amount of compute back.” — Chris Clark

调度器区分两种占用。已分配占用是指工作负载计入预算的时间。这部分占用会消耗工作负载所有者的分配额度，从而影响公平份额预算的计算；在最短运行时间窗口内，这些工作负载不会被抢占。未分配占用不计入任何预算，从一开始就不受保护，并且可能被任何已分配请求抢占。这样一来，即使分配额度与需求不完全匹配，我们也能让 GPU 始终保持满负载运行，并确保团队不会拒绝免费的 GPU 算力。The scheduler distinguishes two kinds of occupancy. Allocated occupancy is time during which a workload is charged to a budget. This draws from the workload owner’s allocations, which affects the fair-share budget calculation, and these workloads are protected from preemption during their minimum runtime window. Unallocated occupancy is not charged to any budget, is unprotected from the outset, and may be preempted by any allocated request. This allows us to keep the GPUs fully occupied even when allocations don’t properly match demand and prevents teams from ever declining free GPU cycles.

调度契约 The scheduling contract

分布式训练还有一个特性，让公平分配资源变得困难：工作负载可能持续运行很长时间。训练作业通常会运行数小时、数天，有时甚至数周。作业一旦被调度，就可能在分配到的 GPU 上运行一周或更久，使其他作业无法获得预算中的运行时间。正是这一系统特性，让 GPU 占坑成为可能；也正因如此，值班工程师不得不与长时间运行作业的负责人协商，处理持续存在的维护问题。An additional feature of distributed training that makes fair resource allocation difficult is that workloads can run for a very long time. Training jobs regularly run for hours, days, and sometimes even weeks. Once scheduled, a workload could remain on its assigned GPUs for a week or more, providing no opportunity for others to receive their budgeted time. This is the system property that made GPU squatting possible. It’s also what forced on-call engineers to negotiate with long-running job owners to address ongoing maintenance issues.

为了解决这些问题，我们引入了“调度契约”。作为使用集群的交换条件，工作负载必须声明其最短运行时间，也就是取得实质性进展所需的最短占用时长。在此期间，工作负载不会被抢占。这既能保证研究人员取得进展，也赋予调度器在这部分进展落实后重新平衡资源的权利，并自动将可恢复的工作负载重新排入队列。用户也可以将最短运行时间设为零，这表示 GPU 时间不应被分配给该工作负载。这类工作负载始终可能被抢占，但它们也不产生费用，因为不会计入任何预算。To address these problems, we introduced a “scheduling contract.” In exchange for access to the cluster, a workload must declare its minimum runtime, or the shortest amount of occupancy required to make meaningful progress. During this time, a workload is protected from preemption. This gives the researcher a guarantee of progress, while giving the scheduler the right to rebalance once that progress is banked, automatically re-queueing resumable workloads. Alternatively, a user can set minimum runtime to zero, which indicates that the GPU time should be unallocated. These workloads are always subject to preemption, but they’re also free in the sense that they are not charged to any budget.

工作负载生命周期遵循以下模式：The workload lifecycle follows this pattern:

1. 提交的工作负载会指定最短运行时间，并注明是否可以恢复。
   The workload is submitted with a minimum runtime and indicates whether or not it is resumable.
2. 工作负载根据公平份额算法进行调度，其权重取决于回溯窗口内实际占用时间与分配时间的比值。
   The workload is scheduled according to the fair-share algorithm, weighted by a ratio of actual occupancy to allocated time in the lookback window.
3. 工作负载会运行至其最短运行时间，这段时间的费用将计入其分配额度。
   The workload runs for its minimum runtime, which is charged to its allocations.
4. 只要相关分配持续优先处理该工作负载，它就可以继续运行。这段时间也会计入其分配的用量。
   The workload may continue running as long as the associated allocations continue to prioritize it over others. This time is also charged to its allocations.
5. 它可能会被抢占并重新排队，然后返回步骤 2。
   It may be preempted and requeued, which returns to step 2.
6. 工作负载完成，并释放其对所有资源的占用。
   The workload completes, releasing its claim on any resources.

这些协议共同为我们的调度器加入了时间片机制。运行中的工作负载可以自动移除并重新排队，从而让公平共享逐渐趋于平衡，并抑制占坑行为。它们还允许不健康的主机在工作负载达到最短运行时间后将其逐步迁出，因此修复工作可以实现全自动化。在规划这项工作时，我们没有意识到最后这一点如此重要。它使需要人工介入的修复减少了 74%，大幅减轻了值班负担。Together, these agreements add time-slicing to our scheduler. Running workloads may be removed and requeued automatically, allowing fair-share to converge and disincentivizing squatting. They also let unhealthy hosts drain their workloads as they reach their minimum runtimes, so repair activities can be fully automated. This last point was more important than we realized when planning this work. It reduced repairs requiring a human-in-the-loop by 74%, which was a massive savings in on-call toil.

模拟 Simulations

我们知道，调度策略的变更可能会带来意想不到的后果。这个问题具有零和性质，给一位研究人员分配时间，就意味着要从另一位研究人员那里拿走时间。在这种交换中处于劣势的用户往往会寻找新的变通办法。在推出基于预算的系统之前，我们希望快速预测哪些地方可能会出现更长的等待时间，并测试一些配置参数，例如回溯窗口的长度，或允许的最小运行时间上限（我们选择了 8 小时）。We know that scheduling policy changes can have unintended consequences. The zero-sum nature of the problem means that giving time to one researcher means taking away from another. Users who lose this exchange tend to look for new workarounds. Before rolling out the budget-based system, we wanted a fast way to predict where those longer wait times might arise and to test configuration knobs like the length of the lookback window or the maximum value to allow for minimum runtime (we chose 8 hours).

我们构建了一个小型模拟环境，以一组工作负载及其提交时间表为输入，让调度器据此做出抢占和 GPU 分配决策。模拟器已知每个工作负载所需的 GPU 数量和总运行时间，因此可以直接跳转到可调度的时刻，并在几秒内模拟多天，分析队列等待时间、抢占事件，以及各项目间的 GPU 时间分布。我们使用历史提交数据和自行构建的场景运行了模拟器，以便更深入地了解这些情况。We built a small simulation environment that takes a set of workloads and their submission schedule as input and allows the scheduler to make preemption and GPU assignment decisions. With the knowledge of each workload’s requested number of GPUs and total runtime, the simulator could jump ahead to schedulable moments and provide analysis of queue wait times, preemption events, and distribution of GPU time across projects for many simulated days in a few seconds. We ran the simulator against both historical submission data and constructed scenarios we wanted to better understand.

我们想要验证的一个假设涉及“调试工作负载”。这类作业只需要少量 GPU，且所需的最短运行时间为 15 分钟或更短，足以让用户确认作业能否成功启动，还是会因 bug 或配置错误而提前崩溃。我们想了解，与大型训练工作负载相比，这类作业的排队等待时间是否更短。大型训练工作负载通常需要许多 GPU，并且要运行数小时才能取得有意义的进展。直觉上，这些规模较小的作业应该能更快排到队列前面，因为小作业比大作业更容易找到可用资源。但排队延迟的具体时长很重要。只等一两分钟，就能催生一种全新的开发实践；但如果要等十分钟，这种做法就不切实际了。One hypothesis we wanted to test involved “debug workloads.” These jobs require a small number of GPUs and a minimum runtime of 15 minutes or less, which is enough for the user to see whether a job launches successfully or crashes early due to a bug or misconfiguration. We wanted to know whether these jobs would see a shorter queue wait time than larger training workloads, which often need many GPUs and hours of runtime to make meaningful progress. Intuitively, these smaller jobs should rise to the top of the queue, since a small job can fit more places than a large one. But the precise queue latency was important. A short wait of a minute or two would unlock a new development practice, but a ten-minute wait becomes infeasible.

由于我们的历史记录中没有足够多这类类似调试的工作负载，因此模拟需要手动构建测试用例数据。结果支持了这一假设：调试工作负载的 p90 等待时间从约 6 小时降至仅 5 分钟。Our simulations required hand-built test case data, because our historical record did not contain a high enough volume of these debug-like workloads. Our results supported the hypothesis, showing p90 debug workload wait times fall from about 6 hours to only 5 minutes.

基线调度器（左）和新的“allocations”调度器（右）的缩小版模拟器可视化。每一行代表一块 GPU；每个条形代表一个作业，按其父级工作负载着色，每个团队对应一种色调；斜线阴影表示作业可中断的时段，红色边缘表示发生了抢占。在基线调度器中，耗时较长的紧急作业不会被中断，而优先级较低的工作发生的抢占也较少。新调度器中，每块 GPU 上的颜色组合更加多样，展现了各团队之间的资源占用轮换。Smaller-scale simulator visualization of the baseline (left) and the new “allocations” scheduler (right). Each row is one GPU; each bar is a job, colored by parent workload with one color hue per team; hatching marks time where a job is interruptible, and a red edge marks a preemption. In the baseline, long urgent jobs are never interrupted, and fewer preemptions occur on lower-priority-level work. The new scheduler has a larger mix of colors on each GPU, illustrating occupancy rotation across teams.

结果 Results

拿到模拟结果后，我们在七月底开始逐个集群 rollout。我们关注的结果包括：我们选择资助的工作负载是否获得了所需的运行时间，新系统是否保持了满负载运行，以及研究人员是否能够理解调度器的工作方式并据此做出明智的决策。With simulation results in hand, we began a cluster-by-cluster rollout at the end of July. The results we care about are whether the workloads we chose to fund received their time, whether the new system maintained full occupancy, and whether researchers could reason about the scheduler to make informed decisions. 

自上线以来，我们观察到用户和团队始终能够获得分配给他们的 GPU 使用时长。我们将团队应得的时长定义为其分配额度按小时计算、但不超过其实际需求的部分。在为期 30 天的测试期间，团队获得了其应得 GPU 时长的 98%；15 个团队中有 13 个获得了 95% 或以上，最差的情况也达到了 90%。变更前后，集群利用率均稳定在 98%，两个时期的需求都超出容量 2 至 3 倍。已交付的 GPU 时长中有 18% 未分配给任何团队，这使我们能够在获得资金支持的用例尚未准备好运行时，仍保持较高的利用率。Since rollout, we have observed users and teams consistently receiving their allocated GPU time. We count the time owed to a team as its allocation capped hour by hour at its actual demand. Over the 30-day test period, teams were delivered 98% of the GPU hours they were owed, and 13 of 15 team allocations received 95% or more with the worst case receiving 90%. Occupancy on the cluster held steady at 98% before and after the change, with demand exceeding capacity by 2-3x in both periods. 18% of delivered GPU time was unallocated, which is how we maintained high occupancy during periods when funded use cases were not ready to run.

我们的模拟器结果经证实与实际结果方向一致，而实际表现优于预测。在新调度器下，调试工作负载的队列等待时间 p90 从 2 小时降至 30 秒；相比之下，基于手工构造的测试场景，模拟预测其将从 6 小时降至 5 分钟。值得注意的是，基线中调试工作负载的样本量较小，因此这些测量结果的方差较大。时间片轮转也带来了队列延迟整体改善这一附带效果：在我们最大的 H100 集群上，队列等待时间中位数从 5 分钟降至 24 秒，p90 等待时间则缩短了约三分之一（从 2.8 小时降至 1.8 小时）。Our simulator results proved to be directionally accurate with real outcomes overperforming our predictions. Debug workload p90 queue wait time fell from 2 hours to 30 seconds under the new scheduler, against a simulated prediction of 6 hours to 5 minutes from hand-crafted test scenarios. It’s worth noting that the smaller sample size of debug workloads in the baseline meant there was higher variance in those measurements. Queue latency in general improved as a side effect of time-slicing: on our largest H100 cluster, median queue wait time fell from 5 minutes to 24 seconds, and p90 wait time fell by about a third (from 2.8 hours to 1.8 hours).

与我们着手解决的三个问题相比：Compared against the three problems we set out to solve:

1. 占坑：短期调试工作负载不到一分钟即可启动，因此占坑的价值降低。此行为产生的成本会计入占坑者的预算，导致他们在真正需要时无法获得运行时间。

   Squatting: Short debug workloads start in under a minute, reducing the value of squatting. The cost of this behavior charges the squatter’s budget, which prevents them from receiving time when they really need it.

2. 优先级膨胀：我们仍允许工作负载声明优先级，但它只会影响团队内部的排序。我们鼓励管理者关注整个团队的优先级，以优化预算的使用。

   Priority inflation: We still allow workloads to declare priority, but it only impacts sorting within a team. Managers are incentivized to monitor priority across the group to optimize the use of their budgets.

3. 值班运维负担：工作负载达到最短运行时长后，不健康主机会自动排空。需要人工介入的修复减少了 74%。

   On-call toil: Unhealthy hosts drain automatically as workloads reach minimum runtime. Repairs requiring a human-in-the-loop fell by 74%.

挑战 Challenges

学习曲线比我们预想的更陡。我们逐步推出了这项变更，因此在初期，研究人员针对不同集群时会遇到不同的行为。此外，我们的界面仍保留了一些旧术语（例如“工作负载优先级”），但这些术语的含义已经发生变化。仅靠文档无法消除困惑。真正有效的是举办实时讲解会，为研究人员提供提问的论坛，并让工程团队结合实际案例，更深入地说明调度器如何以及为何做出优先级决策。The learning curve was steeper than we had assumed. We rolled out the change incrementally, so in the early days researchers experienced varied behavior depending on which cluster they targeted. Furthermore, our interfaces retained some old terminology (like workload priority) whose meanings had changed. Documentation alone did not resolve the confusion. What did work was conducting live explanatory sessions, providing a forum for researchers to ask questions and for the engineering team to provide deeper descriptions of both how and why the scheduler made its prioritization decisions using real examples.

这是一个关键时刻，因为它标志着局面发生了转变：此前，大家经历了一段充满挫折和民间猜测的时期；如今，研究团队则更频繁、更广泛地交流实验所需的 GPU 资源。现在，研究人员参与预算讨论时，对为满足任何新需求所做的权衡有了更清晰的认识。This was a key moment because it marked a pivot from an early period of frustration and folk theories to the current mode, where research groups communicate more often and more broadly about the GPU needs of their experiments. Researchers are now engaging in the budgeting discussion with clearer knowledge of the tradeoffs being made to accommodate any new request.

除了线下会议，我们还在产品上线后新增了可视化图表，让用户更清楚地了解分配给他们的 GPU 时长与预期配额的吻合程度，并直接展示用于对工作负载队列排序的指标。当工作负载被抢占时，用户可以通过这些图表轻松查看原因。这些图表也帮助预算负责人了解其管理的各个项目如何使用 GPU 时长。In addition to in-person sessions, we introduced new visualizations post-launch to give users a better sense of how closely their assigned GPU time is tracking their expected allocations and directly exposing the metric used for sorting the workload queue. This provided a simple place to look when a workload was preempted to understand why. These visualizations also helped budget owners, who could see how GPU time was being used across various projects under their management. 

分配使用情况随时间变化的示例可视化图。Example visualization of allocation usage over time.

并非所有使用场景都得到了改善。除了分布式训练之外，我们的研究人员还会启动交互式会话，在编写训练代码的同时进行数据分析和测试。在旧系统中，研究人员可以让此类会话持续运行长达一周。采用时间切片后，会话受 8 小时受保护运行时间上限的约束；超过分配时长后，会话就可能被抢占。我们没有充分意识到研究人员对这些会话中易失状态的依赖程度。会话被抢占意味着他们必须等待新的会话，并且还要手动重建会话状态。我们对研究人员进行了调查，以了解这个问题的影响范围，随后制定了两个新的路线图项目。我们正在投资建设一个紧邻本地存储的纯 CPU 集群，用于运行专注于数据准备任务的开发会话。这样就能把训练集群的容量留给真正需要它的工作负载。此外，我们计划为这些纯 CPU 工作负载构建可恢复的会话。这样一来，我们就能在工作负载达到最短运行时长后，因维护或时间切片而继续抢占它们，同时也能在其他位置恢复会话，无需研究人员重新构建。这样既能保留新系统在运维和调度方面的优势，也能改善用户体验。Not every use case improved. In addition to distributed training, our researchers launch interactive sessions where they perform data analysis and test training code while they write it. In the old system, a researcher could hold such a session for up to a week. With time-slicing, they were subject to the 8-hour cap on protected runtime, after which a session becomes preemptible if it exceeds its allocation. We did not appreciate the extent to which researchers were dependent on the volatile state of these sessions. Being preempted meant waiting to secure a new session and also rebuilding their state by hand. After surveying researchers to understand the breadth of this problem, we created two new roadmap projects. We are investing in a CPU-only cluster next to our on-prem storage for dev sessions focused on data prep tasks. This will preserve our training cluster capacity for workloads that truly need it. Additionally, we plan to build restorable sessions for these CPU-only workloads. This will allow us to continue to preempt workloads at the end of their minimum runtime for maintenance or time-slicing, while also being able to restore the session elsewhere without the researcher rebuilding it. We can retain the operational and scheduling benefits of this new system while also improving the user experience.

我们会继续密切关注新出现的问题。目前正在调查的一个潜在问题是容量碎片化，这可能会导致规模最大的工作负载排队等待时间变长。我们的初步判断是，最低运行时长保障机制被应用到了这样一类作业上：它们过去依赖可抢占机制，才能突破团队的 GPU 并发上限。以前，这些作业随时可能被中断，导致已运行的时间白白浪费，但同时也更容易为大型作业安排资源。现在，调度器可能不再有那么多机会一次性中断多个作业，以便为一个大型待处理工作负载腾出位置。我们目前正在使用模拟器工具复现这一问题，同时测量生产环境中的实际情况。We remain on the lookout for emerging issues. One potential problem we’re investigating is capacity fragmentation, which may result in queue wait times increasing for the largest workloads. Our intuition is that minimum runtime protection is being applied to the types of jobs that used to rely on preemptible mechanisms to exceed their team concurrent GPU limit. Previously, those jobs could be interrupted at any time, potentially wasting that time, but also making it easier to schedule large jobs. Now the scheduler may have fewer opportunities to interrupt many jobs at once to place a large pending workload. We’re currently using our simulator tools to reproduce this problem while also measuring the ground truth in production.

未来 The future

展望本文所述的调度工作之外，我们的目标是金字塔的顶峰：利用率。我们需要确保引导启动、检查点保存以及训练应用本身都尽可能高效，从而最大限度地发挥每项工作负载获得的调度时间的价值。As we look beyond the scheduling work described here, we are aiming at the capstone of the pyramid: utilization. We need to ensure that bootstrapping, checkpointing, and the training applications themselves are all done as efficiently as possible, maximizing the value of the scheduled time each workload receives.

如果你希望与研究人员紧密合作，应对这类挑战，我们鼓励你了解 Ai2 的开放工程岗位。If you’d like to tackle challenges like these in close collaboration with researchers, we encourage you to explore open engineering roles at Ai2.