DatologyAI：12 万亿词元合成数据的工程实践

简要描述Brief Description

生成 12 万亿个合成词元的经验教训——Bogdan Gaza，DatologyAILessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza， DatologyAI

目录Table of Contents

• 引言
  Intro
• 万亿 token 规模的合成数据
  Synthetic data at trillion-token scale
• 为什么使用合成数据：数据壁垒
  Why synthetic data: the data wall
• DatologyAI 的业务内容
  What DatologyAI does
• BeyondWeb 结果
  BeyondWeb results
• 改写的工作原理
  How rephrasing works
• Where we started: Slurm vs Kubernetes
• The stack: Ray, KubeRay, vLLM
• 今天在 HyperPod 上的设置
  Today's setup on HyperPod
• 一个工作流程：整理、合成、训练、评估
  One workflow: curate, synthesize, train, eval
• 规模化工程经验教训
  Engineering lessons at scale
• 瓶颈 1：S3 元数据（11 天 → 2 小时）
  Bottleneck 1: S3 metadata (11 days → 2 hours)
• 瓶颈 2：GPU 不稳定与检查点机制
  Bottleneck 2: GPU instability and checkpointing
• 瓶颈 3：跨基础设施编排
  Bottleneck 3: Cross-infrastructure orchestration
• 瓶颈 4：推理调优（吞吐量提升 40%）
  Bottleneck 4: Inference tuning (+40% throughput)
• 回顾
  Recap
• 从 300 亿到 12 万亿个合成 token
  From 30B to 12T synthetic tokens
• 我们正在招聘
  We're hiring

简介Intro

万亿级 token 规模的合成数据Synthetic data at trillion-token scale

我们开始吧。大家好，欢迎来听我的演讲。我叫 Bogdan，是 Theology 的联合创始人兼 CEO。今天我们要聊聊，在万亿 token 规模的预训练中扩展合成数据所带来的工程经验。嗯，我们来看看。那么，为什么要……We're doing this. Hey everyone, welcome to my talk. I'm Bogdan. I'm one of the co-founders and the CEO of Theology. Today we're going to talk about the engineering lessons from scaling synthetic data for trillion token scale pre-training. Um, let's see. So why do

为什么要使用合成数据：数据壁垒Why synthetic data: the data wall

我们一开始为什么需要合成数据？事实证明，受 scaling laws（缩放定律）的影响，我们某种程度上遇到了数据瓶颈：要让模型变得越来越好，提升幅度大致呈线性，就需要投入指数级增长的算力和数据。所以，我们现在的预训练规模已经非常庞大。要训练出我不知道该怎么形容、达到 Mythos 级别能力的模型，需要大量数据，而其中很多数据在网上根本找不到。因此，互联网上的数据量是有限的，我们可以用高质量的合成数据来补充。那么今天我们要聊些什么？我们会聊聊一开始该怎么做，也会简单介绍一下 Beyond Web，也就是我们的 syntheta 数据配方。we need synthetic data in the first place? Well, turns out that with the scaling laws, we kind of hit a data wall um you need to put exponentially more compute and exponentially more data to get um models that are getting kind of linearly better and better. So um we're we're at this point where pre-training runs are massive. So in order to do the the kind of the ones that actually get to I don't know the mythos level um capabilities you need a lot of data and a lot of it uh cannot be found on the web. So um internet has limited amount of data um we can supplement it with high quality synthetic data. So what are we going to talk about today? We're going to talk about um you know how do we do this in the first place? We're going to talk about a bit about beyond web which is kind of our syntheta data recipe.

我们要谈谈在 Ethology，我们过去运行万亿级……万亿级 token 合成数据任务的经验所带来的工程经验。我们最近完成了一次约 12 万亿 token 的任务，涵盖网页、数学和代码。我们要谈谈运行规模极其庞大的合成数据任务时学到了什么。然后我们还要聊聊，这件事为什么重要。好，那我们开始吧。我们在……处理合成数据的方式是】【。Um we're going to talk about the engineering lessons from our prior experience here at Ethology of running trillion tail trillion um token scale synthetic data runs. Um we recently completed one for about 12 trillion tokens. That's across web math and code. And we're going to talk about what we learned from running absolutely massive synthetic runs. Um and then we're going to talk about kind of what's the why does this matter in the first place. Um awesome. So let's get started. Um, so the way that we solve synthetic data at

DatologyAI 是做什么的What DatologyAI does

神学，也许我们先来简单介绍一下它。我们算是一种数据整理服务。可以把我们想象成你们预训练和中期训练数据集的炼油厂。我们的客户既有初创公司，也有企业，他们找到我们，嗯，是为了让我们帮助他们满足预训练和中期训练的数据需求。通常，他们手头有大量数据，嗯，他们想尽可能获得最好的数据集，而我们会帮他们找出质量最高的数据点。嗯，完成后，我们会用我们的合成数据配方处理这些数据点——这就是我们今天要聊的内容——从而生成规模极其庞大的数据集。所以，嗯，大致就是这样。theology and maybe let's start with a with a quick intro about it. We're a kind of data curation service. Think of us as the oil refinery for your kind of pre-training and mid-training data sets. We work with customers both startups and enterprises that come to us to um kind of help them with their pre-training and mid-training data needs. So in general they have a lot of data. um they want to get to the best data sets possible and we help them identify the the highest quality data points and um once we do that we pass them through our synthetic data recipe the ones that we're going to talk about today um to get absolutely massive data sets. So um that's a bit

BeyondWeb 结果BeyondWeb results

关于我们，呃，Beyond Web 是我们的合成数据方案。制作合成数据有很多种方法，而我们采用的方法相当不同。下面是一些结果，展示了我们是如何做到这一点的。这里有两张图，我们先看第一张。横轴是模型规模，以十亿参数为单位，分别是 10 亿、30 亿和 80 亿。纵轴是准确率。我们选取了 Neotron Synth、Cosmopedia、QA rap、Red Pajama 等多个数据集，并分别训练了规模为 1B、3B 和 8B 的模型，同时也使用了我们为合成数据制定的方案，我们称之为 Beyond Web。结果显示，我们的合成数据方案取得的表现与其他方法相当；有时甚至更好——以 Neotron Synth 为例，这里的 3B 数据点达到了 8B 数据点的水平。about us uh in terms of beyond web is is our synthetic data recipe. Um there's kind of many ways to do synthetic data but um kind of the the way that we approach it is um quite different. So here's a few results from from um kind of how we do this. So uh there's two plots. Maybe let's start with the first one. Um on the x-axis we have the model size and billions of parameters. Um 1 billion, 3 billion, 8 billion. And then on the y-axis we have um accuracy. So um we've taken kind of a number of data sets like Neotron Synth, Cosmopedia, QA rap, um Red Pajama, and then we've trained models at kind of 1B, 3B and 8B scale together with um kind of the the recipe we've put together for for uh synthetic data which we call beyond web. Um and what we see is that um kind of our synthetic data recipe um kind of matches and and sometimes even um kind of in this case the the 3B data point matches the 8B data point um for neatron synth.

这意味着，使用合成数据时，你可以训练规模稍小的模型，达到训练更大模型才能达到的相同性能。这就是我们这个配方质量方面的一些情况。谈到性能时也是如此。横轴表示 token 数量（单位：十亿），纵轴表示平均准确率。我们展示的结果是，使用的 token 数量分别减少约 2.7 倍和 5.3 倍，就能达到 Neotron Synth 和 Cosmopedia 的性能。因此，如果你想训练一个达到相同性能的模型，可以通过使用更少的数据，从而总体上减少计算量。从更高层面来说，Beyond Web 大致体现了我们对……的思考。So that means that you can train slightly smaller models for the same performance as you would train larger ones by using synthetic data. So um that's a bit about kind of the the the quality of of our recipe. Um same thing here when you when you talk about performance. So on the x axis we have um kind of number of tokens in billions and then the y- axis we have uh the the average accuracy. Um and what we show is that we match the the the performance of um kind of the the um Neotron synth and the Cosmopedia ones in about 2.7x and 5.3x respectively uh less tokens. So uh if you want to train a model to the same performance, you can do that um by using less data and and overall less compute. Um so at a high level what what beyond web is is is kind of how we think about

改写的工作原理How rephrasing works

我们的合成数据方案。我们做的是改写。嗯，我们可以把合成数据流程理解为：先从客户的数据集中找出质量最高的数据点，然后用各种方式改写或重组这些数据点。嗯，这些结果其实有点过时了。我想我们最初是在 2025 年夏末发布的，而现在我们的表现已经比当时好很多很多了，而且还在持续大幅改进。那么，最简单地说，如何获得合成数据呢？你需要某种目标数据集。再说一次，我们做的是改写，或者对现有文档重新构思。背后的想法是，如果你只是给模型一个提示，说“嘿，给我生成一个合成数据点”，那你最终学到的只会是模型最初训练所依据的数据分布中的那些众数。our synthetic data recipe. What we do is we do rephrasings. Um so where we we kind of see the rephrase the the synthetic data process by um kind of finding the highest quality data points in a customer's data set and then we rephrase or restructure this data points in in various ways. Um this is results are actually a bit old. I think the the origin we published this in uh in um I think late summer 25 and at this point we're we're much much better than this um and and we're continuing to improve quite a bit. So how do you get synthetic data right in the in the simplest form you have some sort of target data set again we're doing rephrasings um or reframings of existing documents. So the idea behind it is that um if you just prompt a model and say hey give me a synthetic data um data point you're going to just learn the modes of the distribution the data was was origin trained on.

所以，呃，正如之前提到的，我们的做法是先用一些现有文档来为它提供种子数据。我们会从客户的训练数据集中筛选出高质量文档，然后把它们送入我们的流水线。最简单的情况下，我们会有一个目标数据集。比如，它可以是文本，也可以是图像文本。呃，最简单的形式就是一批提示词，我们把这些提示词加在筛选出的若干文档前面。然后你会有一批 GPU，再用各种模型来生成合成数据。呃，实际操作中，你会对所需时间有一定预期。但实际情况波动要大得多，而且，要达到你希望 GPU 实现的理想吞吐量，真的很难。So uh the way that we do it as mentioned previously is that we seed it with um some sort of existing documents. So we identify high quality documents in the customer's um training data sets and then we we pass them through through our pipeline. At the simplest set, we have a a target data set. Imagine this can be text, this can be image text. Um, in the simplest forms, it is just a bunch of prompts that that we prefix to um a number of documents that we've identified. You have a bunch of GPUs and then you um kind of use various models to generate synthetic data. Um so in practice, you have some sort of um expectations around how much time this is going to take. But in practice, this is a lot more wavy and and it it really it's really hard to match the the kind of ideal throughput that you're trying to get of your um out of your GPUs.

那么，我们该如何管理这件事？So, how do we manage that?

Where we started: Slurm vs Kubernetes

好吧，我们是从哪里开始的？我们起初的情况是，合成数据并没有集成到我们的产品中；我们会在研究环境中处理这类工作，嗯，在一个运行 Slurm 的集群上进行，而这个集群独立于我们基于 Kubernetes 的核心流水线。嗯，我们采用的是双轨制：产品团队和研究团队不会使用同一套代码库。这种方式速度慢、容易出错，嗯，迭代效率有限，而且在某种程度上导致我们的 GPU 利用不足。和所有人一样，我们的 GPU 数量是有限的。总体来说，我们希望能够最大限度地利用 GPU，把训练和合成数据任务整合起来，以统一的方式运行。嗯，那么我们从哪里开始呢？总的来说，嗯，我们希望所有东西都运行在……上。Well, well, where did we start? We started where um synthetic data wasn't integrated in our product was something that we would do in a research setting uh on a cluster that that runs slurm that is away from our core kubernetes-based pipeline. Uh we had this two-track system where kind of the product and the research team would would not use the same type of codebase. This was slow, errorprone, um, limited in velocity and kind of led to underutilizing our GPUs. We have a set number of GPUs like everybody out there. And in general, we want to be able to to maximize the the the use of our GPUs by running this in a in a cohesive way both with training and synthetic data runs. Um, so where do we start? In general, um, we'd like to run everything on on

The stack: Ray, KubeRay, vLLM

Kubernetes。嗯，我们的很多数据流水线要么使用 Ray，要么使用 Spark。今天我们要讨论 Ray、Kube 和 VLM，它们算是我们合成数据库或数据流水线的核心组件。嗯，我们现在的做法是在一个 Kubernetes 集群中设置编排层，在另一个 Kubernetes 集群中运行 Spark。嗯，我们会把 Aster Tracker（我们的数据目录）保留在最初的控制集群中，由它来协调一切。嗯，所有数据都存储在 S3 中，这是我们的存储层。只要它支持与 S3 兼容的 API，我们就能运行它。所以从某种意义上说，一切都需要在 Kubernetes 中运行，因为我们的流水线会部署到各种环境中。嗯，可以是 AWS、GCP、本地部署等。Kubernetes. Um, a lot of our pipeline is either Ray or Spark. Today we're going to talk about um kind of ray and cube and vlm um those are kind of the core components of our synthetic data library um or data um kind of pipelines. So today what we do is we have an orchestration layer in one kubernetes cluster. We um run spark in another Kubernetes clusters and um we we keep the aster tracker which is our data catalog in the original control cluster where everything um kind of gets coordinated um into um all the data is stored in S3 that's our storage layer um as long as it speaks kind of S3 compatible APIs we can run it. So in some ways we everything needs to run in Kubernetes because our pipelines get deployed in all sorts of environments. Um it can be AWS, it can be GCP, it can be onrem etc.

我们想确保，嗯，我们能够在任何 Kubernetes 环境中运行它。另外，我们还有这个 H100 集群。嗯，它运行合成数据任务、训练任务、评估任务，还有其他一些任务。最初这些任务都由 Slurm 编排。嗯，在这两个集群之间移动数据是手动进行的。所以，每当你在 Slurm 中生成合成数据时，都必须把它回填到这个资产……We want to make sure that that um kind of we can run it in any um Kubernetes environment. And then separately we had this H100 cluster. Um it ran synthetic, it run training, it ran eval kind of a number of things. All of them originally orchestrated by slurm. Um moving data in between these two clusters was manual. So whenever you generated synthetic data in slurm, you had to backfill it into the the asset

今天在 HyperPod 上的配置Today's setup on HyperPod

将 tracker 接入我们的数据目录。那么我们现在的情况是怎样的？在某些方面，整体设置基本相同：我们有原来的集群，它负责控制；还有运行 Spark 作业的计算集群。现在，我们又有了另一个 H100 计算集群。就我们而言，所有东西都运行在 AWS 上，具体是在 HyperPod 上。HyperPod 有一款叫作 EKS on HyperPod 的产品。它基本上是在同一条 spine 上部署了一批 H100 集群，并由 Kubernetes 统一编排。 我们会继续按需使用计算资源运行 Spark 和 Ray 作业，包括按需使用 CPU，有时也会按需使用 GPU。特别是对于不需要 H100 的任务，我们可以采用这种方式。然后，我们会像以前那样，以类似的方式进行编排：Spark 在自己的 Kubernetes 集群中运行，而 Ray 则在同一个集群中运行，使用非 H100 的 GPU。至于需要使用 H100 节点的任务，我们会交给另一个 Kubernetes 集群处理。我们内部构建了一个作业调度器，用来编排所有这些任务。tracker into our data catalog. So where are we now? So um kind of same setup in some ways where we have the the original cluster um that that does the control. We have the compute cluster that that we run spark jobs and now we have this other H100 compute cluster in in our case we run everything on AWS. We run it on hyper pod. Hyperpod has this product called um EKS on hyper pod. So it's basically a bunch of H100 clusters on the same spine all of them um orchestrated by uh by Kubernetes. So what we do is we continue to run spark and um ray jobs on demand capacity on demand CPUs and sometimes on demand GPUs um especially for for things that we don't require H100s we can do that um and then we orchestrate it in a similar way that that we've done before where kind of spark runs within its Kubernetes cluster and then ray runs within um the same cluster for um GPUs that are not H100s and anything that we need to do for um H100 nodes we go to the this other Kubernetes cluster and we we kind of have a job scheduler that we've built internally that that orchestrates all of these things.

嗯Um

一种工作流：筛选整理、合成、训练、评估One workflow: curate, synthesize, train, eval

这样做的话，我们就可以快速开展实验。想象一下，你可以运行某种编排流程：运行一批 Spark 作业，生成一批合成数据，然后启动一个训练作业，再运行一个评估作业，并把所有这些步骤编排到同一个工作流中。这对我们的研究团队很有用，也能让研究流程和工程流程使用同样的线路、基础设施、代码和路径。好吧。那么，我们现在进展到哪一步了？嗯，呃，我和某人有个一对一会议，挺好的。我们现在的情况是，合成数据已经成为工作流中的核心部分。我们一直在做实验，也在进行跨集群调度，而且速度快多了。if we do this then this allows us to to run fast experiments. Imagine that you can run kind of um curate where you run a bunch of spark jobs, you run a bunch of synthetic data, then you go and and launch a training job and then you run an eval job and you can orchestrate all of this together in one workflow which um for our research team is is quite useful and this is puts the the the research flow and the engineering flow in in the same lines in the same infrastructure in the same code in the same path. Okay. So, um we where are we now, right? Like where Oh, look. I have a a oneonone with someone on this fine. Um where where we're at is um kind of synthetic data is is a core thing to to our workflows. Um we have we run experiments all the time. We we do this crosscluster scheduling. It's a lot more faster.

它可靠得多，而且我们可以大规模运行它。所以，呃，我们已经……It's a lot more reliable and we can run it at scale. So, um we've

大规模工程经验教训Engineering lessons at scale

最近刚完成了一次大规模的合成数据运行。呃，我们来聊聊从中学到的一些工程经验吧，对吧？那么，呃，在万亿 token 规模下，瓶颈是什么？recently done a massive um synthetic data run. Uh let's talk a bit about the the engineering lessons that that we learned from this right. So um what are the bottlenecks at at trillion token

瓶颈 1：S3 元数据（11 天 → 2 小时）Bottleneck 1: S3 metadata (11 days → 2 hours)

规模？嗯，首先，互联网规模的文本数据集其实相当有限，对吧？网上大约能找到 30 万亿个 token，上下浮动。至于图像，像 DataComp 这样的数据集大约有 128 亿张图像。对于文本，我们通常从现有的高质量数据集开始，比如 FineWeb、DCLM 等。我们可以把它们整合起来，得到大型数据集。但到了这个规模，数据通常都存储在 Parquet 或其他类似格式中，而且会有数百万个数据分区。如果你试图以很高的速率从 S3 获取所有数据，可能会受到限流。尤其是如果你在 Ray 上运行这些任务，Ray 的元数据以及对它的管理就会成为瓶颈，最终导致最糟糕的情况：GPU 闲置。scale? Um well first and foremost um internet scale um for for text data set is is quite limited right on on the web we can find about 30 trillion tokens give or take for um images there's about 2 12.8 8 billion in in things like dataccom for or for text we usually start from existing highquality data sets like fine web dcmotron um we can we can um kind of put them all together and we get this this large data sets um but at this scale um usually everything gets stored in parquet or some sort of of format you have millions of data partitions um if you're trying to fetch all of them from s3 at a high rate you might get throttled and then if you especially if you run this on ray um your your ray metadata uh is becomes and how you manage that becomes a bottleneck and that leads to the the worst thing possible which is that you get idle GPUs.

所以，我们最初看到的瓶颈是：我们使用 S3，尝试从我们的 array head 获取内容。它有某种元数据存储。这速度相当慢。如果发起一个请求需要 10 毫秒，而你要处理——我不知道——成百万上千万个请求，那时间就会累积起来。比如说，如果你最初需要获取元数据，这些操作的开销会更大，而且还有速率限制。所以，如果你需要处理数百万个请求，就会触发速率限制，接着你的工作进程就会变得很慢。于是我们粗略算了一下，要为数以万亿计的 token 填充元数据，大约需要 9 到 11 天，这显然无法接受，对吧？如果你要运行这个任务，很可能会在计算资源上花费数百万美元。So bottleneck um that that we initially seen is that we have um S3 we try to fetch things from our array head. um this has some sort of metadata store. um this is kind of quite slow and and if you do one request that takes 10 milliseconds and and um you're trying to to do I don't know millions and millions of them add I don't know let's say um if you need to do um kind of the metadata fetches originally those are more expensive um and there's also a rate limit so if you need to do millions of them you're going to get rate limited then your workers are going to be quite slow um so we kind of ran the the the back of the anvil of math just to populate the the metadata for um trillions and trillions in tokens. It would take about 9 to 11 days, which is unacceptable, right? You if you're running this, you're probably running on millions of dollars in compute.

你不能让它们就这么等着你花 11 天去获取元数据。所以，我们解决这个问题的方法是，不再直接去 S3，而是可以把请求批量处理。也就是说，不再一次处理一个请求，而是把它们全部批量处理。S3 通过其 API 支持这种方式。这样，你就可以大规模地处理这些请求：把批次的页面大小逐步增加到 S3 允许的上限，也就是每个列表大约 1,000 个请求。这样获取元数据的 API 处理时间就能缩短到大约两个小时。第一个经验是：确保你知道如何管理元数据，并确保你已经考虑过如何将 S3 存储桶的状态纳入你使用的合成数据系统，这样才能开始大规模地开展这项工作。You cannot have them um just waiting for you to fetch metadata for for 11 days. So, the the um way that we've went past this is that um instead of of kind of um kind of going to S3, you can actually do um requests that are um batched together. So, instead of doing one at a time, you can batch them all together. S3 supports this through their APIs. And then this way um you can you can do this uh at at a massive scale by kind of increasing the the um the page size of your if your batches to to kind of the limit that that S3 gives you um which is about a thousand um kind of requests in one list and that gets you to about um kind of two hours of API processing time to get your your metadata. So, first lesson, um, make sure that you know how you manage your metadata and make sure that you have, um, kind of thought through just getting the state of your S3 buckets into whatever synthetic data system that you have in order for you to, um, kind of be able to to even start doing this at scale.

所以，从 11 天缩短到了两个小时。嗯，这是第一个。So, 11 days to two hours. Uh, that's that's the first one.

瓶颈 2：GPU 不稳定性与检查点保存Bottleneck 2: GPU instability and checkpointing

嗯，太棒了。第二个问题是 GPU 不稳定。每当你运行这类大规模合成数据任务时，当然会遇到一些有问题的 GPU，对吧？这意味着机器可能会宕机，GPU 可能会故障，或者发生类似的情况。大规模运行时可能遇到的各种问题都会发生。这会导致进度丢失、计算资源浪费和输出不完整，比如只解析了文件的一半。当然，到头来你还需要更多集群时间，而这总是很难争取到的。我们在流水线中解决这个问题的方式是使用一个 Spark 作业，将提示词和需要改写的文档拼接在一起并输出。Um, awesome. So, the second one is GPU instability. So whenever you're running um these types of large scale synthetic data runs, of course you're going to hit some sort of bad GPUs, right? So that means that the machine might go down. Uh GPU might be bad. Something around those lines. Everything that you can expect when you try to run it at scale would happen. So that leads to loss progress that leads to kind of wasted compute, partial outputs, um if you if you only parse through half of the file. And of course at the end of the day you're going to uh need more cluster time which is always hard to get. Um so the the the way that we do this in in our pipeline is that we have a Spark job that outputs the the the prompts together with the the documents that we need to rephrase that it's it stitches them together.

然后我们启动一个实际能使用 GPU 的 rig 任务，对吧？问题在于，输入和输出之间存在某种一对一映射。这基本上就是我们通常组织这类任务的方式。实际上，如果你的分区相当大，就会出现这种情况：你会完成不少进度。假设一个节点上针对一个文件、一个分区的改写任务需要八个小时，而机器在这八小时的最后五分钟宕机了，那你就会浪费很多时间。所以你得考虑一下：要么设置检查点，在合成数据任务取得进展时定期停下来，把进度写回 S3；要么就把分区设得足够小，这样你就能承受这种宕机造成的损失。Um and then we start a rig job that that can actually uses the GPUs, right? Um so the the problem is that there's some sort of onetoone mapping between input and output. That's that's kind of how we we usually structure these types of jobs. Um, in practice, if your partitions are um quite large, what happens is that um you're going to do a bunch of of of um kind of progress. And then if you if if a refrazing job on one node for one file for one partition takes eight hours and the box goes down in the last five minutes of those eight hours, you're going to lose a lot of time. So you need to think about um kind of you either have to checkpoint and whenever you do progress within the synthetic data um jobs you need to periodically say okay let me stop and like flash this back into S3 or um you need to have partitions that are small enough that you can tolerate that downtime.

所以，你需要考虑分区大小，或者总体来说，采用某种检查点机制。我们两方面都做了一些：调整分区大小，使其更合适，同时也在相关场景中设置检查点。基本上，作业失败时会重试，但我们因此损失的时间很少，也符合我们的设置需求。以前作业可能会连续运行数天后失败，现在则可以通过可恢复的重试来恢复，而且效果符合预期。总结一下：恢复执行、采用某种幂等机制；设置检查点时，可以覆盖写入检查点的文件；此外，还需要某种无缝恢复机制，用于跨……So um you need to to think about either your partition size or in general kind of have some sort of checkpointing. Um what we've done is is a bit of both where we've managed to right size our partition sizes and checkpoint as well when it comes to this things. So um basically whenever um a job fails it gets retrieded but the amount of um kind of time that we lose is it's minimal and it makes sense for the setup that we have. Um so from multi-day job failures now we have recoverable retries and it it it works it works as as expected. So just to recap resumeum execution some sort of IDM potency where whenever you're checkpointing you can override the file that you're checkpointing into and then you need to have some sort of mechanism for for simless recovery um cross

瓶颈 3：跨基础设施编排Bottleneck 3: Cross-infrastructure orchestration

基础设施编排。就我们的情况而言，我们运行数据筛选的集群通常位于一个中心化的位置，而客户可能会在全球各地的不同 GPU 位置运行合成数据作业。你可能会发现，要在一个地方集中大量算力很困难，尤其是对于合成数据而言，因为 GPU 节点之间未必需要高速网络连接。这样一来，你就需要能够在不同集群中编排所有这些作业；你使用的并非单个 GPU 集群，而是多个这样的集群。问题在于，当你尝试调度这些作业时，也许能够为 Ray 作业调度 CPU 资源，但未必能在 GPU 资源池中调度相应的 GPU 资源。infrastructure orchestration. So in in our case the the the cluster where we origin run curation in general our pipeline runs in a centralized location and then customers might have synthetic data um jobs that run in various GPU locations throughout the world right you might it's going to be hard to get especially for synthetic data where you don't need necessarily fast networking in between your GPU nodes to get a lot of capacity in one place so what happens then is that you need to be able to orchestrate all of these jobs um in in various clusters so it's not just one GPU cluster that you're working with you're working with um kind of a number of them. Um the problem here is that whenever you have um and you're trying to to to schedule um these jobs is that you might be able to um kind of schedule your CPU requirements for your ray jobs but not necessarily your um kind of GPU requirements in the GPU pool.

所以你需要考虑如何以某种方式在这两者之间进行调度。现在，一旦有下一个作业进来，你有 GPU 拉取能力，但没有 CPU 拉取能力，对吧？所以一般来说，每当你尝试进行这种跨基础设施的调度或编排时，都需要考虑调度问题，也需要考虑如何有效地进行调度。而在这个例子中，正如我提到的，你最终会遇到无法有效调度的情况。此时，你就需要考虑如何妥善地处理这个问题。归根结底，这是一种装箱问题。那么我们是怎么解决的呢？设置和之前一样。我们有一个运行在不同集群中的编排器。So you need to think about kind of being able to schedule in some sort of fashion across these two things. Now um once you have the the next job coming in um you have GPU pull capacity but you don't have CPU pull capacity right so in general whenever you're trying to do this this type of cross infra or orchestration you need to think about scheduling and you need to think about um kind of being able to schedule this effectively um and in this case as I mentioned you you you will get to a point where you cannot schedule effectively so what happens then is that um you you kind of need to think about the the ways in which you do this in a in a proper fashion. So some sort of bin packing problem at the at the end of the day. Uh so how did we solve this? Um so same setup as before. Um we have this orchestrator that runs in a different cluster.

我们有一个计算集群，用来运行 Spark 和主要需要 CPU 的 Ray 作业；另外还有一个配备 H100 的集群。每当我们进行调度时，都会在这些集群中为所需的特定资源设置专用资源池。因此，如果你有 Spark 方面的需求，就可以使用一个配备 CPU 的 Spark 资源池；同时，还可以为 Ray head 设置专用资源池。考虑某种原子化调度时，要想实现最优调度，就需要同时考虑 CPU 和 GPU 需求。在这个例子中，当我们有更多 CPU 需求需要调度时，由于我们将这两类需求解耦了，就可以以更原子化的方式调度 Ray 的 head 和 worker，不再遇到之前的问题，随后一切都会按预期扩展。We have the compute cluster that runs spark and the the ray jobs that mostly need CPU and then we have the the one that has H100s. Um so whenever we schedule um we have dedicated pools for the specific resources that you need in those clusters. So if you have a spark requirements then you spark you might have a a spark pool that that has CPUs and then you dedicate special pools for um kind of the the ray head. So when you think about um some sort of atomic scheduling you need to think about both your CPU and your GPU requirements in order to do this optimally. So in this case whenever we have more CPU um requirements that we need to schedule because we've decoupled these two things um and and then we can schedule the the um head and the worker for array um in a in a more atomic fashion then we don't have the problems as as before um and then everything scales as expected.

所以，在跨基础设施编排方面，你需要考虑多个提供商，以及运行该系统的多个云平台；还需要考虑调度，并确保在为 CPU 和 GPU 资源进行调度时，尤其是针对同一类作业，采用合理的方式。最后一点是推理优化。也就是说，每当……So cross in front orchestration um you need to think about multiple providers multiple um clouds that you run this in you need to think about scheduling and u make sure that whenever you schedule CPU and GPU resources especially for the same type of job you do that in a way that that makes sense. Um last up is inference optimizations. Um so whenever

瓶颈 4：推理调优（吞吐量提升 40%）Bottleneck 4: Inference tuning (+40% throughput)

运行 VLM 或 SG Lang 时，你需要考虑作业所用的超参数，对吧？我们发现，你需要对此相当严谨，能够对参数进行基准测试和优化，并搭建一个专门针对 VLM 参数的测试框架。只需调整 VLM 和 SG Lang 的参数，就能获得很大的提升；在这个案例中，吞吐量提升了约 40%。所以我们选择了 VLM。目前我们也在测试 HGLang。我们会进行大量批量推理，这和大家熟悉的常规在线推理非常不同。但如果你通过 VLM 来编排这些任务，就需要确保批次大小合适，推测解码的设置正确，等等。you run VLM or whenever you run SG lang um you need to think about the the hyperparameters that you use for your jobs right and uh what what we found is that you need to be u quite disciplined about this and you need to be able to to benchmark this and optimize and have a harness that just focuses on the VLM parameters you can get um a lot of gains um in the case in this case about 40% throughput gains just by tweaking the flats for VLM um and SG line right so in our case we we went with VLM Right now we're actually testing HGLang as well. We do a lot of batch inference. Um which is very much different from from the the normal online one that everybody is used to. Um but whenever you you kind of orchestrate this through VLM, you need to make sure that you have the the right batch size, you have the right speculative decoding um setup, etc.

嗯，如果你对需要优化的参数进行一系列网格搜索，结果发现，在吞吐量方面可以取得相当不错的成果。所以总结一下，嗯，每当你尝试做事情时，确保……Um if you do kind of a bunch of grid searches over the parameters that you need to optimize, turns out you can get quite a lot of uh of results in terms of throughput. So to recap, um whenever you you're trying to do things, make sure

回顾Recap

你会考虑自己所运行的规模。每当作业中断时，你都有一套完善的自动恢复机制。你需要以某种原子化的方式协调 CPU 和 GPU 资源，还需要确保能够遍历各种 VLM 参数。所以，这套配置帮助我们实现的效果相当……that um you think about kind of uh the scale that you're running at. You have a good system for auto recovery whenever jobs go down. You need to orchestrate in some sort of atomic fashion across CPU and GPU resources. Um and you need to make sure that that you kind of sweep over your VLM parameters. So what what this setup helped us achieve is is quite

从 300 亿到 12 万亿合成 tokenFrom 30B to 12T synthetic tokens

很直接。嗯，在 2014 年底——抱歉，是 2024 年底，我们在 SERM 集群中只能通过规模有限的运行生成大约 300 亿个合成 token。嗯，我们对合成数据的需求大幅增加了。然后，去年年底左右，我们能够运行大规模的合成数据任务；在这个例子中，我们为网络数据生成了约 7 万亿个 token，此外还为数学和代码生成了另外 5 万亿个，服务于我们的许多客户。嗯，我们还在继续做消融实验，并寻找更好的合成 G 配方。我们正在积极推进这项工作，敬请关注。我们会发布更多内容。嗯，我认为领域覆盖非常重要。我觉得今天还没谈到的一点是：网络、多语言、数学、代码、法律数据等领域之间有什么不同？straightforward. Um and in at the end of 2014, sorry, 2024, uh we can only output around 30 billion synthetic tokens in kind of limited runs within our SERM cluster. Uh our requirements for synthetic data went up by by a bunch. And then um kind of end of last year, we were able to to run this massive uh synthetic data jobs that that um generate in this case for web about 7 trillion tokens and then for math and code another five for a bunch of our customers. uh we're continuing to ablate and and kind of search for better synthetic G recipes. Um we're actively working towards this. So keep an eye out. We'll be publishing more. And um I think domain coverage is quite important. I think something that we haven't talked today is how does this differ in between web, multilingual, math code, uh legal data, etc.

顺便说一下，我们正在招聘，嗯，遍及整个……By the way, we're hiring um throughout the

我们正在招聘We're hiring

数据基础设施、云基础设施、常规基础设施、项目经理等方面的委员会。之后联系我。嗯，感谢参与这项工作的团队，也感谢大家来听我的演讲。board for data infrastructure, cloud infrastructure, normal infrastructure, program managers, etc. So hit me up afterwards. Uh thank you for the team that worked on this and uh thanks everybody for attending my talk.