F3 Lightning：Meta 如何大规模加速广告机器学习特性投入生产

• Facebook 特征框架（F3）是 Meta 用于编写、编译和提供机器学习特征的框架，这些特征支撑着广告排序模型，而单个请求可能需要超过 20,000 个特征。我们首次在 2020 年 AI @ Scale 大会上介绍了 F3。
  Facebook Feature Framework (F3) is Meta’s framework for authoring, compiling, and serving the machine learning features underlying ads ranking models, where a single request can require more than 20,000 features. We first presented F3 at AI @ Scale 2020. 
• F3 通过将所有功能编译成一个合并图，降低了执行计算成本；但该图同时也是发布单元，因此任何单点故障或无效定义都会阻塞整个发布流程。
  F3 reduced execution computational costs by compiling all features into a single merged graph, but that graph also served as the release unit, so a single failure or invalid definition blocked the entire release pipeline.
• 本文介绍了 F3 Lightning，这是一个构建在 F3 之上的基础设施层，旨在通过将功能发布与共享执行解耦来解决这一挑战，从而让每项功能都能独立编译、验证、部署和回滚。
  This post introduces F3 Lightning, an infrastructure layer built on top of F3 to resolve this challenge by decoupling feature releases from shared execution, so each feature is compiled, validated, deployed, and rolled back independently.
• 最难解决的部分是保留 F3 的效率提升。早期按功能发布的原型增加了 CPU 使用率，因为各个独立功能不再共享计算。我们通过四项优化来弥合这一差距：稳定的计算标识符、统一共享的执行计划、供所有功能共用的单一内存缓冲区，以及用于直接访问数组的稠密索引映射。
  The hardest part to solve was preserving F3’s efficiency improvements. Early per-feature release prototypes increased CPU usage because independent features no longer shared computation. We aimed to close this gap through four optimizations: stable computation identifiers, a single shared execution plan, one shared memory buffer across all features, and dense index mapping for direct array access.
• 借助 F3 Lightning，同等的提取功能如今能够以比上一代 F3 快约 3.6 倍的速度投入生产，同时基础设施计算成本没有增加。更短的发布周期也使得同期能够进行更多轮模型迭代。
  With F3 Lightning, comparable extraction features now reach production ~3.6x faster than the prior F3, with no increase in infrastructure compute cost. This shorter release cycle also enables more model iterations in the same period.

F3 共享执行架构入门知悉A Primer on F3’s Shared Execution Architecture

生产级机器学习系统很少直接使用原始数据。在模型做出预测之前，其特征服务系统可能需要读取已存储的信号，将事件筛选到某个时间窗口内，对数值进行计数或求和、归一化，并合并中间结果。Production machine learning systems rarely consume raw data directly. Before a model can make a prediction, its feature-serving system may need to read stored signals, filter events to a time window, count or sum values, normalize them, and combine intermediate results.

完成这项工作的最简单设计，是将每个特征都视为独立函数。这种设计容易理解，但随着规模扩大，会造成资源浪费，因为各个特征在计算上并非彼此独立。七天点击次数和七天展示次数可能会读取同一个事件源，并应用相同的时间窗口筛选。随后，点击率特征还可以复用这两个结果。如果单独运行每个特征，每次请求都会重复执行该筛选以及其他所有共享步骤。The simplest design to complete this work treats every feature as an independent function. That is easy to understand, but it becomes wasteful at scale, because features are not computationally independent. A seven-day click count and a seven-day impression count may read the same event source and apply the same time-window filter. A click-through rate feature may then reuse both results. Running each feature separately would repeat that filter, and every other shared step, on every request.

F3 采用了不同的方法，将特征提取表示为有向无环图（DAG）。每个节点执行一项工作，例如读取输入、应用过滤器或计算聚合值。一条边表示某项计算依赖于另一项计算的结果。该图的输出就是模型所使用的特征值。F3 approached this differently, by representing feature extraction as a directed acyclic graph (DAG). Each node performs one unit of work, such as reading an input, applying a filter, or calculating an aggregation. An edge means one computation depends on the result of another. The graph’s outputs are the feature values a model consumes.

工程师使用高级语言独立编写每项特性。F3 编译器将每个定义转换为一个小型图，然后把服务所用每项特性的图合并为一个合并图（图 1）。当不同特性共享某项操作（例如上文的七天筛选）时，编译器会将重复操作合并为一个节点。这样，共享的计算在每次请求中只需执行一次，而不必为每项需要它的特性各执行一次。运行时，特征提取就是对这个合并图进行遍历。Engineers author each feature independently in a high-level language. The F3 compiler turns each definition into a small graph, then merges the graphs for every feature a service uses into one merged graph (Figure 1). Where features share an operation, such as the seven-day filter above, the compiler collapses the duplicates into a single node. That shared work then runs once per request, not once for every feature that needs it. At runtime, feature extraction is a traversal of this merged graph.

共享图谱使每次发布都变成非此即彼的选择The Shared Graph Made Every Release All-or-Nothing

F3 的合并图能够高效地共享计算，但也有一个重大缺陷：图本身就是发布单元，一个格式错误的定义就可能拖住合并图中的所有其他功能。F3’s merged graph shared computation efficiently, but it came with a major catch: The graph was also the release, and one malformed definition could hold back every other feature in the merged graph.

图 2 展示了这一挑战。图中有七个独立编写的特性，标记为 A 到 G，其中五个定义有效，但 D 和 G 各自包含一个无关错误。“Attempt 1” 构建在 D 处失败，因此七个特性都无法发布。修正 D 后，“Attempt 2” 构建推进到 G，又一次失败。只有在“Attempt 3”中修复了这两个定义后，完整的合并图才成功进入生产环境。Figure 2 illustrates this challenge. It shows seven independently authored features, A through G, where five definitions are valid, but D and G contain unrelated errors. The “Attempt 1” build fails on D, so none of the seven features ships. After D is corrected, the “Attempt 2” build reaches G and fails again. Only after both definitions are fixed in “Attempt 3” does the complete merged graph make it successfully to production.

随着 F3 的规模扩大，故障变得越来越频繁。F3 最初只服务于几百个特征，在那种规模下，合并都能成功，发布也基本顺利。如今特征数量已超过 20,000，而且不同类型的特征需要根据各自的计算方式进行专门检查，因此新特征和修改过的特征要达到不同的测试标准。模型已经使用的特征在发布前必须通过与模型相关的测试。这些测试的计算成本很高，因此只会在后续发布阶段运行；又因为合并后的图会作为一个整体发布，其中的所有其他特征也都得等这些测试完成。Failures became more frequent as F3 scaled. F3 originally served a few hundred features, and at that size merges succeeded and releases were largely smooth. Today that number is more than 20,000, and different feature types need their own checks depending on how they compute, so new features and modified features are held to different testing bars. A feature that a model already consumes needs model-related tests before it can be released. Those tests are computationally expensive enough that they run only in a later release stage, and, because the merged graph ships as one unit, every other feature in it waits for them too.

F3 Lightning：让每项功能独立发布并共享计算F3 Lightning: Making Each Feature Its Own Release and Sharing Computation

F3 发布后，我们对合并图进行了多轮改进。每次改进都曾暂时提升发布速度，但随着更多功能上线，收益逐渐减弱，因为所有功能仍然一起打包在合并图中。因此，我们构建了 F3 Lightning，使每项功能都能独立发布，同时保留 F3 的效率。如图 3 所示，F3 Lightning 引入了两项改动来实现这一目标：After F3 launched, we made several rounds of improvements to the merged graph. Each one brought back release speed for a while, but the gains eroded as more features landed, because all features still shipped together in the merged graph. So F3 Lightning was built to make every feature releasable on its own while preserving F3’s efficiency.

As illustrated in Figure 3, F3 Lightning introduced two changes to help achieve this:

• 部署粒度：每项功能都是可独立发布的配置。其验证、发布和回滚均不会影响任何其他功能。
  Deployment granularity: Each feature is an independently releasable configuration. It is validated, shipped, and rolled back without affecting any other feature.
• 效率对等：F3 的效率源自合并后的计算图，因此为每项功能单独发布会失去这一效率来源。要在保留按功能发布的同时达到同等效率，就需要对 F3 Lightning 的执行方式进行几项关键优化。
  Efficiency parity: F3’s efficiency came from the merged graph, so giving every feature its own release removes the source of it. Matching that efficiency while keeping per-feature releases required several key optimizations in how F3 Lightning executes.

部署粒度：F3 Lightning 如何隔离功能发布Deployment Granularity: How F3 Lightning Isolates Feature Releases

隔离从编译器开始，每个功能都独立编译成一个配置。该功能配置包含提取该功能所需的一切——按顺序排列的所有必需节点，不借用相邻功能的任何内容。如果功能 C 出现问题，功能 A 仍然可以发布，因为这两个配置彼此毫无关联。Isolation starts in the compiler, where each feature is compiled on its own into a single configuration. That feature configuration carries everything needed to extract that feature — every node it needs, in order, with nothing borrowed from a neighbor. If feature C has a problem, feature A still ships, because the two configurations have no relationship to each other.

由于功能不再等到发布时才合并，因此每项功能都可以按照自己的计划单独选定并发布。发布流程变成了有序队列，而不是要么一次性全部发布、要么全部不发布的批次。变更会根据其能解除哪些阻碍来确定优先级。对于尚无模型使用的新功能，可以按照标准验证流程快速发布；而如果某项功能已被在线模型读取，相关变更则会先经过额外测试。Because features are no longer merged at release time, each one can be selected and released on its own schedule. The release path becomes an ordered queue instead of a batch that ships all at once or not at all. Changes are prioritized by what they unblock. A new feature that no model consumes yet can go out quickly on standard validation, while a change to a feature a live model already reads gets extra testing first.

效率持平：F3 Lightning 如何保持 F3 的服务效率Efficiency Parity: How F3 Lightning Preserved F3’s Serving Efficiency

我们的第一批原型只在部署层面进行了拆分。各项功能可以独立发布，但执行过程不再共享，因此这些原型的 CPU 使用量远高于传统 F3。每个独立发布的配置都会以独立图的形式进入服务进程，并单独执行，从而重复进行 F3 合并后的图已消除的每一次读取、筛选和聚合操作。Our first prototypes separated deployment only. Features shipped independently, but execution was no longer shared, so the prototypes used substantially more CPU than traditional F3. Each independently released configuration arrived at the serving process as its own graph and was executed separately, repeating every read, filter, and aggregation that F3’s merged graph had eliminated.

修复方案是在不恢复合并机制的前提下，重新引入共享执行。启动时，F3 Lightning 不会重建合并后的图，而是获取当前启用的特性配置，将它们共享的工作整合起来，并安排成单一的执行顺序。要恢复到原有水平，需要四个步骤：一个编译器前置条件，以及三项启动优化。The fix was to bring shared execution back without bringing the merge back. At startup, F3 Lightning does not rebuild a merged graph. It takes the active feature configurations, consolidates the work they share, and arranges it into a single execution order. Getting back to parity took four steps: a compiler prerequisite followed by three startup optimizations.

步骤 1：稳定标识符让协作成果清晰可辨Step 1: Stable Identifiers Make Shared Work Recognizable

为了消除重复执行，运行时必须识别出两个独立编译的配置是否包含相同的计算。节点编号无法做到这一点，因为编号只在各个功能的图内有效。同一个七天过滤器在一个配置中可能是节点 2，在另一个配置中则是节点 7；而插入一个节点，就会使其后的所有节点重新编号。传统的 F3 从未遇到过这个问题，因为一次编译就能看到所有功能，并可以全局分配标识符。To deduplicate execution, the runtime must recognize when two independently compiled configurations contain the same computation. Node numbers can’t do this, because numbering is local to each feature’s graph. The same seven-day filter might be node 2 in one configuration and node 7 in another, and inserting a single node renumbers everything after it. Traditional F3 never had this problem, because one compiler run saw every feature and could assign identifiers globally.

F3 Lightning 则让编译器根据每个标识符所代表的工作来生成标识符，而不是根据它所在的位置：操作、参数以及其输入的标识符（见图 6）。由于每个输入的标识符已经反映了它上游的所有内容，因此该标识符涵盖了生成结果的整条工作链。对于输入顺序不会影响结果的情况，输入会按标准顺序排列。聚合操作还会包含定义其行为的属性，例如时间窗口和分组键。因此，即使两个特征相隔数周才完成编译和发布，只要它们定义的是针对相同输入的相同操作，就会获得相同的标识符。如果某项计算的含义发生变化，其标识符也会随之变化。F3 Lightning instead has the compiler derive each identifier from what the work does, not where it sits: the operation, its parameters, and the identifiers of its inputs (see Figure 6). Because each input’s identifier already reflects everything upstream of it, the identifier captures the whole chain of work that produced the result. Where input order doesn’t affect the result, inputs are put in a standard order. Aggregations also include the properties that define their behavior, such as the time window and grouping keys. As a result, two features that define the same operation over the same inputs receive the same identifier, even if they were compiled and released weeks apart. If a computation’s meaning changes, its identifier changes with it.

第 2 步：为所有功能制定统一的执行计划Step 2: One Execution Plan Across Every Feature

服务启动时，F3 Lightning 会为所有启用的特征构建一份统一的执行计划，并通过拓扑排序确保每个节点都排在其依赖节点之后。如果多个特征中包含标识符相同的节点，计划只保留一份，并将所有使用该节点的下游节点连接到它，因此无论有多少个特征使用该节点，都只会计算一次。At service startup, F3 Lightning builds a single execution plan across all active features, using a topological sort to place every node after the nodes it depends on. When several features contain a node with the same identifier, the plan keeps one copy and connects every consumer to it, so that computation runs once no matter how many features use it.

合并可以保证每项计算只运行一次，但计划仍然是一个图，执行器每次处理请求时都要逐个指针遍历它。因此，同一个启动阶段的处理过程会将其展平为一个有序列表，这样每个请求只需沿列表向前执行一遍，而不必遍历图。Merging guarantees each computation runs once, but the plan is still a graph that the executor walks pointer by pointer on every request. The same startup pass therefore flattens it into an ordered list, so each request runs a single forward pass down the list instead of traversing a graph.

共享一个计划会带来风险：单个计算失败就可能影响所有功能。为防止这种情况，如果某个计算失败或其输入不可用，F3 Lightning 会将其下游的所有内容标记为未就绪，并由执行器跳过这些工作。故障只会影响依赖于该计算的节点，彼此独立的分支则会继续运行。Sharing one plan raises a risk: A single failing computation could affect every feature. To prevent this, if a computation fails or its input is unavailable, F3 Lightning marks everything downstream of it as not ready, and the executor skips that work. A failure affects only the nodes that depend on it, and independent branches keep running.

步骤 3：共享内存缓冲区，因为仅靠去重还不够Step 3: A Shared Memory Buffer, Because Deduplication Was Not Enough

完成步骤 1 和 2 后，没有任何计算重复执行，但 CPU 使用率仍高于传统 F3。剩余的计算开销来自内存布局。每个配置都带有自己的请求时数据容器，因此中间结果分散在各自独立的内存分配中。按顺序遍历计划意味着要在这些内存区域之间跳转，而每次跳转都会导致一次缓存未命中。After steps 1 and 2 were complete, no computation ran twice, yet CPU use was still above traditional F3. The remaining compute cost came from memory layout. Each configuration carried its own request-time data container, so intermediate results were scattered across separate memory allocations. Walking the plan in order meant jumping between them, and each jump cost a cache miss.

F3 Lightning 用每个请求一个缓冲区，取代了每个特性各自的容器。启动时，每个特性的引用都会重新绑定到该缓冲区中的槽位。使用单个缓冲区带来了三项改进：F3 Lightning replaced those per-feature containers with one buffer per request. At startup, each feature’s references are rebound to slots in that buffer.

Three improvements follow from the single buffer:

局部性：现在，连续的计划步骤会访问相邻内存，因此某个步骤读取的值通常已经在缓存中。这恢复了合并图一直具备的顺序访问模式，也消除了每次请求都要进行的逐特征分配。Locality: Consecutive plan steps now touch adjacent memory, so the values a step reads are usually already in cache. This restores the sequential access pattern the merged graph always had. It also removes the per-feature allocation on every request.

单一就绪向量：一个与值分开存储的向量取代了按特征跟踪，因此执行器无需读取值本身，就能检查该值是否已就绪。标记必须单独存储，因为空列表是有效结果，所以“为空”不能同时表示“尚未计算”。One readiness vector: A single vector, stored apart from the values, replaces per-feature tracking, so the executor can check whether a value is ready without reading the value itself. The marker has to be separate because an empty list is a valid result, so “empty” can’t also mean “not computed yet.”

不再复制多份：大多数特性值都是列表和映射，消费者不再各自获取一份副本。共享槽位保存对生产者计算结果的引用，因此，供 20 个特性使用的计算只需执行一次，结果也只需存储一次。单个值（标量）仍会被复制，因为复制一个小值比追踪指针更便宜。这样做是安全的，因为每个缓冲区只在一次请求期间存在，而且执行顺序保证每个结果的生命周期都长于其最后一次读取。No duplicate copies: Most feature values are lists and maps, and consumers no longer get their own copy. The shared slot holds a reference to the producer’s result, so a computation used by 20 features is computed once and stored once. Single values (scalars) are still copied, because copying a small value is cheaper than following a pointer. This is safe because each buffer lasts only for one request, and the execution order guarantees each result outlives its last reader.

步骤 4：密集索引映射消除了每次访问的最后一项开销Step 4: Dense Index Mapping Removes the Last Per-Access Cost

值现在集中存放在一起，但查找某个值的计算开销仍然很大。每次访问都必须解析一个标识符，也就是说，需要对键进行哈希、探测桶，并跟随指针。在 Meta 的服务请求量下，查找的计算效率并不高。Values now lived together, but finding one was still computationally expensive. Every access had to resolve an identifier, which meant hashing a key, probing buckets, and following a pointer. Lookups weren’t computationally efficient at Meta’s serving volume. 

最终的修复采用了与步骤 2 和 3 相同的原则，在启动时一次性完成这项工作。F3 Lightning 为每个唯一标识符分配一个连续的整数槽位，并重写所有配置中的每个引用，使其指向这些槽位，因此标识符只在初始化期间使用。读取结果时，只需直接通过数组索引访问共享缓冲区，无需进行哈希运算，也无需在请求路径上查找映射表。The final fix applied the same principle as steps 2 and 3, doing the work once at startup. F3 Lightning assigns every unique identifier a consecutive integer slot and rewrites every reference in every configuration to point at those slots, so identifiers are used only during initialization. Reading a result becomes a direct array index into the shared buffer, with no hashing and no lookup table on the request path.

借助 F3 Lightning，同等的提取功能如今投入生产的速度比传统 F3 快约 3.6 倍，且不会增加基础设施的计算成本。更短的发布周期意味着在同一时间内可以更频繁地改进和重新训练模型。构建 F3 Lightning 也让我们学到了三点：With F3 Lightning, comparable extraction features now reach production ~3.6x faster than on traditional F3, with no increase in infrastructure computational cost. That shorter release cycle means models can be improved and retrained more often in the same period. Building F3 Lightning also taught us three lessons:

部署和执行应当有各自独立的边界：如果同一个制品既决定组件如何运行，又决定它们何时发布，那么其中任何一项出现故障都会阻碍其他所有组件。将这两项工作拆分开来，消除了效率与发布速度之间的取舍。Deployment and execution should have separate boundaries: If one artifact determines both how components run and when they ship, a failure in any one of them blocks all the others. Splitting the two jobs removed a trade-off between efficiency and release speed.

每个请求之间都不会变化的工作，应在启动时完成，而不是放在请求处理路径上：每个请求都有三项完全相同的任务——识别共享工作、确定执行顺序，以及查找每个结果的存储位置。F3 Lightning 现在会在服务启动时一次性完成这些任务，因此独立发布不会增加单个请求的计算成本。Work that never changes between requests belongs at startup, not on the request path: Three tasks were identical for every request: spotting shared work, deciding the order to run it in, and finding where each result is stored. F3 Lightning now does them once, when the service starts, so independent releases add no computational cost to individual requests.

两种设计即使按相同顺序执行相同操作，CPU 计算成本仍可能有很大差异：在 F3 Lightning 中，我们去除了所有重复计算，但 CPU 使用量仍高于传统 F3。每个特征仍各自保留了一份共享结果的副本，分散在内存中。将每种结果整合为一份副本后，差距便消失了。这是整个项目中最不显眼、也最具普遍适用性的发现。在现代 CPU 上，结果存放在哪里、存在多少份副本，重要性可能不亚于执行了多少次计算。Two designs can execute the same operations in the same order and still differ substantially in CPU computational cost: In F3 Lightning, we removed every repeated computation, yet CPU use stayed above traditional F3. Each feature still kept its own copy of the shared results, spread across memory. Consolidating them into one copy per result closed the gap. This is the least obvious result in the project and the one that generalizes furthest. On modern CPUs, where results live and how many copies exist can matter as much as how many computations run.

致谢Acknowledgments

我们谨向 Sourabh Bansal、Ping Chen、Weitao Chen、Benjamin Dummitt、Junhua Gu、Henry Hu、Jing Huo、Joffrey Liu、Rocky Liu、Weixi Ma、Shuby Mao、Sandeep Pandey、Mihir Rastogi、Tim Sombuntham、Matt Steiner、Vivienne Sung、Ritwik Tewari、Vivek Trehan、Arnaud Venet、Ziheng Wang、Gautam Wilkins、Fan Yang、Qiao Yang、Jie Zheng，以及参与开发并将 F3（Meta 的广告机器学习特征计算与服务平台）和构建于其上的功能 F3 Lightning 投入生产的整个团队表示感谢。We would like to thank Sourabh Bansal, Ping Chen, Weitao Chen, Benjamin Dummitt, Junhua Gu, Henry Hu, Jing Huo, Joffrey Liu, Rocky Liu, Weixi Ma, Shuby Mao, Sandeep Pandey, Mihir Rastogi, Tim Sombuntham, Matt Steiner, Vivienne Sung, Ritwik Tewari, Vivek Trehan, Arnaud Venet, Ziheng Wang, Gautam Wilkins, Fan Yang, Qiao Yang, Jie Zheng and the entire team behind the development and productionization of F3, Meta’s Ads ML feature computation and serving platform, and F3 Lightning, the capability built on top of it.