持续学习为何仍未解决：SFT、蒸馏与 RL 的取舍

简要描述Brief Description

为什么至今无人解决持续学习问题Why Nobody Has Solved Continual Learning

目录Table of Contents

• 学习循环及其尚未解决的火车箱
  The learning loop and its unsolved train box
• SFT 能做到而 RL 做不到的事
  What SFT can do that RL can't
• 为什么脱离策略的 SFT 会在生产环境的边缘案例中失效
  Why off-policy SFT breaks on production edge cases
• RL 仍然是在线策略的，但只能提高其采样到的内容的权重。
  RL stays on-policy but can only upweight what it samples
• 开放实验室如何利用蒸馏为强化学习提供热启动
  How open labs use distillation to warm start RL
• 蒸馏模型起点更高，但最终会更早进入平台期，且性能上限更低。
  Distilled models start higher but plateau lower
• 持续学习：不断进行 SFT 会导致什么问题
  Continual learning: what breaks when you keep SFTing
• 为什么强化学习遗忘较少，却难以将知识融入其中
  Why RL forgets less but struggles to bake in knowledge
• 自蒸馏提示与偏差问题
  Self-distillation hints and the bias problem
• 窄任务型工具 vs Cursor 和 Harvey 这类通用智能体
  Narrow tasks vs general agents like Cursor and Harvey
• 前沿实验室早已在运行这一循环，只是速度比较慢。
  Frontier labs already run this loop, just slowly
• 大多数团队都处于中间状态：调整你的 SFT 与 RL 比例。
  Most teams sit in the middle: tune your SFT to RL ratio

学习循环及其尚未解决的训练盒The learning loop and its unsolved train box

我非常期待和大家聊聊我们正在思考的一些事情，尤其是研究方面的内容。至于如何更新模型的权重，比如说，如何掌控自己的智能，这是你能做的一件大事。所以感觉有很多事情正在发生，我会尽量全面地介绍一下我们的想法，以及我们认为前沿实验室在模型后训练、甚至更早期的训练阶段都在思考些什么。这大致就是大家常说的那个循环。你会在所有那些“掌控自己的智能”的幻灯片里看到这个循环。基本上就是：好，我们有一个智能体在执行某种任务。我们能够提取该任务的执行轨迹。Very excited to go through some of the things we're thinking about from particularly the research side. But in terms of how we update the model weights for a model, like, in terms of owning your intelligence, that's one of the big things you can do. So it feels like there's a lot going on, and I'm going to try and, like, give a big context dump of at least, like, what we're thinking about, what we believe even the frontier labs are thinking about when it comes to, like, post-training and even earlier stages of training with your models. So this is kind of like the loop that, you know, everyone refers to. This is the loop you see in all the, you know, own your own intelligence slides. It's essentially like, okay, we have an agent doing some sort of task. We are able to extract traces for that task.

这样我们就能够逐一检查，看看模型做对了什么、做错了什么，然后把这些内容转化成某种形式的评估。它可能是一个带有奖励函数的环境，就像我们在 RL 中看到的那样；也可能是一种评估，最终产出一条完美的 gold trace，供我们进行 SFT，也就是监督微调。不管具体形式如何，总有某种方法可以判断我们生成的轨迹是好是坏。然后还有这个框，代表着训练这个过程——它非常模糊。训练完成后，我们希望模型有所进步，然后就把它部署出去。我今天特别想强调的是，不管别人怎么跟你说，训练这个环节根本还没有解决。这件事真的、真的、真的很难，而你所做的一切都有问题和取舍。We are able to then look through them and see what the model is doing right and what it's doing wrong. And we're going to turn that into some form of an eval. Now, this might look like an environment with a reward function, as we see in RL. It might look like an eval that turns into, you know, a perfect gold trace that we can then SFT on, supervised fine-tune on. Whatever it looks like, there is some way to determine good from bad in the traces that we're generating. And then, you know, there's this box, which is like this very nebulous, like, train. Once we've trained, we're hoping the model gets better and we deploy it. I think the thing I want to like hammer home today is that no matter what anyone tells you, like, the train box is like not solved at all. It's really, really, really hard, and everything that you do has problems and trade-offs.

所以我想，先跟大家说明一下背景：我说的是持续学习范式。也就是说，如果你想掌握自己的智能，就不能只做一次 SFT 或一次 RL。你希望模型能从你收集到的这些轨迹中持续变得更好。这就是建立这个反馈循环的全部意义：你要持续运行这个循环，而不是只做一次。所以今天我想跟大家聊聊，为什么你不能只持续做 SFT，也不能只持续做 RL。没有什么完美的解决方案。我会试着梳理其中的权衡，并概述一下我们可以采用的各种方法。And so I guess for everyone's context, I'm talking about the continual learning regime. So if you're going to own your intelligence, like, you don't want to just do a one-off SFT run or a one-off RL run. You want your model to continue to get better from these traces that you're collecting. Like, that's the whole point of having this feedback loop, is that it is a loop that you're consistently running. It's not a one-off thing. So I guess today, like, I'm going to talk to you about why you can't just continually SFT. You can't also just continually RL. Like, there is no perfect solution. I'm going to try and, like, sketch the trade-offs and the, like, give you an outline of all the different methods that we can use.

SFT 能做到而 RL 做不到的事What SFT can do that RL can't

所以，嗯，也许我们可以从这里入手，比如尝试总结一下围绕 RL、SFT、蒸馏等方面的文献。大家大概都有个模糊的概念，认为 SFT 能把新信息装进模型里。比如，我可以拿一条来自 Fable 5.1 的黄金轨迹，给 Qwen 4B，然后用它进行 SFT；这样一来，Qwen 就会获得新信息，还会得到一些它自己原本永远不会采样到的推理轨迹或路径。这就是 SFT 的一大优点。RL 就做不到这一点。RL 受限于我们所说的模型支持集。所以，如果模型能够采样出某个 rollout，也就是一条推理轨迹——随你怎么称呼——那么如果这条推理轨迹是好的，我们就可以提高它的似然。也就是说，RL 只能利用模型支持集中的内容。So yeah, maybe, like, a place to start here is, like, trying to summarize the literature around RL, SFT, distillation, etc. So everyone probably has this vague idea of SFT as being able to install new information in a model. So I can take a gold trace from, you know, Fable 5.1 and give it to Qwen 4B and SFT on that, and, like, Qwen is going to get new information and it's going to get, like, I guess, reasoning traces or paths that it never would have sampled by itself. That's a great thing about SFT. RL can't really do that. RL is limited to what we say the support is of the model. So if a model can sample a particular rollout, a reasoning trace, however you want to call it, we can upweight the likelihood of that reasoning trace if it's good. So RL can only take things which are in the support of the model.

SFT 某种程度上可以让模型做你想让它做的任何事。现在，这里有很多内容值得拆解，因为，呃，从这个非常简短的描述来看，SFT 似乎正是你想要做的事情。比如，我可以让我的 Qwen3 4B 表现得像 Fable 5.1 一样。只要我有足够多的 SFT 轨迹就行。SFT can kind of make the model do whatever it wants. Now, there's a lot to break down here because, like, from that, like, very brief description, it seems like SFT is, like, exactly the thing that you'd want to do. Like, I can get my Qwen3 4B to act like Fable 5.1. I just get enough SFT traces.

为什么离策略 SFT 在生产环境的边缘案例上会失效Why off-policy SFT breaks on production edge cases

问题出在这里：SFT 是一个非常离策略的过程。所以，我用来学习的数据确实包含非常密集的信号。比如，每个 token 都会给我一个新的奖励，告诉我根据 Fable 5.1，正确的下一个 token 是什么；但这个过程非常、非常离策略。于是我用这些数据对 Qwen3 4B 进行微调，然后把它部署到生产环境中。你知道，我遇到了一个生产环境中的边缘案例，它甚至算不上分布外，只是没有出现在训练分布中。比如说，我以前没见过这个确切的例子。而这种离策略性意味着 Qwen3 4B 比 Fable 5.1 小好几个数量级。所以就假设它犯了一个错误，比如走错了一步。假设它正在解一道数学题。离策略的问题在于，Qwen3 4B 从来没有用 Qwen3 4B 自己犯的错误进行过训练。The problem comes here because SFT is a very off-policy process. So the data that I am learning from, yes, it's got a very dense signal. Like, every token, I'm getting a new reward about what the correct next token is, according to Fable 5.1, but it's very, very off-policy. And so I fine-tune my Qwen3 4B on this and then I go deploy it in production, and, you know, I hit this production edge case which isn't even out of distribution. It's just not in the training distribution. Like, I haven't seen this exact example before. And this off-policyness means Qwen3 4B is orders of magnitude smaller than Fable 5.1. So let's just say it makes, like, one mistake, like one misstep. We say it's solving a math problem. The problem with off-policyness is that Qwen3 4B has never been trained on Qwen3 4B's mistakes.

比如说，它只是在 Fable 5.1 的 rollout 数据上训练过。所以它一旦出错，训练数据里就完全没有示例能告诉它该如何回到正确的推理路径上。这是 SFT 的一个重大失效模式。早期传统的 RL 文献就有人指出过这一点。有一篇叫 DAgger 的论文，我强烈推荐大家读读。它对这个问题提供了很多直观的见解。这也是为什么我们不能只是一味地增加 SFT，除非我们真的能在所有内容上都做 SFT——而这其实就是预训练的思路。Like, it's only been trained on Fable 5.1's rollouts. And so that means that as soon as it makes a mistake, it's got nothing in its training data which shows how to get back onto the correct reasoning path. And this is a big failure mode of SFT. This was first kind of pointed out in the old traditional RL literature. There's a paper called DAgger that I highly recommend people read. Gives a lot of intuitions about this, but it's a reason why we can't just, like, kind of jack up SFT unless we can literally SFT on everything, which is kind of the idea of pre-training.

强化学习仍然是同策略的，但只能提高其采样到的内容的权重。RL stays on-policy but can only upweight what it samples

强化学习则处在某种意义上截然相反的一端，它极其符合 on-policy 的设定。也就是说，我们始终只从打算用于生产环境的模型 Qwen3 4B 中采样。假设我们又是在这样一个数学环境中：我们会提高 Qwen3 4B 生成优质轨迹的概率，也就是它获得正向奖励时的概率，然后降低它生成负向轨迹的概率。不过这里是有上限的，对吧？比如说，我们要解一道 2026 年的 IOI 题。Qwen3 4B 很可能采样不出解答。我现在可能不该这么说，因为连 4B 模型都有可能解出 IOI 的题目，这有点不可思议。不过你知道，如果是非常难的数学题，这么小的模型采样出正确答案的可能性就会非常低。RL, in, like, kind of the opposite end of the spectrum, is extremely on policy. So, we're only ever taking samples from the model, Qwen3 4B, that we intend on using in production. And so let's say, again, we're in this, like, kind of maths environment. Like, we are going to upweight Qwen3 4B's probability of doing good rollouts, so when it gets a positive reward, and then we're going to downweight its negative rollouts. But there is a limit here, right? Like, let's say that we're trying to solve, you know, an IOI problem from 2026. Like, it's very unlikely that Qwen3 4B is going to sample. I probably shouldn't say that now 'cause like even the 4B models are, like, possible. It's possible for them to do IOI problems, which is kind of crazy. But, like, you know, if it's a very hard math problem, it's going to be very unlikely for that small model to be able to sample a correct answer.

如果我们永远都采样不到正确答案，那就永远没法更新它。所以这里存在一种权衡：这种方法极其遵循当前策略，而且你知道，我们不会以这种方式损害模型。模型无论在任何时刻处于什么状态，都完全知道该如何恢复，因为它生成的内容始终来自模型本身。不过，你知道，我们也有一个上限：我们只能提高那些本来就在模型分布中的内容的权重。所以这已经包含了很多信息，但希望我现在开始让你意识到，好吧，这里面有很多这样的权衡。我可以用 RL 来改进模型，但这非常、非常取决于模型的起点。SFT 则不太是这样。我可以拿一些非常好的黄金数据来训练，无论我的模型当前处于什么状态，用这些黄金数据训练模型，都应该能让模型在我所用的训练数据所体现的方向上有所改进。And if we can never sample a correct answer, we're never going to be able to update it. So there is this trade-off, like it's incredibly on-policy and like, you know, we're not damaging the model in this way. The model is going to know exactly how to recover from wherever, at any point in time, because it's always coming from the model. But, you know, we have this ceiling of, like, we can only upweight things which are already in the distribution of the model. And so that's a lot of information, but hopefully, like, now I'm starting to instill into you, like, okay, there's a lot of these trade-offs. I can use RL to improve my model, but that is very, very dependent on where my model starts from. SFT is not really like that. I can take some really good gold data, and no matter where my model is sitting, like, training the model on that gold data should improve according to the data that I'm training on.

但 RL 并不是那样的。当然，SFT 也存在我刚才提到的那些问题。所以，这就是起点。But RL's not like that. But of course, SFT has, like, the problems that I just talked about. So this is, like, the starting point.

开源实验室如何利用蒸馏为强化学习提供热启动How open labs use distillation to warm start RL

我现在想稍微聊聊我们一直在思考的一些问题，比如什么时候该做 SFT/蒸馏，什么时候该做 RL。我觉得这总体来说是个非常有意思的问题，因为你知道，它能让你预测中国的开源实验室应该怎么做，比如开源实验室应该做多少蒸馏、多少 RL？这是一个非常有意思的结果，我们用几种不同的方式复现了它。正如我刚才所说，如果一开始你连采样出一个好的 rollout 的概率都没有，那就没法用 RL。所以很多人会做的事，也是我们认为开源实验室会做的事，就是拿一个预训练后的模型来用，而这个模型在我们关心的任何 RL 环境中，采样出有用内容的概率可能都非常低。I want to talk now a little bit about, like, some of the ways that we've been thinking about, like, when to do SFT/distillation and when to do RL. And I think this is, like, a very interesting question in general because you know it allows you to predict what, you know, the Chinese open source lab should be doing, like how much distillation should an open source lab be doing versus RL? And this was one really interesting result which we replicated in a few different ways. So as I just told you, you can't use RL if it doesn't, if you don't have any probability of sampling a good rollout in the first place. So what a lot of people do and what we believe, you know, the open source labs to do, is they will take a model after pre-training that's probably got a very low likelihood of sampling anything useful on any of the RL environments that we care about.

因此，蒸馏就被用来为这个模型提供热启动。我们会从某个前沿闭源模型中提取一些推理轨迹和操作轨迹，然后在预训练结束时，或是在某种中期训练或持续预训练阶段，把这些数据加入进来。这样，模型就能看到一些正确完成这项任务的示例。这看起来和 SFT 完全一样。说白了，这就是 SFT。我们只是在这些轨迹上进行下一个 token 预测。我们认为，开源实验室使用蒸馏就是为了让模型在特定的 RL 环境中获得一个非零的 pass@k，然后再从那里开始进行 RL，并观察模型逐步提升。这通常就是采用的流程，而且我们看到许多公司都在走这条路。So distillation is then used as a way to warm start this model. So we'll take some reasoning traces and some trajectories from, you know, a frontier closed source model, and we'll throw those in at the end of pre-training or in kind of like mid-training or continued pre-training. So the model sees some examples of how to do this task correctly. And this looks exactly like SFT. Like, this is literally just SFT. We're just doing next-token prediction on these traces. That's what we believe distillation is used for at the open source labs. It's a way to get a nonzero pass at, you know, k for your particular RL environment, and then you do your RL from there and you watch your model climb. And this is typically like the process that's used, and you know, like we see many companies going through this process.

所以他们会有，你知道，很多非常复杂的 RL 环境。他们想让自己的模型在这些环境中变得更强，但模型一开始的 pass@k 实在是非常非常低。比如，pass@256 可能低得离谱，而他们又没有足够的算力，无法长时间用那么大的组规模运行，观察模型逐步提升。所以他们会先用一些闭源轨迹做一点蒸馏，然后再进行 RL。So they'll have, you know, they have, like, a bunch of very complex RL environments. They want their model to get better at them, but the model's starting off at a really, really low pass@k. Like, pass@256 might be, like, really, really low, and they just don't have the compute to be able to run, like, that big of a group size for that long to watch the model climb. So they'll do a bit of distillation on, you know, some closed source traces and then do the RL after that.

蒸馏模型起点更高，但最终性能较低。Distilled models start higher but plateau lower

说重点，我认为这张图展示的是——而且我们用很多种方式都复现了这一结果——很有意思的是，如果你为了获得热启动而进行一定程度的蒸馏，然后正确地进行强化学习，并一直训练到基本上彻底达到平台期，那么蒸馏后的模型起点会高得多。显然，这正是进行蒸馏的目的。但对完全没有经过蒸馏的模型进行强化学习，最终效果会好得多。所以你会看到这样一种曲线：蒸馏曲线起点更高，但它的平台期远低于经过强化学习的模型。这多少有点出乎意料，我们原以为这是最初实验设置中的一个特例，但后来发现，在很多情况下都是如此。To get to the point, I think what this graph is showing, and again, like, we've replicated this in many ways, is that, really interestingly, if you do, like, any meaningful amount of distillation in order to get your warm starting, and then you do RL properly and you do it until, like, you've really, really plateaued, basically you start off much higher after you've done distillation. Obviously, that's the whole point of doing it. But RLing on a model that hasn't been distilled at all ends up getting much better. So you have this kind of, like, curve where the distillation curve starts higher but it plateaus much lower than the model that's been RL'd. And this was a little bit surprising, and we thought this was a quirk of the initial setup that we've done, but this turned out to hold across a lot of things.

而且这也非常取决于你在学习哪种任务。所以看起来，RL 模型在计算或数学类任务上的上限要高得多，纯粹是因为我觉得，蒸馏基本上会限制模型的熵。我们也试着把这一点应用到实际中，比如说，我们会为客户训练很多模型。现在我们会认真考虑，一开始到底要不要做任何蒸馏。如今我们更倾向于这样做：如果算力足够，我们就一路走 RL。也就是说，你得关注不同的事情，比如 RL 任务的课程安排。比如说，如果有些任务的 pass@k 不错，而有些任务的通过率基本为零，你就得给这些任务安排好顺序。And it's very dependent on the type of task that you're learning as well. So it seems like the RL models, like, reach a much higher ceiling in, like, calculation or math-based tasks, purely because like I think basically distillation is constraining the like entropy, you can think of, of the model. So we've kind of, like, tried to apply this in, like, you know, we train a lot of models for customers. Now we're actually really thinking about carefully about whether we do any distillation at all to start off with. Like, now we're much more pilled on, like, if we have the compute, we're going to RL, like, all the way. It means you have to care about different things, like, for instance, you have to care about the curriculum of your RL task. Like, if you've got some task that, you know, you have a good pass@k, and some task where pass is, like basically zero, you kind of have to order this curriculum.

这是另一件你必须操心的事。不过，你知道，如果最终能达到更高的水平，那操心这件事可能还是值得的。It's another thing you have to care about. But, like, you know, if you get to a higher point at the end of the day, then it probably is worth caring about it.

持续学习：不断进行监督微调（SFT）会导致什么问题Continual learning: what breaks when you keep SFTing

另一件我想聊聊的事，算是我们最近看到的一些结果吧，就是持续学习这个概念。我一开始就提到过，基本上如果你只是进行一次性的大规模训练，这些问题其实都算是解决了。比如，你可以用 SFT，模型就会变得更好。当然，分布之外的情况它可能处理得不太好，但你有大量 Fable 5.1 任务或 rollout 示例的话，你的 Qwen 模型就会变得更好。RL 也是一样。你收集一堆环境，进行 RL，模型不断爬坡优化，就能得到一个更好的模型。我觉得真正有意思的是持续学习的模式：你要反复进行这个过程，不是一次，也不是十次，而可能是几百次，甚至上万次。Another thing that, like, I kind of want to talk about in terms of, like, results that we've seen recently is this idea of, like, I guess, yeah, continual learning. So right at the start I mentioned that basically if you're doing a one-off like, you know, big training run, these things are kind of solved. Like, you can use SFT, your model is going to get better. Like, yes, there'll be things outside the distribution that it won't go well on, but like, you know, you've got a big example of, like, Fable 5.1 tasks or rollouts, like, your Qwen model is going to get better. Same with RL. Like, you collect a bunch of environments, you do the RL, the model hill climbs, like, you got a better model. I think what's really interesting to me is, like, the continual learning regime, where you're doing this process, like, not once and not, like, 10 times, but probably, like, hundreds if not tens of thousands of times.

比如说，你实际上的权重更新，至少每周会进行多次。我觉得这是最有意思的一种发展方向，而且我认为很多公司都不得不朝这个方向努力，因为它们需要某种护城河来对抗 Anthropic 或 OpenAI。这里的问题在于，嗯，有几个方面。首先，如果我一直用 SFT 来训练，比如说，我有一个法律助理 LLM，想用它接触到的所有新信息来训练这个法律助理。比如，它了解了这个案件的新情况，知道这个人加入团队成为了一名律师助理，也知道这些信息存储在这家律所的什么地方，等等。所以，还是那句话，我可以进行一次规模较大的 SFT 微调。Like, your actual, like, weight update is happening, you know, at least multiple times a week. I think, like, this is the most interesting world to live in, and I think it's the world that, like, a lot of companies are going to have to shoot towards because, you know, they need some sort of moat against Anthropic or OpenAI. And the thing that breaks down here is, well, it's a few things. So the first thing is like, if I keep SFTing on, let's say I have this, like, you know LLM that's a legal associate, and I'm trying to train this legal associate on all the new information it's seeing. So, you know, it's learned new things about this case, and it's learned that this person has joined as a paralegal, and it's learned that this information is stored here in this firm, and so on. So again, like, I can kind of, like, you know, do a big fine-tune with SFT.

因此，如果我们想学习知识，就会采用一种叫作自我学习的方法。进行一次大规模微调，这没问题。但如果我一遍又一遍地重复这个过程，就会发现有好几件事会发生。首先，我会忘掉基础分布中的一些内容，而这些内容对我当前正在执行的任务很有用。这叫作灾难性遗忘。大多数人都听说过这个概念。我忘掉的不仅会是比如说17世纪的英国文学，甚至到某个时候，我也会开始忘掉法律方面的知识，尽管这些法律知识理应对我和我的法律助理智能体有所帮助。第二，我还会忘掉预训练之后学到的东西。比如说，第一天我针对某个特定案件进行了微调，然后经过100次更新后，我又针对一个完全不同的案件进行了微调。So we use something called self-study if we want to learn, like, knowledge. Do a big fine-tune once, that's fine. If I repeat that process over and over and over again, I find that multiple things happen. The first thing is I forget things in my base distribution which are useful to the task that I'm currently doing. So this is called catastrophic forgetting. Most people have heard about this. I will forget not only, like, you know, English literature from the 17th century, but I will actually start forgetting law stuff as well at some point, even though that law stuff is supposedly helpful to me and my legal associate agent. The second thing is I will forget stuff that I've learned after pre-training. Like, you know, let's say the first day I've fine-tuned on, you know, this particular case, and then, you know, 100 updates later, I fine-tune on a completely different case.

我们发现，模型会忘记之前的案例，而且遗忘速度远快于模型通常发生灾难性遗忘的速度。这真的很有意思。第三个非常糟糕的问题是，模型的通用能力会退化。所以，如果你开始测试它的指令遵循、数学、推理以及各种通用任务，就会发现，在迭代过程中，它在这些任务上的表现确实会大幅下滑。因此，很明显，自学加上 SFT 并不是持续学习的什么灵丹妙药，而且当任务范围非常非常广时，它就不起作用了。不过我也要补充一点：如果任务非常受限，而且数据分布是平稳的、不会发生变化，那么这种方法确实有效。比如，我可以持续在这类任务上微调模型。What we find is that the model will forget the earlier cases, and at a much faster rate than catastrophic forgetting happens in the model in general. That's really interesting. The third really bad thing is that the model's general capabilities will degrade. So if you start testing it on, you know, instruction following, maths, reasoning, all these kind of, like, general tasks, we see that this really fails in the iterative regime. So it's clear that, like, you know, self-study plus SFT is not this like silver bullet to continual learning, and, like, it doesn't work when you have this very, very broad task. I will caveat this with saying that this does work when we have, you know, a very constrained task with a stationary distribution that's not changing. Like, I can keep fine-tuning on that.

我可以继续在这方面生成黄金轨迹，我的模型也会不断变得更好。但对于非常通用、非平稳的任务来说，比如不断有新信息涌入，也不断出现新能力，对自学内容进行 SFT 似乎不太奏效。I can keep generating gold traces on that, and my model will keep getting better. Like, but for very, very general tasks that are non-stationary, like new information is coming in as well as new capabilities, SFT on the self-study stuff doesn't really seem to work.

为什么强化学习不太容易遗忘，却难以将知识融入模型Why RL forgets less but struggles to bake in knowledge

所以你可能会说，好吧，那 RL 呢？我会说，RL 不会像 SFT 那样以同一种方式影响模型。所以，我们不会在 RL 中看到那么严重的灾难性遗忘，也不会看到通用能力退化。而且，我们在进行 RL 的各项任务之间也几乎看不到相互干扰。基本上，原因是——你可以去读相关文献——归根结底，无论你使用哪种 RL 方法，模型都会受到这种隐式 KL 惩罚的约束。它就像是把模型和原始基础模型紧密地绑定在一起。因此，我们进行 RL 时，只会对模型做非常非常小的调整。但 RL 的问题在于，首先，很难把知识灌输进去。所以，正如我所说，只有当模型有可能采样出某种内容时，我们才能通过 RL 改变模型的行为。So you might say, okay, what about RL? RL, I will say, doesn't mess with the model in the same way that SFT does. So, you know, we don't see that much catastrophic forgetting with RL. We don't see, you know, degradation of general abilities. And we see little interference between tasks that we're kind of RLing on. Basically, the reason for this, like, you can read all the literature about it, boils down to the fact that the model has, no matter what RL kind of method you use, basically this implicit KL penalty. It's like tying it very closely to the original base model. So when we RL, we just, like, make very, very small nudges to the model. But the problem with RL is, like, it's very hard to, one, bake in knowledge. So again, as I said, we can only change the behavior of a model in RL if the model is likely to sample something.

模型几乎不可能采样到类似这样的内容：比如，Jenny 以律师助理的身份加入了 XYZ 团队。像这种非常非常具体的知识，很难通过设计强化学习环境来预先嵌入；其次，也很难通过任何形式的效率惩罚来诱导模型学会这些知识。所以你可能会想，如果有世界上最好的效率惩罚机制，就能激励模型去学会某种行为，而不是每次都要重新搜索。但我们从未见过强化学习中发生这种情况。模型会越来越擅长搜索，也会越来越擅长用 grep 搜索，方方面面都会变得更好；但即使效率惩罚再高，它也永远不会受到足够的激励而说：“好吧，下次我会记住这些信息。”这仅仅是因为强化学习的运作机制就是如此。A model is very unlikely to sample something like, you know, Jenny has joined as a paralegal in team XYZ. Like, some very, very specific knowledge is very hard to, like, engineer RL environments to bake in knowledge, and secondly it's very hard to like induce that in any way through any sort of efficiency penalty. So you might imagine if you had the world's best efficiency penalty, you could incentivize the model to, like, learn some behavior instead of having to search for it each time, but we've never seen this happen with RL. Like, the model will get better at searching and it'll get better at grepping and everything, but it will never be incentivized enough, even with the highest efficiency penalties, to say, okay, I'm going to remember this information for next time, simply because of the mechanism of how RL works.

所以我觉得，这就是你们应该从中得出的主要结论。基本上，在迭代模式下，很多这类方法都会失效，而强化学习不足以成为一种通用解决方案，来构建我们的持续学习器。So I think, like, that's the main thing that you should take away from this. Basically, in the iterative regime, a lot of these things break down, and RL is not enough of a general solution to bake in our, like, continual learner.

自蒸馏提示与偏差问题Self-distillation hints and the bias problem

最近出现了很多方法，比如 ECHO，还有最近 Cursor 推出的 Composer。他们尝试了一种做法：在 on-policy 自蒸馏中进行调整，基本上是在 rollout 阶段为模型提供提示，也就是密集级别的提示，从而帮助模型进行 RL。所以他们本质上是在尝试结合两者的优势：一方面设置 RL 奖励，另一方面设置自蒸馏奖励，实际上是试图向模型提供新的信息。我认为，总的来说，即便是自蒸馏，也非常、非常难以奏效，因为它会自然地给 RL 算法引入偏差；而 LLM 到了 RL 阶段时，已经变得非常、非常抗偏差。任何形式的偏差都会影响它们，不过我很乐意之后和大家聊聊这个问题。There's been, like, a lot of approaches, like ECHO and, like, you know, even Cursor recently with Composer. They tried this thing where they adapted on-policy self- distillation to, like, essentially provide hints, like dense-level hints, to the model in doing RL at kind of, like, rollout time. So they're trying to combine the best of both worlds here, essentially, where they go, okay, we're going to have an RL reward, we're going to have a self-distillation reward where we're actually trying to provide new information to the model. I think the general take with, like, self-distillation even is very, very hard to get to work because it naturally introduces bias into your RL algorithm, and LLMs, by the time you get to RL, are very, very, like, anti-bias. Any sort of bias will mess with them, but I'm happy to talk to people about that later.

是的。那么，稍微总结一下……我还有多少时间？好，我们还有一点时间。Yeah. So, to kind of take away... how much time I got? Okay, we got a little bit of time.

特定任务 vs Cursor 和 Harvey 这类通用智能体Narrow tasks vs general agents like Cursor and Harvey

所以，总的来说，我刚才给大家讲了一大堆东西。那么，当你用某种形式的 LLM 或智能体执行一项具体任务，并且想要掌握这个持续学习循环时，这到底意味着什么？我想大家都知道，如果你只有一项任务，而且范围非常狭窄，那你基本上就顺风顺水了，简直要欢呼雀跃。比如说，撰写临床记录：我输入一段转录文本，并希望按照特定的提示结构生成临床记录；或者做语音提取，搭建一个语音转文本、LLM、文本转语音这样的系统。还有这些电话客服智能体，它们也是一个例子：任务范围很窄，而且理论上，我的规则和整体数据分布都非常稳定。So, to take away, like, I've kind of dumped a bunch of stuff on everyone. What does this really mean when you have this particular task that you're doing with some form of LLM or agent, and you want to, like, own this continual learning loop? So, I think everyone's kind of aware that if you have one task and it's extremely narrow, then you're kind of, like, sailing. You're cheering. You're really happy, like if it's, you know, writing clinical notes so I'm dumping in a transcript and expecting some clinical notes with a particular prompt structure out, or you know, I'm doing, you know, voice extraction, so I've got like you know speech-to-text, LLM, text-to-speech kind of setup. I've got these like phone calling customer service agents. That's another example of a task that is like really narrow and theoretically my rules and distribution in general is very stationary.

所以我觉得，你知道，这里最好的做法可能是——这话可别引用我的——就是找一个你能找到的最好的模型，无视服务条款，生成几十万甚至几百万条轨迹，然后用它们做 SFT。再说一次，这话可别引用我。我这可不是什么法律建议之类的，不过这里的思路是，你可以直接拿这些黄金 SFT 轨迹来做 SFT，这样你就会逐渐逼近那个黄金分布；在无限数据的情况下，你最终就能达到它。太棒了。然后，你知道，世界变得越来越具备智能体属性。有人决定给 LLM 配上工具。我们开始让它们更自由地四处活动，直到基本上每个 LLM harness 都变成了某种通用的图灵完备 LLM 机器，让它们可以在任何想用的沙箱里随心所欲地做任何事。So I can, you know, the best thing to do probably here and like don't quote me on this, is, like, take the best model you can find, ignore the terms of service, and generate like hundreds of thousands, if not millions, of traces and SFT on that. Like, again, don't quote me on that. That wasn't legal advice or anything, but the idea there is that you can just take these gold SFT traces and SFT on them, and you're going to asymptote towards that gold distribution, and in the infinite data regime, you're going to get there. That's great. Then, you know, the world got more agentic. Someone decided to give LLMs tools. We started to, like, let them roam free a little bit more until like basically, like, every LLM harness was this like kind of universal Turing LLM machine that allowed them to do whatever they want on whatever sandbox they wanted.

所以在这种情况下，事情要困难得多。举个例子，那些试图在这一层掌握自己的智能的公司，比如 Cursor、Notion、Harvey、Lovable 等等，都意识到，当你的任务足够通用时，基本上就得把自己变成一家前沿实验室，才能掌握自己的智能。这意味着，像我们之前讨论过的那样，要在大量由你自己创建的环境中进行后训练；同时也要做好更早期阶段的训练，也就是中期训练，而这看起来同样像 SFT。从这个角度来说，没错，这是可行的，但非常非常困难。And so, in this case, like, things are a lot harder. As an example, people that have tried to own their own intelligence at this layer have been, you know, Cursor, Notion, Harvey, Lovable, like, a bunch of these companies that are realizing that when your task is general enough, you basically have to turn into a frontier lab in order to own your intelligence. So this means, you know, post-training on a bunch of environments that you yourself create, as, like, we've talked about before, but also, like, doing the earlier stages as well, the mid-training, which again looks like SFT. So from that perspective, like, yes, it's doable, but it's very, very hard.

前沿实验室早已在运行这一循环，只是速度较慢。Frontier labs already run this loop, just slowly

我想传达的最后一点是，我们其实已经解决了这个持续学习闭环。听起来我好像是在说我们还没解决它。我的意思是，在非常细粒度的层面上，我们确实还没解决，尤其是当你在运行模型或更新模型时，比如一天更新好几次。我觉得如果你看看那些前沿实验室在做什么，他们确实在运行持续学习闭环，只不过节奏非常慢。比如，他们训练一个模型，然后发布给所有人使用。有些会查看确切的用户数据，有些会征求用户反馈。他们把所有这些信息汇总起来，看看模型有哪些事情做不到。很好，我们就把这些转化成强化学习环境。你知道，如果模型在这些强化学习环境中的表现得分为零，我们就会利用特权信息，或者用某种方式构造出这些优质轨迹。I think the last thing I want to, like, convey is this continual learning loop, like, we've already solved. Like, it kind of sounds like I've said that this is, we haven't solved it. I mean, we haven't solved it at a really granular level when you're operating or you're updating your model, like, you know, multiple times a day. I think if you look at what the frontier labs are doing, like, they are running a continual learning loop, just on a very slow cadence. So, you know, they train a model, release it to the world. Like, some will look at the exact user data, some will ask for user feedback. They aggregate all this and look at all the things the model can't do. Great, we'll turn that into RL environments. You know, if we get zero on those RL environments, we're going to, like, use privileged information or some way to construct these great trajectories.

我们会用它们进行中期训练，然后从那里开始做强化学习（RL），并以此为基础进行预热。所以我认为，如果你看看这个循环，它已经在运行了，而且同时包含监督微调（SFT）和强化学习。只是到了那个规模，我们就不必担心灾难性遗忘，比如通用能力退化，以及我之前提到的所有那些问题和崩溃情况。We're going to mid-train on them, and then we're going to do RL from there and warm up from there. So I think if you look at this loop, it's already running and it involves both SFT and RL. It's just, at that scale, we don't have to worry about catastrophic forgetting, like general capability degradation, all the things that I mentioned earlier, the breakdown.

大多数团队处于中间地带：调整 SFT 与 RL 的比例Most teams sit in the middle: tune your SFT to RL ratio

嗯，我想这就是我最后想表达的一点。不过，这个房间里大多数人做的事情应该都处在中间地带。比如，你的任务既不算特别狭窄，也不完全是 Cursor 那种。你可以想办法把你希望模型持续精进的知识和能力都融入模型中。你只需要非常审慎地把握 SFT/蒸馏和 RL 的比例。我觉得这是开始思考这个迭代循环时最重要的切入点。所以，技术团队、后训练团队等等，都可以从这里入手，看看并思考自己处在这个光谱的哪个位置。好，谢谢大家。Yeah, I think that's the last thing I wanted to convey. But yeah, most of the people in this room will live kind of in the middle. Like, you have a task that is, like, not quite narrow, but also not quite Cursor. And you will be able to come up with some way to bake both the knowledge and the capabilities that you want to, like, continually get better at into that model. You just need to be very smart about, like, the ratio of SFT/distillation to RL that you do. And I think that's the most important place to start thinking about this loop. And so technical teams, post-training teams, etc. Like, that's the place to start looking and thinking about, like, where on that spectrum you sit. Cool. Thanks, guys.