Shopify CTO Mikhail Parakhin 谈 AI 战略：提高能力上限与商家数字孪生

简要说明Brief Description

Shopify 的 AI 战略揭秘：与首席技术官 Mikhail Parakhin 对话Inside Shopify's AI strategy with CTO Mikhail Parakhin

目录Table of Contents

• Shopify 的 CTO 如何借助 Claude 提升能力上限
  How Shopify's CTO raises the ceiling with Claude
• 衡量 AI 投资回报率和工程效率
  Measuring AI ROI and engineering productivity
• 为什么 Shopify 为每位工程师提供无限量的 token
  Why Shopify gives every engineer unlimited tokens
• 构建数字孪生以预测商家的增长
  Building a digital twin to predict merchant growth
• 用于管理工程团队的 AI
  AI for managing engineering teams
• 给首席技术官的建议：杠铃式代币策略
  Advice for CTOs: the barbell token strategy

Shopify 的首席技术官如何借助 Claude 提升上限How Shopify's CTO raises the ceiling with Claude

你知道文本不过是一串字母吗？大语言模型非常擅长预测下一个字母，然后就能把文本补全。人类的行动也很相似，但对我们来说，更重要的是，公司的行动也很相似。你可以把一家公司表示为一连串行动。这样一来，你就可以创建这家公司的数字孪生体。你把哪些最棘手的问题交给 Fable？交给 Claude？哪些问题的成功让你感到意外？我想出了一个问题的解决方案，这个问题我七年来一直在尝试解决，却始终没能解决，大语言模型也没能解决。那就是一个叫作 Wasserstein 损失函数的东西。让一切都变成高斯分布。大语言模型带走的并不是繁琐的工作，也不是你不想做的工作。You know how text is just a sequence of letters? And LLMs learned really well how to predict the next, and then they can complete the text. Human actions are very similar, but in our case, more importantly, companies actions are similar. You can represent a company as a sequence of actions. And then what happens is that you can create a digital twin of the company. What are the hardest problems that you're throwing at Fable? That you're throwing a Claude? Where you've been surprised by success? I came up with a solution to a problem that for seven years I've been trying to solve. And I was unable to solve it, and LLMs were not able to solve it. Something called the Wasserstein loss function. Making everything Gaussian. It's not that LLMs take away the drudgery of work or the work that you didn’t want to do.

而现在它们能做到了。所以现在你能做一些有意思、更加复杂的事情。也就是说，你能做以前做不到的事。比如说，无论给我多少时间、多少帮手，就算我能调动成千上万最优秀的数学家，我还是做不到。但借助这个模型，我就能做到。至于目前最难的那些问题，我看到的情况是：没有模型，我解决不了；但没有我，模型也解决不了。这需要我们来回协作，打造出一个“半人马”。你知道，用国际象棋来打个比方，我们合作起来比模型单独行动更强，也比人类单独行动更强。所以你是在提高下限。不过别忘了，你也可以提高上限。没错，这正是我的意思。是的，提高上限的影响要大得多。因为要提高下限，你本来就可以更勤奋一些，或者多派些人来做。And now they do it. And so now you can do interesting things, more sophisticated. It’s that you can do things that you previously couldn't. Like, in no amount of time, no amount of helpers, I could have had thousands of the best mathematicians at my disposal, I still wouldn't be able to do it. But with the model I can. And for the best problems right now what I see is that I cannot solve it without the model, but the model cannot solve it without me either. It requires, this back and forth and creating that centaur. You know, using chess analogy, that together is better than the model and better than a human being. So you're raising the floor. But actually, don't forget, you can also raise the ceiling. Exactly. That's that is my point. Yeah. Raising the ceiling is is much more impactful. Because raising the floor, you could have just been more diligent or throw more people at it.

上限是，没什么简单的方法可以扩展。对。这确实是个重大突破。那么你会这么做吗？我想，对于，Ceiling is, there's no easy way to extend. Right. That's a real unlock. So would you do this? I guess, for,

衡量 AI 投资回报率与工程生产力Measuring AI ROI and engineering productivity

你知道，比如说，研究工作，或者之前尚未解决的问题？你可以这么看。这是个新问题，解决方案曾经遥不可及，而现在我们已经解决了。从公司的角度，或者从商业产品的角度，你会如何看待这类事情的投资回报率（ROI）？因为我经常听到的一种说法是，抬高能力下限很容易衡量：比如说，这些工作原本需要工程师来做，现在 Claude 能做了，工程师因此腾出了时间。这相当容易量化。那么，对于另一类问题——能力上限被抬高了，工程师现在能做以前做不到的事——你会怎么看？这是个非常好的问题。我觉得说到 AI，我就直说了：如今在衡量能力下限有多高这件事上，它大概是世界上最好的。you know, like, research or some problem that hasn't been solved before? You can think about it that way. This is a novel problem. The solution was out of reach. Now we've solved it. In a company context, in a business product context, how do you think about the ROI or something like this? Because something I hear a lot is when you raise the floor, it's really easy to measure because you're like, okay, this amount of work would have been done by engineers. Now, Claude can do it. Engineers are freed up. That's fairly easy to quantify. How do you think about the other kind of problem where you've raised the capability ceiling and now engineers are doing something they couldn't have before? It is a very good question. And I feel like in terms of AI, I'm just going to go ahead and say it, it’s probably the best in the world right now at measuring how high is the floor?

当然，他们和其他人一样，不仅会估算 PR 的数量和复杂度，我们实际上还会估算项目的复杂度。在我们的 AI 系统中，系统就是负责把事情做完。我们可以看到，团队能够更快地推进更多项目；按复杂度进行标准化后也是如此。你可以非常精确地衡量生产力提升的百分比。They have not only, of course, just like everybody else, estimate the number of PRs and complexity of PRs, but we, we actually do things like estimating complexity of the project. And in our AI system, just the system get stuff done. We could we could see that, hey, the teams of people are able to execute on more projects faster and, normalized for complexity. You can be very precise at measuring percentages of productivity you're getting.

为什么 Shopify 为每位工程师提供不限量的 tokensWhy Shopify gives every engineer unlimited tokens

你会怎么和 Toby 谈这件事？你对另一家公司的 CTO 有什么建议？他也得和自己的 CEO 谈这件事。首先，要始终使用最大的模型，因为最大的模型最能提高上限。除非你用大模型并比较结果，否则你不会知道。你不知道自己错过了什么，对吧？不行，不行，不行。而且他们觉得应该屈居第二。要始终把最大规模的资源投入到问题上。然后，直到我看到证据为止。否则我们就会一直这么做。这就是为什么在 Shopify，我们给每个人提供无限的 token。这就是为什么要运行我们能找到的最强大的模型。这也是为什么我们在 GPU 推理、训练和优化能力上投入大量资源。How do you have this conversation with, with Toby and, what's your advice to, another CTO at a different company that has to have the conversation with their CEO about this? always use the largest model because the largest model raises the ceiling the most first of all, you don't know unless you use the large model and then and compare the results. You don't know what you're missing, right? It's a no no, no. And that and that they think stay in second. Always throw maximum a thing at the problem. And then until I see the evidence. Otherwise that's what we're going to be doing. And that's why, at Shopify we have unlimited tokens, for everybody. That's why run, the heaviest models. We can find, that's why we, we throw a lot of, a lot of resources in our with, GPU inference and training and ability to optimize.

从某种程度上说，这就是……你只需要确保所有参与决策的人都确实在使用这个模型和产品。他们得明白这是什么。然后你就得不断突破上限，看看能带来什么回报。接下来，如果它有效还是无效，就一目了然了。你们上线过哪些项目，能明显地提升上限？举个例子好吗？In a way, it's a it's, you just have to you have to make sure that everyone that's making the decisions is actually using the model and using the product. They understand what this is. And then you have to just keep pushing the ceiling and seeing those returns. And then it's just obvious if it works or it doesn't work. What's an example of one of these projects you ship that that raised the ceiling in an obvious way.

构建数字孪生以预测商户增长Building a digital twin to predict merchant growth

我可能最引以为豪的，大概是我们的 HST 预测系统。它的基本思路是，文本只是一串字母。大型语言模型非常擅长预测下一个字，然后就能续写文本。人类的行动也很相似，但对我们来说，更重要的是，公司的行动也很相似。你可以把一家公司表示为一连串行动。这家公司可能会开通信贷额度、发货、点击“添加”并做出修改。它们开始接受信用卡，申请贷款。我的意思是，这些都是重大变化。但公司及其所有员工每一秒所做的无数事情，都可以转化为一连串行动。接下来，你就可以创建这家公司的数字孪生，然后在它身上进行操作，而不是在真实公司上操作，再看看会发生什么。My the one that I'm probably, most proud of is is our HST prediction system. The idea is basically, How text is just a sequence of letters. And LLMs learned really well how to predict the next, and then they can complete the text. Human actions are very similar, but in our case, more importantly, companies actions are similar. You can represent a company as a sequence of actions. The company is, I don't know, open credit line ship product. Click add and make changes. They start accepting credit cards, take a loan. And I mean these are large changes. But all the myriad things that the company, all of its employees, every second are doing, it can only be turned into a sequence of actions. And then what happens is that you can create a digital twin of the company, and then you can start doing things to it instead of the real company and then see what happens.

你可以进行反事实干预。你可以说：“哦，如果我们开展一场广告宣传活动会怎样？或者，如果他们提前一天开始发货会怎样？又或者，这一切都围绕我们的商家展开，如果我们给他们一笔贷款，然后找出一系列能最大限度提高他们增长机会的互动，会怎样？”接着，我们真的会在生产环境中这么做。于是我们会主动给他们提供贷款，或者推动他们，向他们发送消息。看看大家。你们最重要的事情是……是更快开始发货，或者……在 Twitter 上开展广告宣传活动，或者……所以，能够在不危及任何真实用户的情况下，模拟一家公司的运作，这会带来巨大的突破。因为现在，你们可以帮助更多人取得成功。这一点非常明确。日志。那本来是不会发生的。这对我们的利润有着非常直接的影响。You can have, counterfactual interventions. You could say, oh, what if we added a advertising campaign? Or what if they started shipping, one day faster, or what if, it's all about our merchants, what if we give them a loan and then we find a sequence of interactions that would maximize their chances of growth? And then we actually do that in production. And so we go offer them a loan or we, push them, send them communication. Look for folks. Your most important thing is to, to start shipping faster or with, start advertising campaign on Twitter or, So, having ability to play with a company without endangering anything, real people is is huge unlock. Because now now you can you can help more people to to be successful. This is very clear. Log. That wouldn't have happened. And that has very direct impact on our bottom line.

然后是财务成果。再说构建这件事，假设你能把它构建出来，过去需要多长时间？但问题是，我觉得过去根本不可能做到。对吧？想想看，你得汇聚足够多的人才，还得把他们召集起来。然后你得有这种预见，知道需要做这件事，接着还得开始收集数据以及做各种准备。我觉得过去从人力上根本做不到。所以关键在于，答案是：过去他们有没有继续做下去？我们压根就不会这么做。这就是我对“提高上限”的定义：开始做一些过去甚至不在考虑范围内的事情。有什么事让你改变了想法？And then the financial result. And just building this, let's say you could have built this, how long would this have taken in the past But this is the thing I don't think it was possible in the past. Right. Just think about it like you would need to get the concentration of talent. You would have to bring that. And then and then you have to have have this premonition that you need to do that, and then you need to start getting collecting data and all this, I don't think, I don't think that was humanly feasible previously. and so the trick here is the answer is like previously, did they continue to we just wouldn't have done it. And that's the if my definition of raising the ceiling. I think you start doing something that previously was not even in consideration set. What's something that you changed your mind about

用于管理工程团队的 AIAI for managing engineering teams

因为 Claude，因为这个模型。我原以为大语言模型并不能真正帮助我更好地管理团队。我错得不能再离谱了。过了一段时间，我意识到自己改变了很多想法，于是创建了一个系统。它会分析各种情况，主动告诉我：“这个项目看起来要开始延期了，截止日期可能会受影响。”它让我能在问题真正发生之前就发现它们，及时处理，还能察觉团队成员的不满情绪，因此整个系统运转得更好了。它也发挥了这样的作用。是的。不过，Shopify 变了。现在，技术方面甚至更重要了，因为我学到的那些简单的东西会……做得很对。不过，大语言模型很擅长捕捉信号，而且总是试图取悦你。所以，不管你问什么，答案往往已经包含在问题里了。because of because of Claude. Because of the model. I thought that the lens cannot really help me manage the team better. I could not have been more wrong. And after a while, I realized, I, I flipped so much, I created a system. That actually analyzes what everything is happening like. and proactively tells me, But this it looks like this project is going to start slipping there. Deadline. it allows me to detect issues before they actually happen and go over it and then detect people unhappiness and so the whole system starts functioning much better. Has the role of it. Yeah. But, Shopify changed. now technical side is even more important because the simple things I learned will go correct. But, the LLM is very good at picking up signals and it tries to please. And so whatever you ask, the answer is very often in the question.

它只会把这一点阐明出来。而且，他们会准确地给你想要交易的东西。重要的是，你真正想要的是实际所需的东西，而不是把脑海中第一个冒出来的想法说出来。因为那样的话，后果可能反而比之前更糟：之前至少不会立刻就把它构建出来，还会有一些讨论。现在却容易多了，直接动手就能把它构建出来。技术能力变得更加重要。而人的作用也变了：现在，工程师要监督的是 LLM，而不是监督那些敲键盘、按按钮的工程师，这完全是两码事。这几乎就像是你成了二级管理者，因为现在你想要分享：嘿，怎样才能最好地把工作分派给不同的警报？[不清楚——可能是“代理”？]And it will just elucidate that. And they give you exactly what you want to trade. It's really important to want what's actually needed not put in words first thing that comes to mind. Because then then your consequences could be actually worse than before, because before, at least it wouldn't be immediately built. There would be some discussion. And now it's so much easier to just go and build it. Technical acumen, becomes more important. And the human component of, supervising now, engineers who are supervising LLMs rather than supervising engineers who are typing, pressing buttons, that's a very different thing. That's it's almost like you get second, become second level manager, because now you want to share. Hey, what the best, best way is to dispatch the work to different alarms? [unclear — possible "agents"?]

你又要如何不忘记他们在做什么，然后又要如何避免他们之间发生冲突？我觉得，这就是为什么我不会说在这个意义上有了巨大的变化。但我会说，重点变得更加集中在这两件事上，比如让团队更高效，让他们更多地参与技术决策，而不是每天都去检查。嘿，因为那件事已经做完了，And how are you going to not forget what they're doing and then how are you going to avoid conflicts between them? I think, that's why I wouldn't say it's dramatically different in, in that sense, but I say the accent is became much more on those two things, like keeping the team more efficient and making them more have making technical decisions, and less on checking day to day. Hey, because that thing been done,

给 CTO 的建议：杠铃式代币策略Advice for CTOs: the barbell token strategy

好，我想以这个问题作结。你有什么建议想给其他 CTO 和技术领导者？他们也在摸索如何采用 quad、如何使用它、如何在组织中推广，以及如何让团队熟练掌握它。 我的主要建议是，尽量不要限制 token。你总是需要设置一些机制，比如断路器，因为各种失控的流程可能会突然开始疯狂消耗 token，做出糟糕的事情。但大模型和小模型之间的差异，可能看起来并不大；实际上却天差地别，而你会希望选对模型。你不知道反事实是什么，所以要弄清楚哪一个才是好的选择。 无论是编码、开发、测试还是自动化研究，都始终采用这种二元杠铃策略：为所有这些任务配备预算最高的大模型；而在生产环境中，则使用成本合适、经过微调且适用于生产的模型。Okay, I want to end on this. What is your advice to other CTOs, other technical leaders that are also trying to figure out how do I adopt quad? How do I use it? How do I scale it out to my org? How do I get my work proficient? my main advice would be try to not limit tokens you always need, like circuit breakers, because they're all these runaway processes that suddenly start burning tokens and and, doing things bad. But it might not look like there's big difference between big models and small models. There's a world of difference and you want it. You don't know the counterfactual, so you want to know which one, which one is good. Always use this dichotomy barbell strategy for anything coding for anything. Development, testing, auto research use. The largest model budget for it is that cost for everything production use, fine tuning is the right model for that production.

要对它进行彻底的自动研究，兼顾成本、吞吐量和质量。生产环境可以用更便宜的模型来省钱，这些模型往往经过微调；然后把省下来的钱投入开发，投到模型升级潜力巨大的地方。就 Fable 5 而言，我曾公开表示，它是一次非常大的飞跃，比之前所有的进步都大得多。我确实反复看到有人反其道而行，在开发阶段试图节省编码 token 的开支，同时又在实际生产服务中浪费资源，运行规模远超所需的大模型。我觉得，只要有人不沿着这条路走，我就能胜过他们。我刚才还在和这里的朋友聊天，说 Boris 彻底摧毁了我……我在……我一直大声宣称，自己最终会破译线形文字 A。这是一种非常有名的文字，青铜时代米诺斯人在克里特岛上使用过，也是我个人很感兴趣的领域。Auto research the hell out of it so that you balance both, cost, throughput and quality. And save money on production by using cheaper models, often fine tuned, and invest that money in development site on the on the place where there's huge potential in upgrading your model to, in Fable 5, I was on the record to say that it's it was such a huge step, like much, much bigger than, like all the previous steps. Yes, I've seen repeatedly people doing the opposite and trying to save money on, coding tokens on unquote on development and then at the same time wastefully running, more of those larger than they need in the actual production services. I feel like I will outcompete anybody who is not on that stretch. I was chatting with, my friends here that, Boris destroyed my, my will to live freely in my, I kept, loudly proclaiming that I will end up, deciphering, Linear A, which is a very famous, writing was used on Crete by Minoans, in the Bronze Age, kind of personal area of interest of mine.

然后，后来鲍里斯发推说，有人用 Claude 做出来了，我就想，真该死，那可是我整套说辞啊。我把退休后打算做的项目都拿出来了。你当时就像是拿着你那台老 Commodore……我只是没能及时做完。没错，没错。非常感谢你过来。And then, and then Boris tweeted that somebody did it using Claude, and I'm like, damn it, that was my whole spiel. I took my retirement project. I was going to work on. You had like, your old Commodore I just didn't finish a time. Exactly, exactly. Thanks so much for stopping by.