78% 的用户想要，实际却很少使用：Bumble 的 AI 功能实验教训

简要说明Brief Description

78%的人想要这项 AI 功能，几乎没人使用它——Bumble 的 Supriya Subudhi78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi， Bumble

目录Table of Contents

• 简介
  Intro
• Bumble 的 AI 照片反馈功能
  Bumble's AI photo feedback feature
• 1. Users ask for it, then ignore it
• 2. The average win hides the tail
• 3. Green dashboard, red bill
• 4. The win came from the model
• 5. There may be no global rollout
• 要问的五个问题
  Five questions to ask

简介Intro

呃，我叫 Supriya。呃，很高兴今天来到这里。我想和大家聊聊，为什么 AI 产品的 A/B 测试很奇怪。对于不了解 A/B 测试的朋友来说，它就是一种评估某个事物的两个版本并比较哪个更好的方法。Uh my name is Supriya. Uh happy to be here today. I'm here to talk to you all about why AB testing for AI products is weird. So, for those of you who don't know what AB testing is, it's just a method used to evaluate two versions of something with each other and see which is better.

所以，这可能是两种邮件版本，你想看看哪一种能获得更多点击。也可能是某个功能的两个版本。也可能只是把一个功能与当前已经上线的版本进行对比。那么，我来快速问个问题。So, that could be two versions of an email where you're trying to see which one's getting more clicks. It could be two versions of a feature. It could just be a feature against what's already live right now. So, let me ask a quick question.

大家举手示意一下好吗？谁觉得如果 A/B 测试结果是正向的，就应该真的上线这个功能？嗯，看来你们很多人都很聪明，已经知道，呃，说到 AI，事情可能会变得有点奇怪。那我们就来聊聊为什么会这样。Can I get a show of hands? Who thinks that if an AB test comes back positive, that the feature should actually ship? Well, looks like a lot of you are smart and already know that uh when it comes to AI, things can get a little weird. But here, let's talk about why.

所以，我在测试 AI 功能时发现，尤其是在消费科技功能中使用生成式 AI 和大语言模型时，A/B 测试得出积极结果，其实只是旅程的起点，而不是终点。接下来我们会讲讲我具体发现的五个问题，以及我们必须如何克服这些问题。So, what I found when um testing with AI features, specifically GenAI and using LLMs in consumer tech features, a positive result from an AB test is actually just where the journey begins. It's not where it ends. And we're going to go over um five reasons that I've specifically discovered that were problems that we had to overcome.

Bumble 的 AI 照片反馈功能Bumble's AI photo feedback feature

那么，我是一名产品经理。我在 Bumble 工作。Bumble 是一款约会应用，月活跃用户超过 2000 万。那么，产品经理是做什么的呢？So, I'm a product manager. I work at Bumble. Bumble's a dating app with over 20 million uh monthly active users. And so, what does a product manager do?

我们代表用户。我们了解他们面临的问题，也努力提供能够解决这些问题的产品，同时对公司的损益负责，因为我们的资源并非无限，无法解决所有问题。因此，为了讲清楚我认为 AI A/B 测试很奇怪的五个原因，我觉得最好先给大家看一个我们实际上线的案例，这样你们也可以尝试将它与自己正在做的事情联系起来。我们上线的功能是一项 AI 照片反馈功能。We represent our users. We know the problems that they face and we try to deliver things that can solve the problems while still being responsible for the P&L of a company because we don't have unlimited resources to be able to solve the problems. So, in order to talk through the five reasons that I think AB testing with AI was weird, I figured it'll be good for you all to have a visual example of what we actually shipped so that you can try to draw parallels with things that you're working on as well. The feature that we shipped was an AI photo feedback feature.

在约会应用中，每个人自然都需要一份约会档案。他们会上传照片，却不知道什么样的照片才算好。因此，我们利用自己对约会档案中哪些做法有效、哪些无效的专业经验，为用户上传的照片提供了由 AI 生成的改进建议。In a dating app, naturally, everyone needs a dating profile. So, they would add pictures and they wouldn't know what good looks like. So, we essentially came up with the expertise that we had of what works and what doesn't work in a dating profile, we gave AI-generated suggestions of how people could improve the pictures that they were adding.

1. 用户提出需求后，却又忽略它了1. Users ask for it, then ignore it

那么，我们来谈谈第一种情况：用户提出需求，然后又置之不理。我刚才提到，产品经理应该了解用户面临的问题，对吧？我们用来做到这一点的工具就是调研。So, let's go into the first one. That users ask for it and then they ignore it. Now, I mentioned that a PM is supposed to know the problems of our users, right? The tools that we have to do that is research.

这段旅程始于探索：人们面临哪些问题？我们又该如何解决？调查问卷和客户访谈都是很好的了解方式。在 Bumble 做研究时，几乎所有人都会提到一个问题：他们不知道自己的交友资料是否足够好，而我们也没有提供任何支持，帮助他们判断自己做得好不好。The journey begins with discovery. What are the problems that people have and how can we solve them? Surveys, customer interviews are great tools to find out. When we do research at Bumble, one of the almost unanimous problems that everyone always talked about was that they didn't know whether their dating profile was good enough and they didn't get any support from us in understanding whether they were doing a good job or not.

因此，当 AI 兴起、LLM 展现出巨大潜力时，我们立刻意识到这是一个可以解决的问题。于是，我们构思了不同的方案，设想 AI 如何帮助人们的约会之旅，但不替他们约会。当我们向大家展示我们可以提供的 AI 支持和指导示例时，反响非常好。我们接触过的人中，近 78% 都表示，他们很乐意接受我们的反馈，并希望我们告诉他们如何做得更好。So, when AI came in and there was all that potential with LLMs, we knew immediately that this was a problem that we could solve. And so, we came up with different versions of how we thought AI could help people's dating journeys without actually doing dating for them. And so, when we showed people examples of AI support and guidance that we could give them, the response was great. Almost 78% of the people that we spoke to came back and said they would love getting feedback from us and tell us and tell them how to do better.

但当我们推出 A/B 测试时，也就是让 50% 的用户获得这项功能，另外 50% 的用户无法使用这项功能，我们发现真正使用该功能的用户只有 10% 到 15%。这让我们认识到，对 AI 持开放态度，并不能真正预测人们是否会使用这项功能。这也让我们回到了产品管理的基本原则：你需要将使用 AI 的整个流程真正融入用户在应用中已经在做的事情。教育和提升认知至关重要。But when we shipped the AB test, which means that with 50% of the people getting the feature and 50% of the people not getting the feature, we only saw 10 to 15% people actually using the feature. What that taught us was that openness to AI doesn't actually predict whether people are going to use the feature. And that's where we go back to product management 101, which is you need to really bake in the entire journey of using AI into what people are already doing in the app. Education and awareness are super critical.

我们没有单独开发一个功能，让用户在应用里自己想办法找到它，而是把它融入了用户原本就在使用的流程中。比如，用户添加照片时，我们会问：“嘿，你想获得一些反馈吗？”当他们想找约会对象却屡屡碰壁，一个匹配对象也找不到时，我们会问：“嘿，你去查看过个人资料里的反馈吗？也许可以做些调整。”这推动了功能使用率的提升。当然，衡量我们想要达成的目标非常重要。What we did was instead of building a standalone feature, which people could just somehow find within the app, we went into the journeys that people were already using. So, when they were adding pictures, we asked them, "Hey, do you want feedback?" When they were hitting a wall when they were trying to find someone to date and not finding any matches, we would ask them, "Hey, did you go and check the feedback in your profile? Maybe you could change something." And that drove up adoption. It's really important that we obviously measure what it is we're trying to do.

嗯，这将构成基础，因为归根结底，如果调查结果是积极的，你就发布了这个功能，却没有实际了解人们是否在使用它，那么你根本就无法提高它的采用率。Um that is going to form the foundation cuz ultimately, if the survey is coming back with a positive result and you ship the feature, and you haven't actually looked at whether people are using it or not, and you won't be able to improve that adoption in the first place.

2. 平均收益掩盖了尾部风险2. The average win hides the tail

我们发现的第二件事是，平均收益触及了尾部。因此，当我们再次推出这项功能时，整体结果非常好。更多人开始在个人资料中添加图片；他们调整个人资料所做的更改次数，也多于没有收到反馈的人。The second thing that we found was that the average win hit the tail. So, when we shipped this feature again, the top-line results were great. More people were adding pictures to their profiles. The number of changes that they were making to adjust their profiles was more than the people who weren't receiving the feedback.

但当我们深入细节时，发现了一些奇怪的情况。AI 生成的建议竟然在告诉人们多笑一笑，可他们在照片里已经在笑了。所以，这让我们意识到，仅凭正向平均分已经不足以评估是否成功。我们必须深入了解细节。But when we went into the details, there were weird things happening. The AI-generated tips were actually telling people to smile more when they were already smiling in their pictures. And so, what that told us is that the positive average was not enough to evaluate the success anymore. We had to go into the details.

在分享我们实际做了什么之前，先来看两个例子。我想，Spotify 有个叫 AI DJ 的功能，它会利用 AI 为用户生成播放列表。再设想一下，你有一款银行应用，它能帮你制定预算，并决定该把钱花在哪些方面，以及每个月花多少。Let's look at two examples before I share what we actually did. So, I think Spotify has something called an AI DJ. It generates playlists for people using AI. And then, let's imagine that there's your banking app which helps you budget and decide what you should spend on and how much you should spend in a month.

这两件事的错误影响范围截然不同。AI 生成的四首歌里有两首不怎么样，这没什么大不了；但如果你确实负担不起，应用却告诉你要花 2,000 美元，那就不行了。这就是防护机制发挥作用的地方，也是我们着手的起点。The scope of error is quite different in both of these things. It's okay if two out of four of the songs that are generated by AI are not good, but it's not okay if an app is telling you to spend $2,000 when you really can't. So, that's where guardrails come into place. That's where we started.

为 AI 生成的反馈设定明确的行为边界至关重要。我们不能告诉人们去改变他们无法控制的事情，比如肤色或身高。这些都不是 AI 应该涉及的内容。接下来，我们要做的就是阅读真实生成的内容。Setting strong guardrails of what our AI-generated feedback should and should not do was super critical. We couldn't be telling people to change things that they can't control, their skin color, their height. These are things that the AI should not touch. The next thing we had to do was reading real outputs.

所以，评估在这方面特别有帮助。现在，在座懂评估的专家可比我多得多。呃，你来对地方了，可以在这里了解这方面的内容。不过，搭建合适的评估工具对我们来说至关重要。我们得以深入了解真实用户在收到反馈时实际看到的内容。So, this is where evals were super helpful. Now, there are way more experts here that know about evals than me. Uh you're in the right place to learn about that, but um setting up the right eval tools were super critical for us. We were able to dig in and see what real users were seeing when they were getting feedback.

这实际上帮助我们判断提示是否真正发挥了作用。因此，最终，成功标准从关注绿色与红色的整体结果，转变为考察数据中的长尾情况：我们是否满意不同用户群体收到反馈的方式？嗯，女性对我们的反馈的反应是否与男性相同？And that was helping us actually see whether our prompt was actually doing a good job or not. And so, ultimately, the success criteria changed from looking at overall results of green versus red to what does the long tail of data have? Are we happy with the way different segments of users are receiving feedback? Um are women responding the same way to our feedback as men?

不同语言的回应是否也像不同国家的人一样，以相同的方式回应反馈？因为法语生成的语言可能与英语生成的语言不同。Are different languages responding like people in different countries responding in the same way to feedback as well? Because maybe the language generated in French is different from the language generated in English.

3. 绿色仪表盘，红色账单3. Green dashboard, red bill

第三件事是绿色仪表盘和红色账单。自然，生成式 AI 的成功是有代价的。刚开始时，我们知道这对用户来说是个大问题，因此希望提供质量最高的反馈和准确度最高的结果。所以，我们选择了最好的 LLM。The third thing is that the green dashboard and the red bill. So, naturally, with generative AI, the success comes with a cost. When we started, we knew that this was such a big problem for our users that we wanted the best quality of feedback and the highest accuracy. So, we went with the best LLM.

当然，这比更简单的方法贵了大约 10 倍。再加上对你构建的解决方案进行评估的成本，整体开销实在太高了。Bumble 对 AI 并不陌生。Now, naturally, that was about 10x more expensive than a simpler approach. And then you add to that the cost of running evals on the solution that you've built. That was just too much. Now, Bumble is not a stranger to AI.

我的意思是，匹配算法本身也是人工智能的一种应用。不过说到机器学习，它的成本和生成式人工智能不一样。算法需要大量的前期成本，包括计——计算成本，以及呃训练成本。但等它运行起来之后，每多一个用户使用它，增加的成本就没那么高了。I mean, the matching algorithm is also a use of AI. But, when it comes to ML, the cost is different compared to GenAI. In an algorithm, there's a lot of upfront cost of compu- compute as well as uh training. But, then after after it's working, the incremental cost for every user to use it is not that much.

借助生成式 AI，产品越成功，使用的人越多、使用频率越高，你的账单也就越高。那么，该怎么办呢？也许一开始别急着用最好的模型。首先，弄清楚自己为什么要使用 AI 非常重要。With generative AI, the more successful your product, the more people that use it, and the more often they use it, the more your bill increases. And so, what do you do? Maybe don't jump to the using the best model from the start. And it's very important to understand why you're using AI in the first place.

我和许多同行聊过，他们刚刚接到公司的要求，要在功能和产品中使用 AI。但你确实需要弄清楚自己为什么要这么做：是为了长期收益，还是为了短期目标？现在，我们以亚马逊为例。There are a lot of my peers that I've spoken to who have just gotten mandates from their companies to use AI in their features and their product. But, you really need to understand why you're doing it. Is it for long-term benefits or is it for something short-term? Now, let's look at Amazon as an example.

他们有一个，嗯，能帮你在亚马逊上买东西的 AI 助手。那么，亚马逊为什么需要推出这个助手呢？是为了获得短期利益吗？好的推荐会带来更多购买，从而在短期内增加收入？还是说……可是，如果推荐效果不怎么样呢？They have um an AI assistant that can help you buy something on Amazon. Now, why would Amazon need to launch this assistant? Is it for short-term benefits that good recommendations lead to more purchases and therefore more revenue in the short-term? Or is it But, what if the recommendations aren't great?

这是否意味着他们不会这么做？也不一定。从长远来看，拥有一个值得信赖的助手，或许会让人们选择亚马逊，而不是其他电商公司。因此，推出这个助手的理由，可能只是为了提升品牌资产，并从长远来看让用户留在自己的平台上。Does that mean they don't do it? Maybe not. In the long-term, having an assistant that you can rely on might make people choose Amazon over other e-commerce companies. And so, the reason to launch the assistant is just for brand equity and a long-term benefit of keeping your users on your platform.

因此，真正理解你为什么要使用 AI 至关重要，因为你的预算以及愿意投入多少，都取决于这一点。从短期收益来看，是的，也许可以根据投资回报率迅速做出决定；但如果这是着眼长远的布局，也许值得争取更多预算。And so really understanding why you're trying to use AI is going to be critical here because your budget and how much you're willing to spend is going to come from that. From a short-term benefit, yes, maybe the ROI can be a quick decision, but if it's for a long-term play, maybe it's worth getting that extra budget.

4. 这次胜利归功于该模型。4. The win came from the model

所以我提到了，我们一开始就直接采用了最好的模型。自然，成本高得离谱，我们也无法将其规模化。因此，我们做了一个简单的决定，改用更便宜的模型。但随着成本降低，结果的影响力也随之减弱，因为准确率同样下降了。So I spoke about the fact that we just jumped to using the best model. Naturally, the cost was way too high and we could not scale that. So we switched what we made the simple decision to switch to a cheaper model. But as the cost went down, so did the impact of the results because the accuracy went down as well.

这基本上说明，我们提供的 AI 反馈以及在 A/B 测试中看到的结果，都是由我们测试的模型造成的，而不是反馈本身这个概念造成的。那么我们做了什么呢？我们在这个阶段学到的是，我们确实需要进行大量测试，而能够以数据为依据做决策至关重要。评测方案确实能帮我们做到这一点。Which basically told us that the AI feedback that we were giving was and the results that we saw from our AB test was due to the model that we were testing, not the concept of giving feedback itself. And so what did we do? What we learned at this point was just that we really need to test a lot and being able to be data-driven was critical. And eval setups really help us do that.

能够测试不同模型对不同数据集的影响程度、比较成本差异，并快速重新测试不同版本的提示词和反馈，这一点至关重要。最终，我们选择了成本最低的方案，也因此取得了不错的结果。Looking at being able to test how much different models can impact different data sets, the difference in the cost, and then being able to retest different versions of prompts and feedback quite quickly, that was critical. And ultimately, we resulted in the cheapest option, which helped us get the good result.

5. 可能不会进行全局推出5. There may be no global rollout

现在说最后一种情况，有时可能根本就不会进行全球推广。Meta 最近推出了 Muse AI。我不确定你们有没有用过。我来自伦敦，所以还没用过。And now the last one, which is sometimes there just may not be a global rollout. So Meta recently launched Muse AI. I'm not sure if you all have used it. I'm from London, so I haven't yet.

这是因为 Muse AI 目前仅在美国和加拿大提供服务。这是因为 AI 监管规定不断变化，而且各地都不尽相同。对于企业来说，不遵守相关规定会带来财务和法律风险。因此，我们启动项目时选择从美国开始。And that's because Muse AI is only available in the US and Canada. And this is because AI regulation is constantly changing and is different all over the world. And there is a financial and legal risk to companies to not be compliant with that. So, when we started our project, we started in the US.

我们在美国上线了，但也就到此为止了。也就是说，我们不得不停下来。我们没法把在美国推出的这项功能原样上线到其他国家。那么，我们是怎么做的呢？We shipped in the US, but then that's it. Like we had to stop. We were not able to ship the feature that was available in the US as is to other countries. And so, what did we do?

我们几乎立刻就意识到，无论你是不是产品经理，无论你是不是有志成为产品工程师或前沿部署工程师，每个人都需要开始成为通才，并从一开始就考虑自己正在构建的产品在合规和法律方面的应用。不能等到以后再考虑。所以，这意味着你可以采用多种方法。你可以从这样一句话开始：“哦，我非常确定自己想把这个功能推向全球各地。”We realized almost immediately that whether you're a product manager or not, whether you're an aspiring product engineer, forward deployed engineer, everyone needs to start being a generalist and thinking about compliance and legal applications of products that they're building early on. It can't come later. So, what that means is you can have multiple approaches. You could start with, "Oh, I definitely know I want to ship this feature all over the world.

因此，我可以先了解所有限制，并在这些限制范围内做出产品。或者你也可以选择这样做：“嘿，我知道美国是最能让我发挥影响力的地方，所以我会在美国推出尽可能最好的产品，之后也许就不在其他地方推出了。”面对这组新数据，你需要做出一系列不同层次的决策。这已经不只是开发产品了。如果你做出了最好的产品，却无法让用户用上，那就毫无意义。所以，你可以选择“好，那就只做一个国家”，也可以承担额外成本，制作一个能在不同国家以不同方式运行的应用版本。Therefore, let me understand all the restrictions and build something within all of those things." Or you could go the approach of, "Hey, I know that US is where I'm going to get my maximum impact and therefore I will launch the best thing possible in the US and then maybe not anywhere else." So, there's going to be a spectrum of decision-making that you will have to make with this new set of data. It's not just about building anymore. If you build the best thing, but you can't get it in the hands of users, it doesn't mean anything. So, you could take the path of, "Okay, one country." Or you could take the additional cost of making a version of the app which works differently in different countries as well.

现在需要做出的就是这个决定。That's the decision that now needs to be made.

要问的五个问题Five questions to ask

总而言之，我认为在开发任何 AI 功能时，大家都应该问自己这五个问题：人们真的在用它，还是只是说他们会用？你们会查看实际输出，还是只看平均值？把解决方案涉及的所有成本都考虑进去之后，它还划算吗？So, to summarize, I think whenever it comes to AI features, when you all are building, ask yourself these five questions. Are people using it or are they just saying that they will use it? Do you read real outputs or just look at the averages? Does it still pay off after factoring in all the costs that went into the solution?

在最佳模型上取得的胜利，是你负担得起的吗？你是否清楚从法律角度来说，可以在哪些市场推出产品？因为现在的问题已不再只是功能是否在 A/B 测试中胜出。谢谢。Is a win on the best model something you can afford? And are you clear legally on which markets you can ship in? Because the question is not just did the feature win the AB test anymore. Thank you.