我们如何打造兼顾用户信任与相关性的 ChatGPT 全新广告系统

传统广告系统旨在响应离散的用户意图信号。用户在线搜索或浏览某些内容，广告系统便会将这些搜索词和页面浏览记录作为事件接收，并据此寻找匹配的广告。为 ChatGPT 构建广告技术栈则有所不同：用户意图会在对话中逐渐形成，并随着多轮对话而变化，有时甚至不会出现合适的广告时机。为了支持这一新范式，我们构建了一套定制的广告投放系统。这种新型广告技术栈基于一个经过后训练的定制前沿大语言模型，该模型会调用工具来执行检索（找出最佳选项）、排序（预测用户参与度）、相关性评估（评估广告的实用性），并运行拍卖以最终选出广告并确定价格。Traditional advertising systems are built to respond to discrete units of user intent. A user searches for or views something online, the ads system receives those queries and page views as events, and the system looks for matching ads based on those events. Building an ads stack for ChatGPT was different: user intent develops across conversations and shifts over multiple turns of a conversation, sometimes without a fitting advertising moment. To support this new paradigm, we built a bespoke ads delivery system. This new type of ad stack is based on a custom, post-trained frontier LLM that makes tool calls for retrieval (identifying the top options), ranking (predicting user engagement), relevance (evaluating the usefulness of the ad), and running auctions for final ad selection and pricing.

我们的广告系统通过三项关键设计决策，支持用户目标不断变化的复杂性：1）实现有状态的广告会话，也就是说，系统会记住对话中的多轮信息，以及多次对话之间的信息；2）采用定制的后训练大语言模型，并通过工具调用架构理解对话上下文和潜在意图，进而有效地选择、排序和投放广告；3）将广告选择流程放在独立于 ChatGPT 答案生成的专属服务中运行，避免广告影响答案，并保持 ChatGPT 中立的表达风格。我们利用后训练大语言模型编排整个广告流程，结合用户当前这轮对话和历史记录，判断何时展示哪则广告最合适。过去，构建这样一个能够自我改进的广告系统需要数年时间；而 ChatGPT Ads 在推出不到 200 天后，年化营收运行率便达到 10 亿美元。如今，自推出以来，已有超过 10 万名广告主在 ChatGPT 上投放过广告。Our ads system supports the nuance of evolving user goals by making three key design decisions: 1) implementing stateful ads sessions, meaning information is remembered across turns of a conversation and across multiple conversations, 2) relying on a custom, post-trained LLM that uses a tool call architecture to interpret a conversation’s context and underlying intent, and then effectively select, rank, and deliver ads, and 3) running the ads selection process separately from ChatGPT’s answer generation, in its own service, to keep ads from influencing answers and maintain ChatGPT’s neutral voice. We use our post-trained LLM to orchestrate the entire ads flow, determining the right ad to surface at the right time based on both the user’s current turn and conversation history. While bootstrapping a self-improving ads system like this used to take years, ChatGPT Ads reached $1 billion in annualized revenue run rate less than 200 days after launch; now, more than 100,000 advertisers have run ads on ChatGPT since launch.

传统的广告请求通常始于一个定义明确的事件。用户提交搜索查询或打开网页，广告系统便会收到请求。这类系统相对简单：它会检索潜在的广告候选项，对其进行排序，通过竞价为展示机会合理定价，并向用户展示最终的广告。ChatGPT 并不会自然地产生这样一个明确的时刻。A conventional advertising request usually starts with a well-defined event. A user submits a search query or opens a webpage and the ads system receives a request. This type of system is fairly simple: it retrieves potential ad candidates, ranks them, runs an auction to appropriately price the opportunity, and shows the resulting ad to the user. ChatGPT does not naturally produce such a concrete moment.

在 ChatGPT 对话中，人们带着各种目标而来，而不是只想搜索一个问题、得到一个答案。他们可能在规划旅行、为企业挑选软件、布置家居，或学习一项新爱好。这些事情往往无法在一轮对话中完成。ChatGPT 经常会在过程中帮助用户理清哪些方面最重要。In a ChatGPT conversation, people arrive with goals rather than a one-answer search query. They might be planning a trip, choosing software for their business, furnishing a home, or learning a new hobby. This doesn’t all happen in one turn of a conversation. ChatGPT often helps the user clarify what matters along the way.

例如，我们的工程团队曾询问 ChatGPT 如何为一场比赛进行训练：对话从一个关于跑步的一般性问题开始。几轮交流后，他们开始询问有助于训练的跑鞋。随着每项限制条件逐渐提出，对话的商业含义也随之变化，而且这种个性化上下文会在多次会话中不断演变。如果系统依据第一条消息采取行动，可能会完全判断错用户意图，进而损害用户体验，甚至妨碍用户。As an example, our engineering team asked ChatGPT how to train for a race: A conversation begins with a general question about running. Several turns later, they ask about shoes to help train. The commercial meaning of the conversation changes as each constraint arrives—and this personalized context evolves across multiple sessions. A system that acted on the first message could retrieve the wrong intent entirely, potentially degrading the user’s experience and getting in their way.

构建我们的广告系统，要求我们超越请求-响应式广告架构，从零搭建广告技术栈，以发掘潜在的经济机会，同时优先考虑用户信任或价值。Building our ads system required us to move beyond the request-response advertising architecture and build an ads stack from scratch to find potential economic opportunities while prioritizing user trust or value.

如果像传统广告系统那样，将每条消息都转化为独立请求，在我们的聊天场景中会带来一些风险：1）过度简化用户意图，导致误解；2）在用户的真实需求尚未明确之前就过早启动广告流程；3）展示妨碍而非助力用户体验的广告；4）每次请求发生变化时都低效地重新运行广告流程，造成干扰并浪费成本。广告系统（由召回、排序和投放组成）并不围绕单个事件构建，而是维护一个有状态的会话。这意味着，在符合条件的轮次持续期间，广告会话会一直运行，并随着信息的演变不断记忆和更新信息。ChatGPT 会将初始请求上下文发送到广告服务器。广告系统会跟进并适应逐步展开的对话，而不是依赖单个扁平化查询。Turning every message into a standalone request, like a traditional ads system, would have carried a few risks in our chat use case: 1) misreading the user goal by oversimplifying it, 2) beginning the advertising flow too soon, before the user’s real need has become clear, 3) showing ads that get in the way of the user journey instead of supporting it, and 4) inefficiently rerunning the ads flow every time the request changes, creating noise and wasting cost. Instead of building around individual events, the ads system (consisting of retrieval, ranking, and delivery) maintains a stateful session, meaning the ads session runs for the duration of an eligible turn, remembering and updating information as it evolves. ChatGPT sends the initial request context to the ads server. The ads system follows and adapts to the developing exchange rather than relying on a single flattened query.

广告选择流程与 ChatGPT 的正常回复生成并行运行，以确保广告流程不会影响 ChatGPT 的回复。广告由一个独立系统根据自身的资格、质量和投放限制，利用对话内容进行筛选和选择。这一重要的架构边界——不同的服务使用各自独立的模型和代码——使 ChatGPT 核心服务器能够专注于生成自然回复；与此同时，广告系统会独立评估是否存在合适的广告机会，并在符合条件时，通过单独的投放流程准备一个赞助展示位。The ads selection process runs alongside ChatGPT’s normal response generation to ensure the ads flow doesn’t influence ChatGPT’s responses. Ads are filtered and selected through a separate system consuming the conversation under its own eligibility, quality, and delivery constraints. This important architectural boundary—different services that use separate models and separate code— lets core ChatGPT servers handle generating the organic response; meanwhile, the ads system independently evaluates whether there’s an appropriate advertising opportunity and, if so, prepares a sponsored placement through a separate serving process.

除了保护用户体验之外，这种隔离还带来了一些实际好处。ChatGPT 不依赖广告系统——这两项服务并未以任何方式交织，也不共享内容生成模型。因此，如果广告投放系统不可用或运行缓慢，ChatGPT 的自然回复仍会照常进行。这意味着广告系统不会导致 ChatGPT 产品宕机或延迟。这种隔离的另一项好处是，它为有价值的广告设定了很高的门槛。不展示广告也是一种正常且成功的结果。只有在广告显然对用户和广告主都有益时，系统才会展示广告。Beyond protecting the user experience, this separation has some practical consequences. ChatGPT has no dependency on the ads system—the services are in no way intertwined and don’t share a content generation model. So if the ads serving system is unavailable or slow, ChatGPT’s organic response continues unaffected. This means the ads system can’t take down or delay the ChatGPT product. The other benefit of this separation is that it keeps the bar very high for valuable ads. Returning no ad is an ordinary, successful outcome. The system only delivers an ad if it’s clearly beneficial to users and advertisers.

我们广告技术栈中的另一个关键架构组件，是经过后训练的 LLM。它负责解读对话，并通过智能体循环运行整个广告流程——调用广告系统的核心功能工具，涵盖检索、排序、竞价，最终到广告投放。我们称这个 LLM 为编排器。The other key architectural component in our ads stack is the post-trained LLM that interprets conversations and runs the entire ads flow in an agentic loop—making tool calls to the core functions of the ads system, from the retrieval stage to ranking to auction and, ultimately, delivery. We call this LLM the orchestrator.

广告流程的第一步是理解用户意图，以启动检索流程；这一流程会快速、广泛地搜集潜在广告，供后续筛选。随着对话内容逐步到来，检索问题会变得更丰富，但也更模糊，因为广告系统需要区分当前意图与过时的上下文。协调器必须解读用户对先前消息的引用，识别用户何时是在假设情境下发言，并避免把每次提及某种产品或服务都当成购买请求。The first part of an ads flow is understanding user intent to start the retrieval process, which is a fast and deliberately broad sourcing of potential ads to later filter through. Receiving the conversation incrementally creates a richer but more ambiguous retrieval problem because the ads system needs to distinguish current intent from stale context. The orchestrator must interpret references to earlier messages, recognize when a user is speaking hypothetically, and avoid treating every mention of a product or service as a request to buy something.

广告编排器会利用可用的对话和用户上下文，通过不断迭代寻找合适的广告，并反复决定何时调用哪种工具，最终目标是为该对话和上下文找到最合适的广告（兼顾用户价值与广告主价值的广告）。如果用户选择个性化广告，编排器还会考虑用户过往对话中可用的非敏感信息，以找到最匹配的广告。The ads orchestrator uses the available conversation and user context, iterating to find good ads and repeatedly choosing when to call which tool with the underlying goal of finding the most appropriate ad (the ad that balances high user value and high advertiser value) for that conversation and context. If the user chooses to personalize ads, the orchestrator also considers available nonsensitive information about the user’s past conversations to find the best ad matches.

随着对话的推进，广告编排器会获取更多最新信息。此时，它可以基于更新后的上下文重新调用一组工具，以找到更合适的广告并将其返回给客户端；如果这些新广告能及时到达，客户端就可以使用它们。As the conversation progresses and the ads orchestrator gets more updated information, the orchestrator can run a fresh set of tool calls with the updated context to find better ads and return them to the client, which can then use these new ads if they arrive in time.

该协调器旨在平衡用户价值与广告商价值，并始终将用户置于首位。The orchestrator is designed to balance user value and advertiser value, always putting users first.

广告系统由多个独立工具组成，编排器会反复调用这些工具，为对话找到最合适的广告。概括来说，编排器会将对话转换成简短的文本摘要，再将其转换为语义嵌入（即存储起来、便于广告系统筛选的含义），然后利用这些语义嵌入检索候选广告，筛选出符合投放资格的候选广告（依据预算、用户设置和其他初步检查），评估其质量和相关性，并进行最终排序，以找出适合在聊天回复末尾展示的高价值广告机会。因此，每则广告都必须达到较高的投放资格和相关性标准，才有资格展示。The ads system consists of individual tools, and the orchestrator calls them iteratively to find the best ads for a conversation. At a high level, the orchestrator turns conversations into brief text summaries, which are then turned into semantic embeddings (stored meanings that are easier for an ads system to sift through), retrieves ad candidates using those semantic embeddings, filters those candidates for eligibility (based on budget, user settings, and other preliminary checks), assesses their quality and relevance, and does a final ranking to identify high value opportunities for placement at the end of a chat response. Each ad must therefore meet a high bar on eligibility and relevance in order to be eligible for placement.

编排器 LLM 首先准备一个或多个查询，以体现相关的商业意图。这可能包括改写对话上下文，将商业可能性拓展到一对一搜索查询之外，然后生成语义表示；在此过程中，编排器会将多轮对话提炼成摘要版本，供检索系统使用，从而有针对性地从我们的数据库中检索广告。The orchestrator LLM first prepares one or more queries representing the relevant commercial intent. This can include rewriting conversation context to broaden the commercial possibilities beyond 1:1 search queries and then creating semantic representations, where the orchestrator distills a multiturn conversation into a summarized version for the retrieval system—retrieving ads from our database in a focused way.

只有在深入理解用户意图时才展示广告，才能让用户和广告主双方受益。多个查询可以保留意图的不同方面。例如，关于规划旅行的对话可能包含交通、住宿或活动等方面的不同需求。检索系统可以保留这些区别，而不是把整段对话压缩成一个字面字符串。Ad placements will be better for both users and advertisers if they only appear when there’s deep understanding of the user’s intent. Multiple queries can preserve different aspects of intent. A conversation about planning a trip, for example, might contain distinct needs related to transportation, lodging, or activities. The retrieval system can preserve these distinctions rather than compressing the entire conversation into one literal string.

举例来说，让我们来看一段关于徒步鞋的潜在对话序列：As an example, let’s look at a potential conversation sequence about hiking shoes:

1. 两天的优胜美地之旅应该安排哪些景点？

   What should I cover on a two-day trip to Yosemite?

2. 下周会冷吗？

   Will it be cold next week?

3. 新推出的[插入品牌]产品线？

   New [insert brand] product lines?

尽管用户没有将鞋类品牌问题与优胜美地之行联系起来，但在我们的广告系统中，这段对话可能会由两个查询来表示，例如[适合徒步旅行的鞋子]和[适合旅行的新品 [品牌] 产品线]。这可能只是第一次尝试；编排器的设计目的，是不断迭代这些表示方式，直到找到我们广告库存中对用户极具价值的广告。Although the user hasn’t connected the shoe brand question to the Yosemite trip, this conversation might be represented in our ads system by two queries, such as [shoes appropriate for hiking trip] and [new [brand] product lines good for travel]. This might only be the first attempt; the orchestrator is designed to iterate on these representations until it finds ads in our inventory that are highly valuable to the user.

检索是有意构建一个范围广泛的广告候选池的过程。候选广告可以来自不同的检索策略，之后可在相同的投放约束下竞争。我们构建了多个索引，分别针对不同的系统和产品约束进行优化（例如，商品信息流广告的索引会考虑商品的正面或负面评价，以便在检索时找到尽可能好的广告。这一点与其他可能没有相关评价的广告无关）。每个索引都针对某种近似最近邻（ANN）搜索进行优化，并使用表示用户意图的嵌入向量。这种 ANN 搜索是一种针对大型语义索引进行检索的近似算法：它能非常快速地检索广告，在保留大部分相关候选项的同时，力求找到较好的结果，而不是寻找“正确”的结果。Retrieval is the process of creating a deliberately broad pool of ad options. Candidates may come from different retrieval strategies and can later compete under the same serving constraints. We build multiple indices that optimize for different system and product constraints (for example, the index for product feed ads is designed to be aware of positive or negative product reviews as it fetches the best possible ads. This is not relevant to other ads that may not have reviews associated with them). Each index is optimized for some version of approximate-nearest-neighbor (ANN) search, using embeddings representing the user’s intent. This ANN search is an approximation algorithm for retrieval over large semantic indices, meaning ads are fetched very quickly, getting close to a good solution rather than looking for “the right” solution, while retaining most of the relevant candidates.

过滤的目标是排除所有不符合投放标准的广告候选项，评估标准包括相关性、出价和预算。根据候选项和配置的不同，过滤检查可能包括创意素材和广告系列的可用性（广告系列或广告创意是否处于启用状态）、广告系列定向（例如广告系列获准投放的地区）、政策限制（广告内容必须满足的合规要求）、预算可用性（广告系列是否仍有可用预算）、投放节奏（广告系列现在是否应该消耗预算，还是应在当天其他时间消耗）、版位兼容性（此广告适合投放在哪些类型的对话中），以及广告主级别和广告系列级别的去重（为每位广告主选择适当数量的广告，确保各家企业都能公平参与）。The goal of filtering is to eliminate all ad candidates from consideration that don't meet the bar for placement when it comes to relevance, bids, and budget. Depending on the candidate and configuration, filtering checks can include creative and campaign availability (whether the campaign or ad-creatives are active), campaign targeting (e.g., which locales a campaign is allowed to operate in), policy restrictions (compliance requirements around what can be advertised), budget availability (whether or not a campaign still has budget available to spend), pacing (whether a campaign should be spending budget now versus another time in the day), placement compatibility (what kinds of conversations this advertisement is eligible for), and advertiser and campaign level deduplication (picking an appropriate number of ads per advertiser to ensure fair participation across businesses).

通过资格检查的候选广告将进入评分阶段，协调器会调用各种评分系统进行评估，以满足投放方面的考量，例如语义相关性（广告是否与用户的目标相关？）、预估互动度（用户是否可能浏览、点击广告，并促成广告主期望的结果？）以及历史表现（截至目前，该广告在向用户展示时的表现是否良好？）。竞价前，协调器还会筛选广告，以确保其相关性和安全性。这些系统会逐步缩小候选范围，形成一个可参与竞价的候选广告池，但它们仍不会决定最终投放的广告。Candidates that survive eligibility checks move on to a scoring phase, where the orchestrator makes tool calls to various scoring systems—for serving considerations such as semantic relevance (is the ad relevant to the user’s goal?), predicted engagement (is the ad likely to get viewed, clicked, and converted into the advertiser’s desired outcome?), historical performance (has the ad performed well when shown to users so far?). The orchestrator also filters ads to ensure relevance and safety before the auction. These systems narrow the field into a smaller candidate pool ready for auction, but they still don’t determine the final delivered ad.

编排器会结合对话上下文评估准备好的候选广告，并选择足够合理、可以进入最终竞价的候选项。编排器不会直接决定展示哪条广告，也不会设定价格。它会提名与上下文相符的候选广告，供竞价工具评估。The orchestrator evaluates the prepared candidates in the context of the conversation and chooses which ad candidates are plausible enough to enter the final auction. The orchestrator doesn’t directly decide which ad will be served, nor does it set a price. It nominates contextually appropriate candidates for the auction tool to evaluate.

拍卖流程会应用最终的投放机制，包括拍卖准入（达到相关性和价值的最低门槛）、投放节奏控制（使广告预算消耗更加均匀）、出价和拍卖特征计算、质量调整后的排序、定价以及最终选择。The auction applies the final serving mechanics, including auction admission (passing a minimum threshold of relevance and value), pacing (spending ad budget more evenly), bid and auction-feature computation, quality-adjusted ranking, pricing, and final selection.

竞拍结束后，广告可能无法投放的原因有两种，这两种情况都特有于 ChatGPT 的对话媒介。After the auction, there are two ways an ad might fail to deliver, both specific to the ChatGPT conversational medium.

1. 这则广告在聊天中可能显得不合适：尽管完整性模型（负责判断何时适合投放广告的模型）认定这段对话适合展示广告，且该广告本身也符合政策要求，但有时广告在某些对话语境中仍可能显得不合适。协调器会阻止展示这些广告。

   The ad might not feel appropriate in the chat: Even though the conversation itself was deemed safe for ad placement by integrity models (the models that decide when it’s appropriate to serve an ad), and the ad itself was deemed policy-compliant, sometimes the ad might feel inappropriate in the context of certain conversations. The orchestrator blocks these ads from being shown.

2. 广告请求可能无法及时送达：如果 ChatGPT 模型已经准备好回复，但广告尚未准备好投放，广告请求就会超时，广告也无法送达。因此，广告系统的目标之一就是尽可能快速地运行。ChatGPT 模型的响应时间是动态的，我们不会为了等待广告加载而延迟聊天回复。

   The ad request might not arrive in time: If the ChatGPT model is ready to respond, but the ad is not yet ready to place, the ad request times out and the ad doesn’t get delivered. One goal of the ads system, therefore, is to run as fast as possible. The ChatGPT model response time is dynamic, and we don’t delay chat responses to wait for ads to load.

如果上述所有检查均通过，我们就会将广告发送到前端进行渲染。我们将此视为服务器端广告投放，但这并不意味着一定会向用户展示广告——这是有意为之。If all of the above checks are met, we send the ad to the frontend for rendering. We consider this a server-side ad delivery, but we still might not show the ad to the user—and this is by design.

在 ChatGPT 这类对话式产品中，服务器端送达广告素材并不意味着广告一定会展示；广告还必须及时到达，才能纳入当前轮次。广告素材包括标题、描述、目标链接以及任何行动号召。此外，广告系统还会返回一段由模型生成的前导语，即一段简短、中性的介绍，用来说明广告与对话之间的关联。将前导语与广告素材分开，既能保留广告主传达的信息，也能让 ChatGPT 保持中立的语气。In a conversational product like ChatGPT, server-side delivery of an ad creative does not guarantee an ad will be displayed; the ad must also arrive in time to be included in the current turn. The ad creative includes the title, description, destination link, and any call to action. Additionally, a model-generated preamble, which is a short, neutral introduction explaining how the ad relates to the conversation, is also returned by the ads system. Keeping the preamble separate from the creative preserves the advertiser’s message while keeping ChatGPT’s voice neutral.

ChatGPT 前端会以捆绑形式接收前言和广告创意。如果它们在 ChatGPT 的回复完全生成之前到达，前端就会在 ChatGPT 的回复下方展示广告，并明确标注为赞助内容。如果到达得太晚，该轮对话中就不会显示广告。这就是服务器端广告投放与面向用户的广告展示之间的区别。The ChatGPT frontend receives the preamble and ad creative as a bundle. If it arrives before ChatGPT's response is fully generated, the frontend displays the ad below ChatGPT’s response, clearly labeled as sponsored. If it arrives too late, the ad is not shown for that turn. This is the difference between server-side ad delivery and a user-facing ad impression.

我们的广告技术栈采用传统的日志记录和数据管道，主要有三项优势：广告主可以更轻松地将现有系统与我们的技术栈集成；广告主可以利用报告数据改进策略；我们的团队也能据此发现系统中有待改进的环节。除了这些报告和优化工具，我们还部署了一个高质量模型，用于评估广告投放效果并训练其他广告模型，从而形成强大的自我优化闭环。Our advertising stack uses traditional logging and data pipelines, which have three main benefits: advertisers can more easily integrate their existing systems with our stack, advertisers can improve their strategies with the reporting data, and our team can find areas of the system to improve. Beyond these reporting and optimization tools, we deploy a high quality model to evaluate our delivery and train other ads models, creating a powerful self-improvement loop.

对事件经过进行详细拆解，有助于我们分别理解并改进广告系统的各个部分。日志处理系统还会记录细粒度的结果，例如：Detailed breakdowns of what happened help us understand and improve each part of the ad system in isolation. The logs processing system also records granular outcomes such as:

• 该轮次不符合投放广告的条件。

  The turn was not eligible for advertising.

• 广告会话已启动，但未请求检索。

  An ads session started but no retrieval was requested.

• 检索未返回任何候选项。

  Retrieval returned no candidates.

• 已检索到候选项，但未能完成数据水合或通过资格检查。

  Candidates were retrieved but failed hydration or eligibility.

• 广告已投放，之后收到了互动。

  An ad was delivered and later received an interaction.

这些详细拆解有助于我们分别改进广告投放系统的各个部分。改进机会识别所需的证据，与改进检索、候选项质量、竞价参与度或投放可靠性所需的证据各不相同。These detailed breakdowns help us improve each part of the ad serving system in isolation. Improving opportunity detection requires different evidence from improving retrieval, candidate quality, auction participation, or delivery reliability.

基于模型的评估，其核心理念是：由我们最强大、推理能力最高的模型观察并评判系统的结果，然后通过训练让后训练阶段的 LLM 表现得更好。例如，如果由最强大的模型收集反馈，并训练这些小型工作模型尽可能出色地完成任务，我们就可以用更小、更高效或推理能力较低的模型来负责编排或检索。The idea behind model-based evaluation is that our strongest, highest-reasoning models watch and judge the results of our system, then improve the post-trained LLM by training it to perform better. For instance, we can use smaller, more efficient models or lower-reasoning models for orchestration or retrieval if our most powerful model gathers feedback and trains these small workers to do the best job possible.

我们会收集并根据用户反馈（点击和隐藏）采取行动，也会开展用户调查，并通过我们的广告管理工具直接收集广告主的反馈。在产品开发过程中，我们也随时听取团队成员的意见，并直接倾听 ChatGPT 用户和广告主的需求。相比模型的观察性反馈，这些反馈更能影响我们的发展方向，也是我们能够快速迭代、做出正确产品决策的原因之一。模型或许能够理解用户意图并进行批判性思考，从而提供有价值的内容，但品味是由人塑造的。We gather and act on human feedback (clicks and hides from users), as well as surveys for users and direct feedback from advertisers through our Ads Manager. We also make our team available for feedback as we build, and directly listen to what both ChatGPT users and advertisers want. This source of feedback informs our direction even more than the model’s observational feedback and is part of what allowed us to iterate toward the right product decisions quickly. The model may be able to understand user intent and think critically to deliver something valuable, but people shape taste.

一年前，我们还没有广告系统，只有少数几个人在思考这个领域的问题。如今，我们已经拥有一套成熟的广告系统，年化营收规模超过 10 亿美元，并由一个基于 LLM 的编排器驱动广告流程。Codex 是至关重要的效能倍增器，让我们能够以无与伦比的速度从零搭建系统，并实现这一重要的使用量里程碑。从一开始，我们就围绕使用 Codex 快速构建和扩展系统来组建团队、设计开发工作流程。我们为 Codex 建立了防护措施，确保代码能够可靠、安全地交付。同时，我们还为 Codex 配置了数据基础设施插件，大幅缩短了洞察发现与系统改进落地之间的周期，涵盖新产品功能、质量提升和系统性能优化。A year ago, we had no ads system and only a few people thinking about this problem space. Now we have a robust ads system powering more than $1 billion in annualized run rate, with an LLM-based orchestrator running the ads flow. Codex was a fundamental force multiplier that allowed us to bootstrap our system to achieve this major usage milestone with unparalleled speed. From the start, we set up the team and development workflows to use Codex to rapidly build and scale the system. We built guardrails for Codex to reliably and safely ship code. Inn parallel, we set up data infrastructure plugins for Codex to massively tighten the loop of insight gathering and landing improvements to the system, including new product features, quality improvements, and better system performance.

我们的系统边界（以我们的广告原则为基础）与基于 LLM 的编排器相结合，使我们能够让广告发挥实用价值，而不把每次对话都视为广告请求。展望未来，我们将继续尝试更多动态广告形式，以提升用户价值（例如，让用户能够通过广告直接与商家交流）和广告主价值。该技术栈采用基于 OpenAI 模型并配有持续反馈循环的自我改进架构，这将加快我们不断提升相关性和个性化水平的步伐。The combination of our system’s boundaries (grounded in our advertising principles⁠) and our LLM-based orchestrator lets us make advertising useful without treating every conversation as an ad request. Looking ahead, we’ll continue to experiment with more dynamic ad formats to increase user value (e.g. giving users the ability to talk directly to businesses through ads) and advertiser value. The stack’s self-improving architecture, built on OpenAI models with continuous feedback loops will accelerate our ability to keep improving relevance and personalization over time.