CASTLE：在第一次真实查询之前训练自然语言搜索

我们如何利用去标识化的预订和搜索信号，联合生成合成训练数据（查询和相关性标签），以解决 Airbnb 自然语言搜索的冷启动问题。How we addressed the cold-start problem for Airbnb’s natural language search by jointly generating synthetic training data — queries and relevance labels — from de-identified booking and search signals.

By: Wendy Ran Wei, Hao Li, Weiwei Guo, Xiaowei Liu, Xueyin Chen, Chun How Tan, Malay Haldar, Soumyadip Banerjee, Kedar Bellare, Huiji Gao, Sanjeev Katariya, and Stephanie Moyerman

引言Introduction

自然语言搜索让房客能够用自己的语言表达需求，例如“滑雪场附近的浪漫小木屋”或“可携带宠物、能住八人的海滨别墅”，而无需浏览各种筛选条件和类别。我们需要为 Airbnb 推出这种搜索方式，并确保它从一开始就能良好运行。然而，在我们还没有真实用户查询可用于训练模型的情况下，就必须先训练模型；同时，还需要一套合理的相关性标签（根据查询推断得出），将查询与匹配的房源对应起来。在没有先前流量的情况下，这两者都必须从零开始合成。这就是自然语言搜索的冷启动问题。Natural language search lets guests express intent in their own words, such as “romantic cabin near skiing” or “pet-friendly beach house that sleeps eight,” rather than navigating filters and categories. We needed to introduce this kind of search for Airbnb, and to have it work well from the start. However, this required training the model before we had actual user queries to train the model on, and a reasonable set of relevance labels (inferred from the queries) to map queries to matching listings. Without prior traffic, both must be synthesized from scratch. This is the cold-start problem for natural-language search.

其他组织也面临过类似挑战，并创建了相应的方法来应对。然而，现有方法分别处理查询集合和相关性标签这两项需求。基于段落的方法（如 InPars [1]）会提示大语言模型（LLM）根据单篇文档生成查询，然后依赖额外的 LLM 判断来分配相关性标签，从而引入第二种误差来源。Promptagator [2] 通过少样本示例提高了查询的多样性，但仍受限于其生成分布与真实用户行为之间的差距。Other organizations have faced similar challenges and have created methods to address them. However, prior approaches address the need for a body of queries and the need for relevance labels separately. Passage-based methods such as InPars [1] prompt a large language model (LLM) to generate queries conditioned on a single document, then rely on additional LLM judgments to assign relevance labels, introducing a second source of error. Promptagator [2] improves query diversity using few-shot examples, but is still limited by the gap between its generation distribution and real user behavior.

在这篇文章中，我们介绍了面向自然语言搜索的对比式与种子引导训练（Contrastive And Seed-guided Training for natural Language sEarch，CASTLE），这是一个利用 Airbnb 自有的去标识化预订数据，同时生成查询并标注相关性的框架。介绍 CASTLE 的论文已被第 35 届 ACM 信息与知识管理国际会议（简称 CIKM 2026）接收为完整研究论文 [3]。In this post, we describe Contrastive And Seed-guided Training for natural Language sEarch (CASTLE), our framework for jointly generating queries and producing relevance labels from Airbnb’s own de-identified booking data. Our paper describing CASTLE was accepted as a full research paper at the 35th ACM International Conference on Information and Knowledge Management, more widely known as CIKM 2026 [3].

挑战Challenges

为搜索生成有用的合成训练数据，不只是生成语法通顺的查询。合成查询的分布必须符合真实用户的实际搜索方式，包括他们偏好的措辞、查询长度，以及他们通常会提及的属性。如果查询不符合现实中的搜索模式，基于这些数据训练的模型就难以良好泛化。Generating useful synthetic training data for search requires more than producing grammatically plausible queries. The distribution of synthetic queries must match how real guests actually search; their preferred phrasing, query length, and attributes they tend to mention. If queries fail to match real-world search patterns, the model trained on them will generalize poorly.

我们使用合成查询分布与真实查询分布之间的 Kullback–Leibler（KL）散度（衡量一个概率分布与参考分布差异程度的指标）来测量这种不匹配，比较的维度既包括表层查询统计数据，也包括属性类型（房产类型、便利设施、位置等）。以 KL 散度指标衡量，先前方法的得分介于 9.33 和 26.2 之间，与真实用户分布相去甚远。除了分布对齐之外，我们还需要相关性标签。生成查询只是问题的一半；确定哪些房源与之相关，则是另一半。We measured this mismatch using Kullback–Leibler (KL) divergence (a measure of how much one probability distribution differs from a reference distribution) between synthetic and real query distributions — both over surface-level query statistics and over attribute types (property type, amenity, location, and so on). Prior approaches scored between 9.33 and 26.2 on the KL divergence metric: far from the real-user distribution. Beyond distributional alignment, we also needed relevance labels. Generating a query is only half the problem; knowing which listings are relevant to it is the other half.

解决方案Solution

CASTLE 通过一种机制同时应对这两项挑战：根据用户隐私授权，从去标识化的预订会话中筛选出对比式房源对。客人浏览多个备选房源后预订了某个特定房源，这一行为便隐含地体现了偏好：该房源胜过其他房源，被客人选中。我们将这些房源对（正例、负例）作为查询生成的结构化输入，其中负例是从其他浏览过但未预订的备选房源中随机抽取的。房源特征包括便利设施、房屋详情，以及对近期公开住客评价的一句话总结；其中的个人身份信息已被移除。由于我们要求 LLM 生成一个适用于正例房源、但不适用于负例房源的查询，因此相关性标签从构造上就是正确的，无需事后判断。至关重要的是，“正例”和“负例”由用户过去的实际预订行为决定，而非由 LLM 自行判定，因此后续生成相关性标签的过程不存在循环论证。CASTLE addresses both challenges through a single mechanism: contrastive listing pairs derived from de-identified booking sessions, selected in accordance with user privacy permissions. When a guest books a specific listing after viewing several possible alternatives, that event implicitly encodes preference: the booked listing was selected over the others. We use these (positive, negative) pairs of listings — where the negative is randomly sampled from the other viewed-but-not-booked alternatives — as the structured input for query generation. Listing features include amenities, property details, and a one sentence summary of recent public guest reviews, with personal identifying information removed. Because we instruct the LLM to generate a query that is satisfied by the positive listing but not the negative, the relevance label is correct by construction; no post-hoc judgment is required. Crucially, what is “positive” and “negative” is determined by actual, previous user booking behavior, not by the LLM itself, so there is no circularity in the subsequent generation of relevance labels.

查询生成。我们结合三种提示变体，以最大限度地提高覆盖面和真实性。基于模板的提示确保在常见查询模式中实现结构多样性。少样本提示使用约 500 条从目标用户研究中收集的种子查询作为语言示例，使生成的查询贴近真实住客的表达方式。基于属性的提示会围绕两个房源之间的具体差异属性来构造查询，例如壁炉或靠近滑雪度假村。明确的多样性机制会促使模型在不同样本中变换查询长度和措辞，从而避免“查询坍缩”——即未受条件约束的 LLM 反复生成同一个简短短语。Query generation. We combine three prompt variants to maximize coverage and realism. Template-based prompts ensure structural diversity across common query patterns. Few-shot prompts use approximately 500 seed queries collected from targeted user research as linguistic exemplars, grounding generated queries in authentic guest language. Attribute-grounded prompts focus the query on specific distinguishing attributes between the two listings, such as a fireplace or proximity to a ski resort. An explicit variety mechanism encourages the model to vary query length and phrasing across samples, preventing “query collapse,” where an unconditioned LLM converges on the same short phrase repeatedly.

这些变体共同生成的词数分布标准差为 σ = 2.48，与真实用户（σ = 2.49）非常接近；相比之下，不含种子引导机制的基线方法（InPars 和 Promptagator）为 σ = 0.99。值得注意的是，真实的 Airbnb 房客往往使用简洁、聚焦意图的短语进行搜索，通常为四到八个词，而不是冗长的自然语言句子。CASTLE 旨在匹配这种真实世界的分布，而非生成刻意冗长的查询。Together, these variants produce a word-count distribution with standard deviation σ = 2.48, closely matching real users (σ = 2.49), compared to σ = 0.99 from baselines (InPars and Promptagator) without the seed-guided mechanism. It is worth noting that real Airbnb guests tend to search with concise, intent-focused phrases, typically four to eight words, rather than long natural-language sentences. CASTLE is designed to match this real-world distribution, not to generate artificially verbose queries.

灵活的输入设计。对于无法进行对比式 grounding 的房源对，例如没有预订记录的新上架房源，我们会补充使用虚拟评审器（VJ）——一种为额外的查询–房源对分配相关性标签的 LLM 评估器。这种做法扩展了对长尾意图的覆盖范围，同时不牺牲核心训练集通过对比式构造获得的质量保证。我们还引入了真实查询种子变体 CASTLE-R，专为已有部分真实流量积累的生产环境而设计。Flexible input design. For listing pairs where contrastive grounding is not available, such as newly listed properties with no booking history, we supplement with a Virtual Judge (VJ), an LLM evaluator that assigns relevance labels to additional query–listing pairs. This extends coverage to long-tail intents without sacrificing the quality guarantees of the contrastive construction for the core training set. We also introduce a real-query seed variant, CASTLE-R, designed for production settings where some real traffic has begun to accumulate.

结果Results

我们从查询分布对齐和标签质量两个方面，将 CASTLE 与此前的两种方法——InPars 式生成和 Promptagator——进行评估。We evaluate CASTLE against two previous approaches, InPars-style generation and Promptagator, on two axes: query distributional alignment and label quality.

在与真实用户查询进行 KL 散度比较时，CASTLE 的得分为 1.01，较最佳基线 Promptagator（9.33）提升了 9.2 倍。这种优势在属性类型层面同样成立：CASTLE 的属性类型 KL 散度为 0.08，甚至低于我们人工整理的问卷种子查询（0.09），这意味着生成的查询比种子查询本身更忠实地捕捉了真实住客意图的多样性。On KL divergence against real user queries, CASTLE achieves 1.01 — a 9.2× improvement over the best baseline (Promptagator at 9.33). The improvement holds at the attribute-type level as well: CASTLE’s attribute-type KL of 0.08 is lower than even our hand-curated survey seed queries (0.09), meaning the generated queries capture the diversity of real guest intents more faithfully than the seeds themselves.

为验证标签质量，我们邀请三位独立的领域专家对抽样的 200 组三元组进行了人工评估。三位评审者对 CASTLE 相关性判定的认可率均为 91–93%，证实这种对比式构造方法无需人工参与标注流程即可生成高质量的相关性标签。To validate label quality, we conducted a human evaluation with three independent subject matter experts reviewing 200 sampled triplets. Annotators agreed with CASTLE’s relevance assignments at 91–93% across all three raters, confirming that the contrastive construction produces high-quality relevance labels without any human annotation in the loop.

除了冷启动之外，随着真实流量不断积累，CASTLE 生成的合成数据仍能持续发挥价值。由于生成过程以特定房源特征为条件，它会自然地对自然流量中很少出现的长尾意图进行过采样，例如客人偶尔才会搜索的房源类型或设施组合；而这些意图一旦出现，往往又十分重要。Beyond cold-start, CASTLE’s synthetic data continues to provide value once real traffic accumulates. Because generation is conditioned on specific listing characteristics, it naturally oversamples long-tail intents that appear rarely in organic traffic, such as property types or amenity combinations that guests only search for occasionally, but that have high importance when they do appear.

结论Conclusion

CASTLE 的关键洞见在于，真实的预订行为与 LLM 生成并非相互竞争，而是彼此互补。来自预订会话的对比式房源对以真实的平台行为为查询生成提供依据，并天然提供相关性标签；而 LLM 提示则带来了仅凭行为数据无法提供的查询多样性和语言表达丰富度。二者结合，能够弥合合成用户分布与真实用户分布之间的差距，而单独采用任何一种方法都无法做到这一点。The key insight in CASTLE is that authentic booking behavior and LLM generation are complementary, not competing. Contrastive listing pairs from booking sessions ground query generation in real platform behavior and provide relevance labels by construction, while LLM prompting provides the query diversity and linguistic variety that behavioral data alone cannot supply. Together, they close the gap between synthetic and real user distributions in a way that neither approach achieves independently.

展望未来，我们正在探索多轮对话式搜索：住客可能先搜索“滑雪场附近的小木屋”，再进一步 уточ明确“允许带狗的那种”。在这种场景下，冷启动数据生成会更加困难，而对比式落地方法或许能推广到更广泛的场景。如需了解完整的技术分析，请参阅我们发表于 CIKM 2026 的论文：arxiv.org/abs/2605.21812（DOI：10.1145/3799682.3841071）。Looking ahead, we are exploring multi-turn conversational search, where a guest might begin with “cabin near skiing” and refine to “one that allows dogs” — a setting in which cold-start data generation is an even harder problem, and where the contrastive grounding approach may generalize further. For the full technical treatment, see our CIKM 2026 paper at arxiv.org/abs/2605.21812 (DOI: 10.1145/3799682.3841071).

有兴趣应对这类挑战吗？浏览我们之前发布的内容，或了解我们的开放职位。Interested in tackling these kinds of challenges? Browse our previous publications or explore our open roles.

致谢Acknowledgments

我们谨向 AI Platform—Relevance & Personalization 团队的各位合作者致谢，感谢他们在整个项目期间给予的支持。We would like to thank our collaborators across the AI Platform — Relevance & Personalization team for their support throughout this project.

参考文献References

[1] Jeronymo, Vitor 等。“InPars-v2：大型语言模型作为信息检索的高效数据集生成器。” arXiv:2301.01820，2023 年。[1] Jeronymo, Vitor, et al. “InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval.” arXiv:2301.01820, 2023.

[2] Dai, Zhuyun 等。《Promptagator：基于 8 个示例的少样本稠密检索》。ICLR，2023 年。[2] Dai, Zhuyun, et al. “Promptagator: Few-shot Dense Retrieval From 8 Examples.” ICLR, 2023.

[3] Wei, Wendy Ran 等。《CASTLE：面向冷启动自然语言搜索的对比式与种子引导式训练》。ACM 信息与知识管理国际会议（CIKM），2026 年。DOI：10.1145/3799682.3841071。[3] Wei, Wendy Ran, et al. “CASTLE: Contrastive and Seed-Guided Training for Cold-Start Natural Language Search.” ACM International Conference on Information and Knowledge Management (CIKM), 2026. DOI: 10.1145/3799682.3841071.

所有产品名称、徽标和品牌均归各自所有者所有。本文中使用的所有公司、产品和服务名称仅用于识别目的。All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this post are for identification purposes only.

CASTLE：在首次真实查询之前训练自然语言搜索，最初发表于 Medium 上的 Airbnb 技术博客，人们仍在通过标注和回应这篇文章继续讨论。CASTLE: training natural language search before the first real query was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.