我们能否信任 LLM 判断一个研究想法是否新颖？

📄 Paper | 📊 Data | 💻 Code

今天，我们发布《旧想法，新问题》（Old Ideas, Novel Problems），这项研究探讨大型语言模型评估科学研究构想新颖性的可靠程度。我们还将同步发布一套基准、数据和代码，用于测试新颖性评判器，并提供一条自动化流程，供研究人员构建新的评估集。Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas. Alongside the paper, we’re publishing a benchmark, data, and code for testing novelty judges, plus an automated pipeline researchers can use to build new evaluation sets.

在 Ai2，我们一直在开发支持科学发现不同环节的 AI 系统。DataVoyager 帮助研究人员探索和分析结构化数据，AutoDiscovery 生成并检验假设，而 Theorizer 则整合大量文献中的证据，提出候选科学理论。与 Asta 一同，这些项目展现了这样一类系统的潜力：帮助研究人员从理解既有研究，逐步迈向提出、检验和完善新想法。At Ai2, we’ve been developing AI systems that support different parts of scientific discovery. DataVoyager helps researchers explore and analyze structured data, AutoDiscovery generates and tests hypotheses, and Theorizer synthesizes evidence across large bodies of literature into candidate scientific theories. Together with Asta, these projects point toward systems that can help researchers move from understanding prior work to proposing, testing, and refining new ideas.

但生成研究思路只是实现科学发现自动化所面临的挑战之一。这些思路还必须带来真正的新贡献，这意味着这些系统需要能够区分真正新颖的研究方向与主要只是重新包装现有工作的方向。But generating research ideas is only part of the challenge of automating scientific discovery. Those ideas also need to contribute something genuinely new, which means these systems need a way to distinguish a novel direction from one that mostly repackages existing work.

这一判断通常会交给另一个 LLM——我们称之为新颖性评判器。新颖性评判器会评估一个研究想法相对于既有工作的创新程度，既可以单独评估，也可以与另一个想法进行比较。目前并没有构建这类评判器的标准方法；研究团队会自行选择模型、提示词、评估形式、推理预算，有时还会选用文献检索系统。这些选择可能会实质性地影响评判结果。然而，这类评判器通常只在人类撰写的论文上进行验证，而非在它们实际要评估的 AI 生成想法上进行验证。That judgment is often handed to another LLM—what we call a novelty judge. Novelty judges assess whether a research idea is new relative to prior work, either on its own or in comparison with another idea. There’s no standard way to build one; research teams choose a model, prompt, evaluation format, reasoning budget, and sometimes a literature-retrieval system. Those choices can materially affect the verdict. Yet these judges are often validated only on human-written papers rather than the AI-generated ideas they are actually meant to evaluate.

我们的新基准测试结果表明， novelty 评估本身远不够可靠。在六种 LLM 评审模型——GPT-5.1、GPT-5.2、GPT-5.4、Claude Sonnet 4.5、Claude Opus 4.5 和 Claude Opus 4.6——中，对评估设置做看似微小的改动，就会导致它们对相同创意给出截然不同的判断。在一项实验中，仅仅修改提示词中的几个词，就会让模型判断两个创意中哪个更新颖的准确率变化超过 50 个百分点。检索和额外推理并不能稳定提升判断效果，而两个专门构建的 novelty 评估系统，表现都不如仅通过提示词调用的 Claude Opus 4.6 基线模型。Our new benchmark results suggest novelty evaluation itself is far from reliable. Across six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas. In one experiment, changing just a few words in a prompt shifts a model’s accuracy in identifying which of two ideas is more novel by more than 50 percentage points. Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.

如果 AI 将在科学发现中发挥更大作用，我们就需要更可靠的方法来判断它是否真的提出了新想法。我们的工作为研究这一问题提供了工具，同时也表明仍有许多问题尚未解决。If AI is going to play a larger role in scientific discovery, we need more dependable ways to tell whether it is actually proposing something new. Our work provides tools for studying that problem, while showing how much remains unresolved.

为新颖性评判者构建基准测试 Building a benchmark for novelty judges

为了测试新颖性评审者能否可靠地区分新颖性较高和较低的研究构想，我们首先需要一些案例，其中有充分证据表明其底层科学贡献的原创性。To test how reliably novelty judges can distinguish more novel research ideas from less novel ones, we first needed examples where there was strong evidence about the originality of the underlying scientific contribution.

因此，我们围绕 ICLR 2026 的同行评审构建了一条自动化流程。我们没有把论文是否被接收当作新颖性的替代指标，而是从评审意见中挖掘出评审者明确讨论某项贡献原创性的段落。这一区别至关重要：只要论文在其他方面足够出色，即使并无特别新颖之处，也可能被接收。同样，一篇包含原创性贡献的论文也可能因其他方面存在不足而被拒稿。So we built an automated pipeline around ICLR 2026 peer reviews. Rather than treating paper acceptance as a proxy for novelty, we mined the reviews themselves for passages where reviewers explicitly discussed a contribution’s originality. That distinction is key; a paper can be accepted without being particularly novel as long as it is strong enough in other aspects. Likewise, a rejected paper with an original contribution can have other weaknesses.

我们通过筛选 ICLR 各研究领域中评分排名前 10% 的已接收论文，构建了一个包含 154 条由人类撰写的高新颖性研究想法的数据集；这些论文的审稿人均明确肯定了其新颖性，且没有审稿人对此提出异议。我们采用相反的标准构建了一个低新颖性数据集：从评分排名后 10% 的被拒论文中选取 145 条想法，这些论文的审稿人提出了原创性方面的担忧，且没有审稿人明确肯定其新颖性。在对评审挖掘流程进行的单独检查中，标注员在 50 个案例中的 47 个案例里认同所提取的新颖性信号极性；即使在意见不一致的案例中，也没有任何一例会改变某位审稿人被判定为赞扬还是质疑论文新颖性的结果。We created a high-novelty pool of 154 human-authored research ideas by selecting accepted papers in the top-rated 10% of their ICLR research area whose reviewers positively identified novelty and where no reviewer disputed it. We applied the reverse criteria to build a lower-novelty pool: 145 ideas from rejected papers in the bottom 10% whose reviewers raised originality concerns without any reviewer positively identifying novelty. In a separate check of the review-mining process, an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases; none of the disagreements would have changed whether a reviewer was counted as praising or disputing the paper’s novelty.

这两个数据池共同构成了我们的第一个评估场景——Human-Only，用于测试新颖性评审者能否区分有充分证据表明新颖性较高或较低的人类创意。但在实践中，新颖性评审者越来越多地被用于评估 AI 生成的创意。因此，我们创建了第二个场景——Human+Generated。Together, these two pools form our first evaluation setting, Human-Only, which tests whether novelty judges can tell apart human-authored ideas with strong evidence of higher or lower novelty. But in practice, novelty judges are increasingly being used to evaluate AI-generated ideas. So we created a second setting, Human+Generated.

对于这个样本池，我们将同样的 154 个经验证的人类构想，与一个简单的 LLM 生成器产出的 154 个构想配对。我们假设生成的构想不如对应的人类创作构想新颖；这些只是弱标签，并非对每个构想都经过核实的判断。生成器只接收了一个研究领域，以及一条要求提出新颖构想的指令，没有进行文献检索、使用工具、迭代或接收反馈。这样，我们就能对比基础生成器产出的构想与来自 ICLR 排名前 10% 的论文的贡献；这些论文的审稿人曾明确认可其原创性。For this pool, we paired the same 154 validated human ideas with 154 ideas produced by a simple LLM generator. We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea. The generator received only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration, or feedback. This gave us a conservative comparison between ideas from a basic generator and contributions from top-10% ICLR papers whose reviewers had explicitly endorsed their originality.

作为额外检查，一位专家独立审阅了30组人类创作与生成内容的配对，其中着重纳入了我们最强的评审判断错误的案例；我们还将人类和 AI 生成的创意改写为统一结构，以检验评审是否只是对写作风格差异作出反应。专家在全部30组中都更偏好人类创意，并认为生成的创意没有一项具有新颖性。As additional checks, an expert independently reviewed 30 Human+Generated pairs, heavily weighted toward cases our strongest judge got wrong, and we rewrote the human- and AI-generated ideas into a common structure to test whether judges were simply reacting to differences in writing style. The expert preferred the human idea in all 30 pairs and judged none of the generated ideas novel.

重要的是，我们的许多核心发现完全不依赖标签。如果两种合理的评估方案针对同一对观点给出相反的答案，这种分歧本身就揭示了不稳定性。Importantly, many of our core findings don’t depend on the labels at all. If two reasonable evaluation setups give opposite answers for the same pair of ideas, that disagreement itself reveals instability.

细微的变化可能导致截然不同的判决。 Small changes can produce very different verdicts

我们使用由人类撰写和 AI 生成的研究构想集合，测试了六种 LLM 评审模型——GPT-5.1、GPT-5.2、GPT-5.4、Claude Sonnet 4.5、Claude Opus 4.5 和 Claude Opus 4.6——通过逐项评估（即对新颖性进行绝对评分）和成对比较（即比较两个构想，找出更具新颖性的那个）来评估新颖性的能力。我们每次只改变一个因素：提示词、推理预算、评审模型是否能查看相关论文、构想的呈现格式、合并多少项评审结果以得出最终结论，以及——对于 AI 生成的构想——由哪个模型生成。Using our collections of human-authored and AI-generated research ideas, we tested how well six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – could evaluate novelty through both pointwise (i.e., absolute novelty ratings) and pairwise (i.e., comparing two ideas to find the most novel) comparisons. We varied one factor at a time: the prompt, reasoning budget, whether the judge could see related papers, how the ideas were formatted, how many judgments were combined into a final verdict, and – for AI-generated ideas – which model generated them.

性能方面最大的变化来自我们提示模型判断新颖性的方式。The largest shifts in performance came from how we prompted the model to judge novelty.

例如，我们用两个提示词向 GPT-5.4 展示了相同的 Human+Generated 配对——每组都包含一个经验证的人类创意和一个 AI 生成的创意。一个提示词告知模型，评审者已认定每组中恰有一个创意是新颖的，并要求它判断是哪一个；另一个提示词则只问哪个创意是新颖的。当我们改用更简单的提示词时，准确率从 93.5% 降至 40.9%——低于随机猜测水平，而且模型对超过一半的相同配对改变了判断。For example, we showed GPT-5.4 the same Human+Generated pairs – each containing one validated human-authored idea and one AI-generated idea – with two prompts. One prompt told the model that reviewers had found exactly one idea in each pair to be novel and asked it to identify which one; the other simply asked which idea was novel. When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.

相比之下，同样的提示词改动在 Human-Only 数据集上仅使准确率变化了约 7 个百分点。该数据集将人类撰写的想法与强弱不等的新颖性证据配对。这表明，新颖性评审器在评估人类撰写的研究时可能显得相对稳定，但在评估 AI 生成的想法时，其表现可能截然不同。By contrast, the same prompt change shifted accuracy by only about seven percentage points on the Human-Only dataset, which pairs human-authored ideas with stronger and weaker evidence of novelty. That suggests a novelty judge can appear relatively stable on human-authored research but behave very differently on AI-generated ideas.

话虽如此，人类撰写的创意也未能幸免。在 Human-Only 数据集上，我们同样观察到了显著变化；同一项变更使 Claude Opus 4.6 的准确率提高了 21 个百分点。只是与 AI 生成的创意相比，这些变化的影响没那么严重。That said, human-authored ideas weren’t immune. We saw substantial shifts on the Human-Only dataset as well; the same change moved Claude Opus 4.6 accuracy by 21 points. The shifts were just less severe than on the AI-generated ideas.

也不存在普遍更优的提示词。例如，在单独评估各个想法时，去掉一条提醒评审不要把微不足道或渐进式的贡献视为新颖之处的指令，能提高某些评审的准确率，却会降低另一些评审的准确率。换句话说，对某个模型效果良好的提示词，未必适用于其他模型。There was also no universally better prompt. For example, when judging ideas individually, dropping an instruction that warned judges not to treat trivial or incremental contributions as novel improved accuracy for some judges while reducing it for others. A prompt that works well for one model may not transfer to another, in other words.

我们还测试了新颖性评判器的一种常见用途：比较两个 AI 生成器提出的想法。我们将 Claude Opus 4.5 与 Claude Sonnet 4.5 生成的想法进行配对比较，也将 GPT-5.4 与 GPT-5.1 生成的想法进行配对比较。在测试的提示词变更中，六个评判器给出的判定平均有约四分之一到三分之一发生了变化。无需假设哪个生成器产出的想法更具新颖性，也能衡量这种分歧；这表明，评判器提示词的选择会影响对比构思系统的结果。We also tested a common use of novelty judges: comparing ideas from two AI generators. We paired ideas from Claude Opus 4.5 with Claude Sonnet 4.5, and GPT-5.4 with GPT-5.1. Across the tested prompt changes, roughly a quarter to a third of verdicts changed on average across six judges. This disagreement can be measured without assuming which generator produces more novel ideas, showing how the choice of judge prompt can affect comparisons between ideation systems.

更多搜索和推理并不一定能带来更准确的新颖性判断。我们为 GPT-5.6 Sol 提供了超高推理预算，并允许它在仅限 arXiv 的范围内进行实时迭代搜索，随后将其与标准检索评审器进行比较；后者各自只能看到预先检索出的五篇摘要。（本次比较中的所有评审器都针对 Human+Generated 中的每个创意各运行一次。）智能体式评审器的 macro-F1 得分为 0.78，比 Claude Opus 4.6 的 0.89 低 11 个百分点，与更简单的 Claude Opus 4.5 和 GPT-5.4 配置大致相当；按当前 API 价格计算，它每次决策的成本却约高出 10 倍。（在此语境下，macro-F1 衡量评审器识别新颖与非新颖创意的能力；数值越高越好。）More search and reasoning didn’t necessarily produce better novelty judgments. We gave GPT-5.6 Sol an extra-high reasoning budget and live, iterative search restricted to arXiv, then compared it with standard-retrieval judges that each saw five pre-retrieved abstracts. (All judges in this comparison ran once per idea on Human+Generated.) The agentic judge scored 0.78 macro-F1 – 11 points behind Claude Opus 4.6 at 0.89, and in roughly the same range as simpler Claude Opus 4.5 and GPT-5.4 setups – while costing roughly 10× more per decision at current API prices. (In this context, macro-F1 measures how well a judge identifies novel and non-novel ideas; higher is better.)

尽管专门构建的评估器成本高出许多，但准确度也没有更高。在 Human+Generated 数据集上，一个采用低推理强度的简单 Claude Opus 4.6 提示，通过三次调用进行多数投票，达到了 0.854 的宏平均 F1，总评估成本为 2.59 美元。相比之下，基于文献检索与重排序流程的 Idea Novelty Checker 得分为 0.642，成本为 30.66 美元；Sakana AI 的 AI Scientist 则通过迭代检索 Semantic Scholar 来评估新颖性，得分为 0.772，成本为 82.05 美元。（这两种专用评估器都使用 Claude Opus 4.6，每项决策调用一次。）Purpose-built evaluators weren’t more accurate either, despite costing substantially more. On the Human+Generated set, a simple Claude Opus 4.6 prompt using low reasoning and majority voting across three calls achieved 0.854 macro-F1 at a total evaluation cost of $2.59. By comparison, Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66, while Sakana AI’s AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. (Both dedicated evaluators used Claude Opus 4.6, with one call per decision.)

换句话说，简单的提示式基线不仅比这两个专门构建的系统更准确，而且成本也低得多。In other words, the simple prompted baseline was both more accurate and considerably cheaper than either purpose-built system.

综合来看，这些结果表明，对于系统日益需要评判的 AI 生成创意，新颖性评估尤其脆弱。Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.

迈向更可靠的新颖研究评估 Toward more reliable evaluation of novel research

我们的结果表明，基于 LLM 的新颖性评估仍是一个亟待解决的重要问题。对于人类撰写的研究似乎可靠的评审模型，在评估 AI 生成的想法时可能会有不同表现；对一个模型有帮助的提示词可能会损害另一个模型的表现；而更多的搜索或计算资源也不一定能带来更一致的判断。Our results suggest that LLM-based novelty evaluation remains a major open problem. Judges that appear reliable on human-authored research may behave differently on AI-generated ideas; prompts that help one model can hurt another; and more search or compute does not necessarily produce more consistent judgments.

未来的工作应当在评审者实际会评估的各类创意上测试他们，衡量其对不同提示和模型的敏感度，改进纳入既有文献的方式，并构建能够随着科学文献增长和 AI 系统进步而不断演进的基准。我们的基准和可复用数据流程只是一个起点——研究人员可以将其应用于新的同行评审、其他科学领域，以及未来几代新颖性评审者。Future work should test judges on the kinds of ideas they will actually evaluate, measure sensitivity across prompts and models, improve how prior literature is incorporated, and build benchmarks that can evolve as the scientific literature grows and AI systems improve. Our benchmark and reusable data pipeline are a starting point—researchers can apply them to new peer reviews, other scientific communities, and future generations of novelty judges.

自动化构思不仅需要生成看似合理的研究方向，还需要能够判断这些想法是否真正新颖的方法。通过让当前新颖性评估器的弱点更易于衡量，我们希望此次发布能帮助研究人员构建更稳健的方法，评估 AI 生成的研究想法。Automated ideation depends on more than generating plausible research directions; we also need ways to tell when those ideas are genuinely new. By making the weaknesses of current novelty judges easier to measure, we hope this release helps researchers build more robust ways to evaluate AI-generated research ideas.