Android Bench 2 新增对长时任务、代理式评估和连续评分的支持
原题Android Bench 2 Adds Support for Long-Horizon Tasks， Agentic Evaluation， and Continuous Scoring
读前速览
Google 发布了 Android Bench 2.0，这是其 AI 评估框架的一次重大更新，引入了长时任务、基于代理的评估和连续评分系统，以更好地评估复杂 Android 开发任务的性能。
长时任务的引入
Android Bench 2.0 引入了长时任务（LHT），这些任务模拟了复杂、多天的开发项目。这些任务（例如升级依赖项或从头构建应用程序）允许更现实地评估 AI 处理持续开发工作的能力，而原始版本仅测试了短期、增量的变更。
向连续评分的转变
评估方法从二进制的通过/失败系统更新为连续评分模型。这一变化允许对性能进行更细致的评估，其中识别出部分成功（例如完成 90% 的任务），而不是将其作为总失败处理。
基于代理的评估
该版本支持基于代理的评估，其中 AI 代理与开发环境交互以完成任务。这种方法特别适用于测试模型在发展过程中如何导航复杂的态空并从错误中恢复。
性能洞察
该基准的结果提供了关于 AI 能力的具体洞察。例如，数据表明 AI 模型在确定性转换（例如从 Java 转换为 Kotlin）方面表现良好，但难以处理需要深度运行时验证或涉及破坏性框架变更（例如移植跨平台应用程序）的任务。
完整摘要
Android Bench 2.0 通过引入长时任务（LHT）扩展了原始基准的能力，这些任务模拟了多天的开发项目，例如依赖项升级和完整的应用程序转换。该更新还从二进制的通过/失败系统转向了连续评分模型，该模型考虑功能、视觉保真度和避免退化。结果表明，虽然 AI 在编写新代码和执行确定性转换方面表现出色，但它仍然难以处理需要深入运行时验证或涉及破坏性框架变更的任务。
Google 发布了 Android Bench 2.0，这是其用于评估 AI 模型和智能体完成 Android 开发任务能力的基准测试框架的一次重大更新。此次更新引入了长时程任务（LHT）、基于智能体的评估和持续评分，以便更好地评估模型在复杂、多步骤开发任务中的表现。Google has released Android Bench 2.0, a major update to its benchmark framework for evaluating AI models and agents on Android development tasks. The update introduces long-horizon tasks (LHTs), agent-based evaluation, and continuous scoring to better assess performance on complex, multi-step development tasks.
Android Bench 于几个月前推出，用一系列常见开发任务评估 AI 模型，并将权限、导航和连接等方面的 Android 最佳实践纳入其中。在最新版本中，Google 扩展了该基准测试，涵盖了更广泛的开发任务。Launched a few months ago, Android Bench evaluates AI models against a set of common development tasks, incorporating Android best practices in areas such as permissions, navigation, and connectivity. With the latest release, Google has expanded the benchmark to cover a broader range of development tasks.
今天，我们发布首批长时程任务（LHT）。这类任务复杂度很高，需要工程师花费数天甚至一周才能完成。我们还将推出智能体评测，首先评测各模型提供商推出的智能体。Today we're releasing the first set of long-horizon tasks (LHT), which are tasks of great complexity that take an engineer multiple days or even a week to complete. We are also introducing agentic evaluation, starting with agents from corresponding model providers.
原始版本侧重于对现有代码仓库进行增量修改，而 Android Bench 2.0 则引入了长周期任务（LHT）。Google 将这类任务描述为工程师可能需要“数天甚至一周才能完成”的工作。任务包括升级依赖项、添加新功能、从零开始构建应用，或将跨平台应用转换为 Android 应用。While the original version focused on incremental changes to existing repositories, Android Bench 2.0 introduces long-horizon tasks (LHTs) that Google describes as work an engineer might take "multiple days or even a week to complete". These tasks include upgrading dependencies, adding new features, building apps from scratch, or converting a cross-platform app to Android.
2.0 版本的一项重大变化，是从二元的通过/失败评估转向更细致的评分体系。此前，即使智能体成功满足了几十项其他要求，一项边缘情况断言未通过，也可能导致一项复杂任务被判定为失败。One major change in version 2.0 is the shift from binary pass/fail evaluation to a more nuanced scoring system. Previously, a complex task could be marked as failed because of a single failing edge-case assertion, even if the agent had successfully met dozens of other requirements.
我们通过综合考量功能实现情况、视觉还原度以及是否避免了回归问题等因素来计算完成率。对于偏离评估说明或结构约束的情况，我们也会施加客观的评分扣分。We calculate this completion rate through a combination of factors like functionality, visual fidelity, and avoiding regressions. We also apply objective scoring penalties for deviations from evaluation instructions or structural constraints.
Android Bench 2.0 提供的结果还有助于判断哪些任务更有可能在 AI 辅助下取得成功。例如，Google 报告称，“AI 编写新代码的表现优于重构现有代码”，这可能反映出重构和迁移需要理解代码库的架构复杂性。The results provided by Android Bench 2.0 also help identify which tasks are more likely to succeed with AI assistance. For example, Google reports that "AI does a better job at writing new code rather than refactoring existing code", which can reflect the fact that refactoring and migrations require an understanding of the architectural complexity of the codebase.
同样，AI 在一些“成熟且确定性较高的转换”方面表现出色，即使是在较大的代码库中也是如此。例如，将 Java 转换为 Kotlin、用 Ktor 替换 Retrofit，或引入 ViewModel 层。Similarly, AI performs well on several "well-established, deterministic transformations", even in larger codebases. Examples include converting Java to Kotlin, swapping Retrofit for Ktor, or introducing a ViewModel layer.
相反，在不少情况下，模型仍然难以应对，包括需要运行时验证的任务（例如依赖注入图缺失）、涉及框架的重大变更，或遇到尚未发布的库所带来的知识盲区。尤其是，在将跨平台应用移植到 Android 这一任务中，顶尖模型的完成率也只有 80%；总体而言，这仍是一个“尚未解决的挑战”。On the contrary, in a number of cases model still struggle, including with tasks requiring runtime validation (like missing dependency injection graphs), involving breaking framework changes, or running into knowledge gaps with unreleased libraries. In particular, the best-in-class model achieves only 80% completion rate in porting cross-platform app to Android, which remains overall an "open challenge".
The updated Android Bench 2.0 dashboard includes recent models, such as Gemini 3.8 Flash, Gemini 3.7 Flash, OpenAI GPT-6, Anthropic Fable 5.1, Kimi K3, and Qwen 3.8 Max. At the time of the article, Claude Opus 5.5 was at the top of the leaderboard with a 32% LHT pass rate, followed by GPT 6 Astra at 28%.