我将 4 款模型的推理档位调至「高」：只解决了一个问题，却为所有消耗买了单

这是提交给 Kaggle 基准测试挑战赛的作品。This is a submission for the Kaggle Benchmarking Challenge

我给 gpt-5.4-mini 出了一道逻辑谜题：七个人、七天、十条线索，“谁在星期五做演讲？”I gave gpt-5.4-mini a logic puzzle: seven people, seven days, ten clues, "Who gives the talk on Friday?"

将推理力度设为“无”后，它回复道：With reasoning effort set to none, it replied:

Cleo

FINAL ANSWER: Cleo

18 个输出 token。$0.00024。错了。（答案是 Fay。）第二次重复时，它逐字给出了同一个错误答案。18 output tokens. $0.00024. Wrong. (The answer is Fay.) It gave the same wrong answer, word for word, on the second repeat.

在 high 模式下，它消耗了 1,333 个 token，花费约 0.006 美元，并回答了 Fay。看起来这似乎支持始终选择 high。At high it spent 1,333 tokens, cost about $0.006, and said Fay. That looks like an argument for always choosing high.

在另一个条目中，我也让同一个模型以 high 模式统计“You have a chisel and a drill.”里的工具数量。它回答说是 10,016 个。On another item I asked the same model, also at high, to count the tools in "You have a chisel and a drill." It answered 10,016.

两条回复来自同一个模型，使用相同的设置，并针对同一项基准测试。这篇帖子就用这两个例子说明了一件事：推理档位有时影响很大，但大多数时候只会让账单更贵。Both replies came from the same model, setting and benchmark. That is the whole post in two examples: the reasoning dial sometimes matters a lot. Most of the time it only makes the bill bigger.

简而言之：Reasoning Dial 是一个 Kaggle 基准测试，只改变一个 API 参数 reasoning_effort（none / low / medium / high），其他条件全部固定：相同的 60 道代码生成题目、相同的提示词，以及相同的确定性评分器。对 4 个模型进行了 1,800 多次评分调用后发现，high 只带来了一项具有统计意义的准确率提升：gpt-5.4-mini 在逻辑谜题上的准确率从 15% 提升至 97.5%。在 12 个“模型 × 任务”组合中的 7 个里，准确率没有变化，但每答对一道题的成本却增加了 1.5 至 3.4 倍。这个拨盘也并非统一的调节器：一个模型在 high 下使用的 token 数是原来的 14 倍，一个模型因不接受 none 而返回 HTTP 400 错误，另一个模型则在 none 下照样进行推理。假设和分析方案已在主实验开始前预注册。主实验总成本：4.20 美元。

TL;DR: Reasoning Dial is a Kaggle benchmark that changes one API parameter, reasoning_effort (none / low / medium / high), and keeps everything else fixed: the same 60 code-generated questions, the same prompt and the same deterministic grader. Over 1,800 graded calls on 4 models, high produced exactly one statistically real accuracy gain: gpt-5.4-mini on logic puzzles, 15% → 97.5%. In 7 of 12 model × task cells, accuracy did not change while cost per correct answer rose ×1.5 to ×3.4. The dial also isn't one instrument: one model spends ×14 more tokens at high, one rejects none with an HTTP 400, and one thinks at none anyway. Hypotheses and analysis were pre-registered before the main run. Total main-run cost: $4.20.

我的基准测试 My Benchmark

• 🏆 Kaggle benchmark (leaderboard): https://www.kaggle.com/benchmarks/abeeralodhi/reasoning-dial
• 🎛️ The four public tasks, one per dial position: dial-none · dial-low · dial-medium · dial-high
• 💻 Code, raw run files and pre-registration: https://github.com/Abeera81/reasoning-dial

我的基准测试项目 What I Benchmarked

几乎每个现代模型 API 都有类似的参数：reasoning_effort、thinking、reasoning。开发者每天都会设置它，通常凭感觉决定：Almost every modern model API has a version of this parameter: reasoning_effort, thinking, reasoning. Developers set it every day, usually by feel:

• 这是个难题，所以很高。
  "It's a hard question, so high."
• 它处于生产环境中，所以较低。
  "It's in production, so low."

几乎没人衡量移动它究竟会带来什么变化。Almost nobody measures what moving it actually changes.

于是我构建了一个基准测试，其中唯一的变量就是这个调节参数。相同的条目、相同的提示后缀、相同的解析器、相同的评分器。对于每个模型，四个 Kaggle 任务的差异仅在于 LEVELS 的值和两行任务名称。（push CLI 将任务名称读取为字符串字面量，因此无法做到只差一行。）So I built a benchmark where the only variable is the dial. Same items, same prompt suffix, same parser, same grader. For each model, the four Kaggle tasks differ only in the LEVELS value and the two task-name lines. (The push CLI reads task names as string literals, so one line of difference wasn't possible.)

我在每种设置下测量了三项内容：I measured three things at each setting:

1. 准确性。严格且确定性：正则表达式会读取最后一行 FINAL ANSWER:。不存在任何 LLM 评判器。
   Accuracy. Strict and deterministic: a regex reads the last FINAL ANSWER: line. There is no LLM judge anywhere.
2. 输出词元。推理词元也包括在内，因为无论你能否看到它们，你都需要为它们付费。
   Output tokens. Reasoning tokens are included, because you pay for them whether or not you can see them.
3. 每个正确答案的成本。Kaggle 模型代理报告的每次调用纳美元成本除以正确答案的数量。
   Cost per correct answer. The per-call nanodollar cost that Kaggle's model proxy reports, divided by the number of correct answers.

三种旨在彼此相互矛盾的问题 Three kinds of question, chosen to disagree with each other

以下是锁定测试集中的一个真实 Distract 项目，与模型看到的内容完全一致：Here is a real Distract item from the locked test set, exactly as the models saw it:

You have a chisel and a drill.

A shop in Easton sold 8,377 levels last week.

A friend shows you this code:
```python
stock = ["saw", "hammer", "screwdriver"]
print(len(stock) * 3)
```

A classmate is working on a homework problem: if 44 crates each hold 37 saws, how many are there altogether?

Question: Calculate how many tools you have.

Answer instruction: Answer with a whole number.

End your response with a final line in exactly this format:
FINAL ANSWER: <answer>

有人没读过第一行就回答了 2。这正是这道题有用的原因。A person answers 2 without reading past the first line. That is exactly why the item is useful.

我在运行任何内容之前记下了自己的猜测。 I wrote down my guesses before running anything

正式运行前，我提交了 docs/PREREGISTRATION.md。文件中包含四项假设、冻结的提示词和解析器，以及统计方法：配对 Wilcoxon 检验、对全部 12 项“模型 × 系列”检验进行 Holm 校正，以及进行 5,000 次重复的项目簇自助法。在同一次提交中，我还用 SHA-256 哈希锁定了 60 个测试项目（90dd5498…0042）。如果哈希不匹配，分析脚本会拒绝运行。Before the main run, I committed docs/PREREGISTRATION.md. It contains four hypotheses, the frozen prompt and parser, and the statistics: a paired Wilcoxon test, Holm correction across all 12 model × family tests, and a 5,000-rep item-cluster bootstrap. In the same commit I locked the 60 test items under a SHA-256 hash (90dd5498…0042). The analysis script refuses to run if the hash doesn't match.

锁定后所做的每项更改都会记录在该文件中，注明日期并附上说明。共有三项更改，但都没有涉及条目、提示词、解析器或统计数据。Every change made after the lock is logged, dated and explained in that file. There are three, and none of them touches the items, the prompt, the parser or the statistics.

我如何在 Kaggle Benchmarks 上构建它 How I built it on Kaggle Benchmarks

这篇文章中的所有内容都来自这条流水线。生成器使用固定的种子，测试集通过哈希锁定，评分器则是一个正则表达式。四个 Kaggle 任务之间唯一的变化是推理力度值。Everything in this post comes out of this pipeline. The generator uses a fixed seed, the test set is locked by hash, and the grader is a regex. The only thing that changes between the four Kaggle tasks is the reasoning-effort value.

整个实验之所以可行，得益于 kaggle-benchmarks 库的一项功能：llm.prompt() 接受 reasoning= 参数，而 Kaggle 的模型代理会将其转换为各供应商各自的 reasoning_effort。一行代码即可在 OpenAI、Anthropic、Google 和开放权重模型上运行相同的实验。以下是每次调用的核心代码：The whole experiment is possible because of one feature of the kaggle-benchmarks library: llm.prompt() accepts a reasoning= argument, and Kaggle's model proxy translates it into each vendor's own reasoning_effort. One line of code runs the same experiment on OpenAI, Anthropic, Google and open-weight models. Here is the core of every call:

# src/dial/task_template.py.txt  (inlined into each Kaggle task by build_tasks.py)
with kbench.chats.new(name) as chat:
    try:
        raw = llm.prompt(prompt, reasoning=level, seed=SEED_BASE + repeat)
        traces = kbench.last_reasoning_traces()
    except Exception as e:
        raw, traces = None, None
        rec["error"] = f"{type(e).__name__}: {e}"   # a provider 400 becomes an "api_error", which is data for H1
    u = chat.usage
rec["output_tokens"] = u.output_tokens               # hidden reasoning tokens included
rec["cost_nanodollars"] = u.total_cost_nanodollars   # what Kaggle's proxy actually charged

chat.usage 是这项基准测试能够衡量成本而不是靠猜测的原因。每次调用都会记录自己的令牌数和以纳美元为单位的成本，这些记录最终会出现在 Kaggle 的运行文件中。chat.usage is the reason this benchmark can measure cost instead of guessing it. Every call records its own token count and its own cost in nanodollars, and those records end up in Kaggle's run files.

每个 dial-<level> 任务都会以嵌套任务的形式扇出全部 120 次调用（60 个项目 × 重复 2 次）：Each dial-<level> task fans out all 120 calls (60 items × 2 repeats) as a nested task:

records = _records(dial_call.evaluate(
    llm=[llm], evaluation_data=df, n_jobs=8, on_failure="continue"))

on_failure="continue" 很重要。当 gpt-oss-120b 没有拒绝任何内容时，运行不会崩溃：它会记录这次拒绝，而这次拒绝会成为一项发现。on_failure="continue" matters. When gpt-oss-120b rejects none, the run doesn't crash: it records the rejection, and the rejection becomes one of the findings.

评分过程刻意设计得很简单。提示词以“请在回复末尾严格按照以下格式另起一行：FINAL ANSWER: ”结束，解析器会取最后一个匹配项：Grading is deliberately boring. The prompt ends with "End your response with a final line in exactly this format: FINAL ANSWER: ", and the parser takes the last match:

# src/dial/parse.py
_MARKER = re.compile(
    r"FINAL[ \t]+ANSWER[ \t]*[*_`]*[ \t]*[:：][ \t]*(.*)$",
    re.IGNORECASE | re.MULTILINE,
)

它可以处理 **FINAL ANSWER:** 42 和全角冒号，剥除 <think> 块，并且已纳入代码库的测试套件。我没有使用该库的结构化输出（schema=），因为启用推理时它会出问题：代理会返回 <think>…</think>{json}，导致 JSON 解析失败。纯文本加上我自己的解析器，是唯一能在所有四家供应商上以相同方式工作的方案。It handles **FINAL ANSWER:** 42 and full-width colons, strips <think> blocks, and is covered by the repo's test suite. I didn't use the library's structured output (schema=) because it breaks when reasoning is on: the proxy returns <think>…</think>{json} and JSON parsing fails. Plain text plus my own parser was the only approach that worked the same way across all four vendors.

Kaggle CLI 处理了循环中的其余部分：The Kaggle CLI handled the rest of the loop:

kaggle b t push dial-high -f tasks/dial_high.py --wait        # also runs it once on Kaggle's default model
kaggle b t run dial-high -m gpt-5.4-mini-2026-03-17           # one model, one dial position
kaggle b t download dial-high -o results/main                 # per-call records → analysis/analyze.py

第一个命令产生了我没预料到的副作用。推送任务会让它在 Kaggle 的默认模型 gemini-3.7-flash 上运行一次。那些验证运行完整且顺利地跑完了所有锁定项目，因此 Gemini 就这样加入了阵容，而我并没有选择它。The first command had a side effect I didn't plan for. Pushing a task runs it once on Kaggle's default model, gemini-3.7-flash. Those validation runs were full, clean runs of the locked items, so Gemini joined the lineup without me choosing it.

Kaggle 的配额教会我一些东西的那部分 The part where Kaggle's quota taught me something

Kaggle 会在每次调用运行前，按其可能产生的最高费用从你的额度中预留成本：输入费用加上按模型输出价格计算的 128,000 个输出 token 费用。对于 gpt-5.4-mini，每次调用预留的费用约为 0.58 美元，即使这次调用的实际费用只有 0.0002 美元。Kaggle reserves each call's maximum possible cost against your quota before the call runs: the input cost plus 128,000 output tokens at the model's output price. For gpt-5.4-mini that is about $0.58 reserved per call, even when the call actually costs $0.0002.

并行发起 8 个调用时，一个任务就会预留约 4.61 美元。我第一次运行 gpt-5.4-mini 时，同时启动了多个任务，480 次调用中有 378 次因配额限制返回 HTTP 403 拒绝响应。实际支出远未达到上限。With 8 calls in parallel, one task reserves about $4.61. My first gpt-5.4-mini run launched several tasks at once, and 378 of 480 calls came back as HTTP 403 quota refusals. Real spend was nowhere near the cap.

我制定了一条规则，并将其写入预注册：配额导致的 403 属于数据缺失，绝不算作模型行为。我丢弃了这些运行结果（它们列在 results/superseded.json 中，分析时也已排除），并在配额重置后一次只运行一个任务，重新测试 gpt-5.4-mini。重跑过程顺利：拒绝次数为 0，API 错误数为 0。I made a rule and wrote it into the pre-registration: a quota 403 is missing data, never model behavior. I discarded those runs (they're listed in results/superseded.json, and the analysis excludes them) and re-ran gpt-5.4-mini one task at a time after the quota reset. The re-run was clean: 0 refusals, 0 API errors.

• gpt-5.4-mini：无级别时为 0.044 美元，低级别时为 0.371 美元，中级别时为 0.588 美元，高级别时为 0.713 美元，总计 1.72 美元。
  gpt-5.4-mini: $0.044 at none, $0.371 at low, $0.588 at medium and $0.713 at high, $1.72 in total.
• Claude Sonnet 5：已从候选列表中移除。每次调用的额度预留约为 1.28 美元，8 次并行调用（≈10.24 美元）仅此一项就会超过 10 美元的每日配额。分析报告将其标记为“已移除（平台配额预留）”，而不是悄无声息地将其排除。
  Claude Sonnet 5: dropped from the lineup. Its reservation is about $1.28 per call, and 8 parallel calls (≈$10.24) exceed the $10 daily quota on their own. The analysis reports it as "dropped (platform quota reservation)" rather than leaving it out silently.

已测试的模型 Models Tested

总计 1,800 次评分调用：60 个项目 × 重复 2 次 × 15 个模型级别。最终数据中有 0 次 API 错误和 0 次配额拒绝，输出令牌数为 1,964,378。主运行成本为 4.20 美元；整个项目（包括试点和两轮校准）共消耗约 6.38 美元的 Kaggle 配额。1,800 graded calls in total: 60 items × 2 repeats × 15 model-levels. There were 0 API errors and 0 quota refusals in the final data, and 1,964,378 output tokens. The main run cost $4.20; the whole project, including the pilot and two calibration rounds, cost about $6.38 of Kaggle quota.

关于控制的一点说明：Kaggle 的代理会悄然丢弃 temperature 参数，因此我不能声称 temperature 为 0；Google 模型也会丢弃 seed 参数。这就是为什么每个单元格都重复两次，以及为什么分析中报告了噪声底（summary.md 第 7 节）。A note on control: Kaggle's proxy silently drops temperature, so I can't claim temperature 0, and the Google models also drop the seed. That is why every cell has two repeats, and why the analysis reports a noise floor (§7 of summary.md).

研究结果 Findings

1. 表盘并非一种仪器（H1：✅ 支持） 1. The dial is not one instrument (H1: ✅ supported)

每次调用的输出令牌数中位数（对数刻度）。同一参数名称会产生四种不同的行为。Median output tokens per call (log scale). The same parameter name produces four different behaviors.

同一个参数对每个供应商来说含义都不同：The same parameter means something different for every vendor:

其中有两点让我感到意外：Two of these surprised me:

• 在 Gemini 上，none 并不意味着“不思考”。设为 none 时，它输出 token 数的中位数为 453，比 gpt-5.4-mini 设为 high 时消耗的还多（254）。在 none 或 low 时，它不会返回推理轨迹，而它的一行回复看起来和完全没有思考的模型没什么区别。你为这种思考付了费，却看不到它。
  none doesn't mean "no thinking" on Gemini. At none it spends a median 453 output tokens, more than gpt-5.4-mini spends at high (254). It returns no reasoning trace at none or low, and its one-line replies look the same as a model that didn't think at all. You pay for that thinking but can't see it.
• gpt-oss-120b 不支持 none，而是会返回 HTTP 400 错误。如果你在多供应商配置中设置 reasoning_effort="none"，这四个模型中就有一个会报错。
  none doesn't exist on gpt-oss-120b. It returns HTTP 400 instead. If you write reasoning_effort="none" into a multi-vendor config, one of these four models will error.

按类别拆分来看，调节幅度的影响很大程度上取决于问题类型。gpt-5.4-mini 在 Deduce 上的 token 数从 18 → 1,579 → 2,856 → 3,040（×169）。在 Arith 上则只从 128 增至 220。对于答案就在第一句话中的问题，gpt-oss-120b 在高档位处理 Distract 时消耗的 token 是低档位的 ×20.6。Split by family, the effect of the dial depends heavily on the question. gpt-5.4-mini's Deduce tokens go 18 → 1,579 → 2,856 → 3,040 (×169). On Arith they only go 128 → 220. gpt-oss-120b spends ×20.6 more tokens on Distract at high than at low, on questions whose answer is in the first sentence.

2. 一项真正的准确率提升，而且来自第一步（H3：✅ 仅适用于 gpt-5.4-mini） 2. One real accuracy gain, and it came from the first step (H3: ✅ for gpt-5.4-mini only)

各层级的准确率及其 95% 项目簇自助法置信区间。逻辑面板中的虚线表示在 7 个名称中随机猜测的准确率。在 none 级别，gpt-5.4-mini 的正确率为 15%，与随机猜测的准确率大致相当。Accuracy at each level with 95% item-cluster bootstrap CIs. The dashed line in the logic panel is random guessing among 7 names. At none, gpt-5.4-mini is right 15% of the time, which is about the random-guess rate.

从无到高的变化为 +82.5 个百分点（95% 置信区间：+70 至 +95，Holm 校正后的 p = 0.00056）。这是本研究中唯一同时满足两项预注册标准的效应。The change from none to high is +82.5 percentage points (95% CI +70 to +95, Holm-adjusted p = 0.00056). It is the only effect in the study that passes both pre-registered criteria.

大部分提升来自第一步：从“无”到“低”增加了 60 个点。之后每提升一步，准确率的增幅都会变小，代价却更高。Most of the gain comes from the first step: none → low adds 60 points. Each further step adds less accuracy and costs more.

全部 12 项预注册检验中，只有一项达到显著性。六项检验无法提升，因为模型在最低设置下已经得分 100%。gpt-oss-120b 在逻辑方面增加的 17.5 个百分点看似有所提升，但经 12 项检验的多重比较校正后，这一结果未能达到显著性（Holm p = 0.597）。All 12 pre-registered tests. One is significant. Six can't move because the model already scores 100% at its lowest setting. gpt-oss-120b's +17.5 points on logic look like a gain but don't survive the correction for 12 tests (Holm p = 0.597).

其他模型的结果就没那么令人兴奋了，但对旋钮调节的问题很重要：Claude Haiku 5.5 在“无”档位下解这些谜题的得分为 97.5%，Gemini 则为 100%。对它们来说，调高 Deduce 的档位已经没有什么可改进的了。The other models have a less exciting result that matters for the dial question: Claude Haiku 5.5 scored 97.5% on these puzzles at none, and Gemini scored 100%. For them, turning the dial up on Deduce had nothing left to fix.

3. 更多思考并没有造成明显影响，但你可以看到它出了问题（H2：❌ 不支持） 3. More thinking didn't significantly hurt, but you can see it going wrong (H2: ❌ not supported)

我原以为，带干扰项的计数会是努力越多反而越糟的情况（H2）。预注册检验结果并非如此：没有任何模型在高努力水平下显著变差。I expected counting with distractors to be where more effort backfires (H2). The pre-registered test says no: no model got significantly worse at high.

• gpt-oss-120b 从 75%（低）降至 67.5%（高）：下降 7.5 个百分点，置信区间为 −22.5 至 +7.5。这一结果不显著。
  gpt-oss-120b went from 75% (low) to 67.5% (high): −7.5 points, CI −22.5 to +7.5. That isn't significant.
• gpt-5.4-mini 从 75% 提升到了 85%，同样不显著。
  gpt-5.4-mini went from 75% to 85%, also not significant.
• Haiku 和 Gemini 在每个级别都得了 100%。
  Haiku and Gemini scored 100% at every level.

H2 不受支持，我会如实报告这一点。H2 is not supported, and I'm reporting it that way.

这些错误依然值得一读。所有模型在所有层级给出的每个错误 Distract 答案，都是多算了。没有任何模型算得偏少。模型并非搞不清自己拥有多少；它们只是把干扰项的数字也加进了计数。The errors were still worth reading. Every wrong Distract answer, from every model, at every level, was an over-count. No model ever answered too low. The models weren't confused about what they owned; they added the distractor numbers to the count.

主运行在 high、decomposed 配置下的三条回复。article_visuals.py 在绘图前会根据原始运行文件和锁定项校验每一个总和，因此这些数字都不是手动输入的。前两条回复中，模型把提示里的所有数字都加了起来。第三条则是替别人做了作业，并把结果当作答案报告出来。Three replies from the main run at high, decomposed. article_visuals.py asserts every sum against the raw run files and the locked items before it draws, so none of these numbers are typed by hand. In the first two replies the model adds every number in the prompt. In the third it solves someone else's homework and reports that as the answer.

较长的回答也往往是错误的。在 high 设置下，gpt-5.4-mini 的错误 Distract 回答输出 token 数中位数为 412；正确回答则为 91.5。gpt-oss-120b 的这一模式更加明显：错误回答用了 5,776 个 token，而正确回答用了 747 个。最极端的情况下，gpt-oss-120b 花了 21,875 个 token 和近四分钟（235 秒），回答 6，而正确答案是 5。The longer replies were also the wrong ones. At high, gpt-5.4-mini's wrong Distract answers used a median of 412 output tokens; its correct answers used 91.5. gpt-oss-120b showed the same pattern more strongly: 5,776 tokens for wrong answers against 747 for correct ones. In the most extreme case, gpt-oss-120b spent 21,875 tokens and almost four minutes (235 s) to answer 6 when the answer was 5.

随着投入程度增加，gpt-oss-120b 的表现也变得不那么稳定。在 Distract 的高投入档位下，它对 45% 的项目在两次重复测试中给出了不同答案，而低投入档位下这一比例为 0%。gpt-oss-120b also became less consistent as effort went up. On Distract at high, it gave different answers on the two repeats for 45% of items, compared with 0% at low.

因此，H2 未能通过显著性检验，我不会声称它通过了。但“投入更多努力反而让结果变差”也不是恰当的总结。更准确地说，投入更多努力让它有了更多过度计数的空间。So H2 fails as a significance test, and I won't claim it passed. But "more effort made it worse" isn't the right summary either. A better one is more effort gave it more room to over-count.

4. 该法案（H4：✅ 支持，描述性） 4. The bill (H4: ✅ supported, descriptively)

把档位调高换来了什么。十二个单元格中有七个落在零线上：成本变为原来的 ×1.5 到 ×3.4，准确率却没有变化。只有一个数据点明显高于零线。What turning the dial to high bought. Seven of twelve cells sit on the zero line: same accuracy at ×1.5 to ×3.4 the cost per correct answer. Only one point is high above the line.

这就是我会展示给任何在生产环境中设置 reasoning_effort 的人的图表：This is the chart I'd show anyone who sets reasoning_effort in production:

• 在 12 个单元格中的 7 个里，调至高档除了价格上涨外没有带来任何变化。准确率保持不变，但每个正确答案的成本涨至原来的 1.5 至 3.4 倍。
  In 7 of 12 cells, high changed nothing but the price. Accuracy stayed the same while cost per correct answer rose ×1.5 to ×3.4.
• 在 Arith 的对照组中，所有模型的准确率都保持不变，但每答对一题的成本却上升了 ×1.65（gpt-5.4-mini）至 ×2.5（gpt-oss-120b）。在已经解决的问题上投入更多精力毫无收益。
  On Arith, the control family, every model stayed at its accuracy while cost per correct rose ×1.65 (gpt-5.4-mini) to ×2.5 (gpt-oss-120b). Spending more effort on an already-solved problem buys nothing.
• 在 Distract 上，gpt-oss-120b 每个正确答案的成本约为 25 倍（$0.00006 → $0.00151），得分低 7.5 分（差异不显著）。
  gpt-oss-120b on Distract paid about ×25 per correct answer ($0.00006 → $0.00151) and scored 7.5 points lower (not significant).
• 唯一一次重大胜利也代价高昂。gpt-5.4-mini 的逻辑修复使每个正确答案的成本提升至原来的 8.5 倍。
  The one big win was expensive too. gpt-5.4-mini's logic fix raised cost per correct answer ×8.5.

每个正确答案的成本（美元，对数刻度）。每条线都从左向右上升，没有一条线下降。Cost per correct answer (USD, log scale). Every line rises from left to right. None of them falls.

一个反直觉的细节：在 Deduce 上，gpt-5.4-mini 设为 none 时，每个正确答案的成本最低（0.0019 美元，而 high 时为 0.016 美元），尽管它有 85% 的时候会答错。只有在你能分辨答案正误时，每个正确答案的成本才有参考意义。在大多数实际用途中，你无法分辨，所以别把那一行理解为建议你选择 none。One counter-intuitive detail: on Deduce, gpt-5.4-mini at none is the cheapest per correct answer ($0.0019, versus $0.016 at high), even though it's wrong 85% of the time. Cost per correct answer only tells you something when you can tell which answers are right. In most real uses you can't, so don't read that row as advice to choose none.

记分牌 Scoreboard

我会对选择推理力度值的人说些什么 What I'd tell someone choosing a reasoning-effort value

这些数据来自每个家族的 20 个项目、2 次重复和 4 个模型，因此应将其视为起点，而不是规则：These come from 20 items per family, 2 repeats and 4 models, so treat them as starting points rather than rules:

1. 先测量“无”这一项。四个模型中有两个（Haiku 和 Gemini）在最低设置下的每个类别上都已达到或接近 100%。
   Measure none first. Two of the four models (Haiku and Gemini) were already at or near 100% on every family at their lowest setting.
2. 如果“无”不行，先尝试“低”，再尝试“高”。对于那个需要调节旋钮的模型，从“无”调到“低”后，得到了 82.5 分中的 60 分。
   If none fails, try low before high. For the one model that needed the dial, none → low delivered 60 of the 82.5 points.
3. 不要假设不同供应商对同一个级别的处理方式相同。none 在一家会报错，在另一家会隐藏思考过程，在第三家则会真正关闭该功能。
   Don't assume a level does the same thing across vendors. none errors on one, is invisible thinking on another, and is a real off switch on a third.
4. 注意简单问题却得到冗长回答的情况。在 Distract 中，冗长的回复都是错误答案。
   Watch for long answers to simple questions. On Distract, long replies were the wrong ones.

惊喜 Surprises

• 最需要调节推理强度的模型，恰恰是在“none”档位下真的什么都没有的那个。gpt-5.4-mini 在“none”档位下用 18 个 token 回答逻辑谜题，正确率和随机猜测差不多。
  The model that needed the dial most was the one where none really meant none. gpt-5.4-mini answered logic puzzles in 18 tokens at none and was right about as often as a random guess.
• Gemini 的 none 并非免费。它在 none 设置下消耗的隐藏 token 中位数为 453 个，超过了 gpt-5.4-mini 在 high 设置下的消耗量。
  Gemini's none isn't free. It spent a median of 453 hidden tokens at none, which is more than gpt-5.4-mini spent at high.
• 所有错误都偏向同一个方向。每个答错的 Distract 答案都偏高，没有一个偏低。
  The errors all went in one direction. Every wrong Distract answer was too high, and none was too low.
• 推送一个任务后，我的阵容中就多了一个模型。Gemini 是通过 Kaggle 默认模型的验证运行进入这项研究的。
  Pushing a task added a model to my lineup. Kaggle's default-model validation runs are how Gemini got into the study.
• 配额系统差点制造出一个假发现。378 次配额拒绝可能会被解读为“gpt-5.4-mini 在高 effort 下失效”。预注册规则（403 = 缺失数据）将这些情况排除在外。
  The quota system nearly faked a finding. 378 quota refusals could have been read as "gpt-5.4-mini fails at high effort". The pre-registered rule (403 = missing data) kept them out.

明确说说局限性 Limitations, plainly

• 样本量较小。每个系列有 20 个项目，每个单元重复 2 次。大多数“无效果”的结果意味着在这样的样本量下无法检测到效果，而非证明效果为零。
  Small samples. 20 items per family and 2 repeats per cell. Most "no effect" results mean not detectable at this size, not proven zero.
• 上限。Haiku 和 Gemini 几乎在所有方面都达到或接近 100%，所以我可以说调节这个旋钮会让它们付出什么代价，却无法说它能带来什么收益。要评估这一点，需要更难的题目。
  Ceilings. Haiku and Gemini are at or near 100% almost everywhere, so I can say what the dial costs them but not what it buys. Harder items would be needed for that.
• 合成任务和一种提示格式。这些问题旨在隔离各种影响，而非反映真实工作负载。
  Synthetic tasks and one prompt format. These questions were built to isolate effects, not to represent real workloads.
• 无法控制温度。Kaggle 的代理会丢弃温度参数，Google 的模型也会丢弃 seed。重复次数和翻转率就是我的替代方案。
  No temperature control. Kaggle's proxy drops temperature, and the Google models also drop seed. The repeats and the flip rates are my substitute.
• 成本是 Kaggle 的估算成本。您的服务提供商的定价可能有所不同。
  Cost is Kaggle's proxy cost. Your provider's pricing may differ.
• 阵容不完整。Claude Sonnet 5 已被移除（配额预留），而 gpt-oss-120b 没有“none”级别。它的比较从“low”开始。
  Incomplete lineup. Claude Sonnet 5 was dropped (quota reservation), and gpt-oss-120b has no none level. Its comparisons start from low.
• 推理过程的部分视图。只有 Gemini 返回了推理轨迹（中等和高等级）。对于其他模型，输出 token 是思考过程的唯一证据。
  Partial view of reasoning. Only Gemini returned reasoning traces (at medium and high). For the rest, output tokens are the only evidence of thinking.

我接下来会测量的内容 What I'd measure next

• 更难的 Deduce 题目（更多人物、更多线索），看看能否让 Haiku 和 Gemini 摆脱满分天花板，并检验调整档位是否能带来提升。
  Harder Deduce items (more people, more clues) to get Haiku and Gemini off the ceiling and see whether their dial buys anything.
• 更多干扰项和重复项。gpt-oss-120b 得分低 7.5 分，翻转率为 45%，这与 H2 的预测方向一致，但样本量太小，无法对其进行充分检验。
  More Distract items and repeats. gpt-oss-120b's −7.5 points and 45% flip rate are the direction H2 predicted, but the sample is too small to test them properly.
• 延迟——每次调用都会记录这项数据（latency_s、backend_latency_ms），但这篇帖子并未对此进行分析。
  Latency, which every call already records (latency_s, backend_latency_ms) and this post doesn't analyze.
• Sonnet 和其他模型一次运行一个任务，以适应配额预留。
  Sonnet and other models, run one task at a time to fit the quota reservation.

复现它 Reproduce it

本文中所有得出数值的内容均为公开的：生成器、解析器、任务构建器、分析脚本、预注册文件，以及每一份 Kaggle 原始运行文件。Everything that produced a number in this post is public: the generators, parser, task builder, analysis script, pre-registration and every raw Kaggle run file.

git clone https://github.com/Abeera81/reasoning-dial && cd reasoning-dial
pip install -r requirements.txt
python -m pytest -q                 # 68 tests: generators, parser, task builder, analysis
python analysis/analyze.py          # rebuilds results/summary.md and every figure from results/main/

analyze.py 会在进行任何计算之前，检查锁定测试集的 SHA-256。要添加模型，请复制四个公开 Kaggle 任务中的任意一个并运行。由于每个任务只控制拨盘等级，因此比较保持公平。analyze.py checks the locked test set's SHA-256 before it computes anything. To add a model, fork any of the four public Kaggle tasks and run it. Because each task holds only the dial level, the comparison stays fair.

这个调节旋钮确实能发挥作用。对于某种问题上的某个模型，它让结果从猜测变成了解题。至于我测过的其他所有情况，它大多只是抬高了价格。The dial is a real control. For one model on one kind of problem, it made the difference between guessing and solving. For everything else I measured, it mostly raised the price.

调高之前，先看看设为 none 会得到什么。如果你用自己的模型跑这些任务，我想看看它们生成的阶梯。🎛️Before you turn it up, check what none gets you. If you run your own models through the tasks, I'd like to see the ladders they produce. 🎛️