OpenAI：按成功任务优化 AI 成本

Brief Description

OpenAI deployment engineers Mandeep and Sapto discuss optimizing AI applications for cost per successful task. They explain model sizing, evaluation-driven trade-offs, prompt caching, programmatic tool calling, reasoning effort, and delayed processing, drawing on customer examples.

Table of Contents

• Cost per Task, Not Cost per Token
• Right-Size the Model with Evaluations
• Four Levers Beyond Model Selection
• Improve Cache Reuse and Control
• A Practical Optimization Workflow

Cost per Task, Not Cost per Token

Host: We are joined by two members of OpenAI’s AI deployment engineering team to discuss AI cost optimization. We will cover the right metric, model sizing, and levers beyond the model itself, using examples from customers. Developers often start with the input and output token prices on our website. What do those numbers miss?

Mandeep: I lead the applied AI engineering team in San Francisco. A model that is cheap on a cost-per-token basis may not be the cheapest on a cost-per-task basis. A cheaper model might consume or produce more tokens to complete a task, or it might not complete the task at all, requiring a human to intervene. Businesses and developers should therefore look at cost per task.

A personal example illustrates this. Last week, I needed to travel to New York, but a canceled flight left me looking for an alternative that fit my schedule. I called the airline and spoke to its customer-support chatbot. I patiently explained that I wanted to reschedule without taking a red-eye flight, but the chatbot could not understand. I eventually requested a human, who booked a flight through Houston so I could arrive before my team’s offsite.

Even if the airline’s model was cheaper per token, the actual task cost included both the model’s tokens and the human’s time. That is the metric the airline should consider.

Host: Unfortunately, that is a relatable example. Token efficiency also matters, and OpenAI invests in it alongside model performance.

Mandeep: Perplexity, a customer we work closely with, reported that GPT-6 Astra produced 9% more accurate results on its research benchmark at half the cost compared with its previous model. Although Astra may have a higher token price, fewer tokens and successful completion can make it more efficient per task.

Notion likewise reported better accuracy and half the cost per task after moving to a newer model. These are examples of customers benefiting from migration and task-level efficiency rather than simply comparing list prices.

Right-Size the Model with Evaluations

Host: Model selection remains a major decision. How should developers right-size a model?

Mandeep: Begin by clearly defining the task and the accuracy the business expects. For the airline, an objective might be resolving 80% of inquiries through the chatbot. A setup using GPT-Live with a delegate model might have handled my inquiry without human escalation.

The goal is to find the model, harness, and configuration that achieve the required accuracy at the best cost. A useful concept is the Pareto curve: here, we compare accuracy and cost. Test different models, settings, configurations, and harnesses; measure their task accuracy and cost; then find the configuration that meets the accuracy threshold most efficiently.

Host: Some tasks suggest a model immediately: a smaller model such as Luna for routing, or Astra for complex planning. What if the choice is unclear?

Mandeep: Our model family spans different points on the accuracy-cost frontier. Astra offers frontier intelligence, Sol provides strong intelligence with task-cost efficiency, and Luna is another option. For simple classification or extracting a few document fields, test the smaller, more efficient model early. For tasks requiring greater intelligence, start with Astra. As you test, plot the frontier and find where the configuration meets the performance threshold at the best cost.

Host: So start with evaluation sets, find a model that meets the quality target, then optimize cost. What does “Luna maxing” mean?

Mandeep: Customers are often surprised that Luna with high or extra-high reasoning effort can outperform larger models from earlier generations or competitors. Some offer Luna as their primary coding model and switch only when more intelligence is required. Customers tell us Luna works well for 80–90% of their workflows. Do not discount it without testing it.

Host: We also announced the Decisions API for simpler tasks. But choosing a model is only one lever. What else can developers change?

Four Levers Beyond Model Selection

Sapto: The same model family can cost very different amounts depending on how much information you pass in and what output you request. Four useful levers are prompt caching, programmatic tool calling, reasoning effort, and Batch or Flex processing.

Prompt caching reuses preprocessing for repeated input. A company-support assistant might use the same policies and documents for every customer. Shared information can be reused while the model processes new information and produces a fresh response each time.

For programmatic tool calling, consider an online ordering system. An agent calls several tools to determine whether inventory can fill orders, what is missing, and what is arriving. Does the model need to read and compare every intermediate result? Often it does not.

The model can run a small program that coordinates those tool calls, combines and compares the results, and sends a packaged report back for judgment. You avoid returning every piece of intermediate data to the model and reduce the number of model calls in the workflow.

Clio, which provides AI-powered cloud management software for law firms, reported 38% fewer prompt tokens in a multi-step document-analysis workflow using programmatic tool calling, with no loss of quality. The improvement came from changing how the application interacted with the model.

Reasoning effort is another setting to test. It differs from answer length: even a short answer may require substantial reasoning. For routing, begin with lower reasoning effort and run evaluations to see whether quality holds. For difficult or differentiated tasks, additional tokens may be worth paying for. Iterative evaluations should guide the choice.

Finally, ask how long you can wait. If you are processing thousands of documents and do not need immediate responses, OpenAI’s Batch API offers a 24-hour processing window at 50% lower token pricing than standard synchronous requests. Batch and Flex are options for work that tolerates delay.

The model-family question should be accompanied by another question: how much work will you make the model do? Trace a complete task, examine its input and output, look for repeated workflows, and count tool calls. Change one setting, measure the effect, and iterate as the application and models evolve. The objective is a successful outcome at lower cost and acceptable latency, rather than simply fewer tokens.

Host: To recap, caching can substantially reduce input cost, with cached-input pricing showing savings of up to 90%. Programmatic tools keep unnecessary intermediate processing out of the context window. Reasoning effort can be tuned, and delayed processing costs less when urgency allows it. The API now also allows reasoning-effort changes without breaking the cache.

Improve Cache Reuse and Control

Host: Prompt caching is especially important. What common mistakes limit cache hit rates?

Sapto: Prompt caching is enabled by default, but many customers do not use it fully. Our main recommendation is to keep the beginning of the prompt consistent whenever possible. The shared instructions can reuse preprocessing, while information outside the matching portion is processed freshly. Combined with newer model families, this can reduce both cost and latency.

Blitzy, an AI-powered software-development company, reported that switching from a single structured-output call to Luna’s tool-calling loop increased cache reuse from 24% to 90%. Across thousands of production calls, it also reported 8.5 times fewer output tokens and 87% lower cost than GPT-5.4 Mini, while handling 2.2 times more context.

Host: What if prompts change frequently across queries?

Sapto: Earlier model families used automatic caching with system-selected boundaries based on token intervals. You could keep a stable prefix, but could not choose your own boundaries.

The GPT-5.6 family adds explicit breakpoints. Think of them as instructions to cache up to a particular point. For example, long-running coding instructions can remain stable while tasks change at the end. Put a breakpoint after the stable instructions and before the changing task, so the stable portion is reused while the task is processed freshly.

Explicit-only mode gives more control over which portions you cache. It is useful when you want to avoid caching frequently changing content that will not be reused.

GPT-6 Astra retains those controls and adds the ability to change reasoning effort during an ongoing conversation without breaking the cache.

Host: Tools can also change without breaking the cache through allowed-tools controls and other parameters. These features provide flexibility while preserving cache reuse.

We have also shipped a prompt-caching dashboard on the admin website to measure and track application cache hit rates. A diagnostics API compares two queries and identifies where the cache miss happens, helping developers find what to optimize.

A Practical Optimization Workflow

Host: We have covered successful-task cost, model selection, caching, programmatic tool calling, reasoning effort, and Batch or Flex processing. What should developers do next?

Mandeep: First, define the task and the accuracy threshold the business expects. Second, create a representative set of tasks showing how the application will be used. Try different models, settings, reasoning levels, prompts, and caching arrangements. Plot the Pareto frontier and select the model, configuration, and harness that best meet the required accuracy at the best cost.

Sapto: Test one change at a time. When it improves the result, retain that setting and continue evaluating as your application and model evolve.