AI Agent Cost Per Resolved Task: Why Rankings Reshuffle

The published leaderboards engineers cite most — Aider Polyglot, SWE-bench Verified, Terminal-Bench, and Princeton’s HAL — now carry both a capability number and a dollar figure. Read them together and a single normalization step, AI agent cost per resolved task, reorders the field: the model you assume is cheapest is often not the one that clears the most tasks per dollar. This is a methodology analysis, not a tutorial or a tool review.

How This Was Researched

This analysis is based on official documentation, published leaderboards, and vendor pricing pages — we did not run any of these agents ourselves. Sources were published leaderboards (Aider Polyglot, SWE-bench Verified, Terminal-Bench, HAL, tau-bench, and the Artificial Analysis methodology page) plus vendor pricing pages from OpenAI, Anthropic, and Google Cloud Vertex AI. The method was to compile published per-task cost and resolve-rate figures, derive $/resolved-task arithmetic, and compare that against official cache-pricing multipliers.

It was not covered: we did not run any agent ourselves, and we did not cover self-hosted or open-weight serving economics, nor negotiated enterprise rates. Scale context comes from the HAL paper, which reports 21,730 agent rollouts across 9 models and 9 benchmarks at roughly $40,000 total spend, with 2.5B tokens of shared agent logs. Last researched: September 2026.

What Does AI Agent Cost Per Resolved Task Actually Measure?

AI agent cost per resolved task divides average cost per task by resolve rate, so it measures dollars per successful outcome rather than dollars per attempt. Artificial Analysis defines “Cost per Task” as the weighted-average USD to complete one Intelligence Index task, multiplying input, cached, and output token prices by tokens consumed, weighted by benchmark weights, then dividing by task count (Artificial Analysis methodology).

The core derivation is:

$/resolved-task = average cost per task ÷ resolve rate

The same methodology page warns that “models that produce longer answers or more reasoning tokens will have a higher cost per task, even at identical per-token prices,” and its blended price assumes a 7:2:1 ratio of cache-hit : input : output tokens. Applied to one Aider row: gpt-5 (high) costs $29.08 across 225 tasks at 88.0% resolve, so $29.08 / (225 × 0.880) = $29.08 / 198 = $0.147 per correct task (Aider Polyglot leaderboard).

Why Resolve-Rate Rankings Differ From Dollar-Normalized Rankings

Resolve-rate rankings differ from dollar-normalized rankings because resolve rate compresses enormous cost spreads into a few percentage points. On the Aider Polyglot leaderboard, gpt-5 (low) resolves 81.3% at $10.37 while o3-pro (high) resolves 84.9% at $146.32 — derived $0.057 versus $0.766 per correct task, roughly 14× cost for 3.6 percentage points (Aider Polyglot leaderboard).

The SWE-bench bash-only mini-SWE-agent rows show the same pattern: MiniMax M2.5 (high) at $0.0733 per task and 75.8% resolve derives to $0.097 per resolved task, while Claude 4.5 Opus (high) at $0.7539 and 76.8% derives to $0.982 — about 10× cost for +1.0 pp. HAL states the framing plainly: “Agents can be 100x more expensive while only being 1% better” (HAL).

Arithmetic, shown: $0.0733 / 0.758 = $0.097; $0.3560 / 0.758 = $0.470; $0.4736 / 0.728 = $0.651; $0.5515 / 0.756 = $0.729; $0.7539 / 0.768 = $0.982. The 1D leaderboard’s first-place model is last on cost per resolved task.

How Multi-Trial Retries Distort Cost-Per-Task Headlines

Multi-trial retries distort cost-per-task headlines because published headline percentages often reflect more than one attempt. Aider publishes pass@1 and pass@2 separately and they sit 30+ points apart: gpt-5 (high) moves 52.0% → 88.0%, o3-pro (high) 43.6% → 84.9%, and gpt-5 (low) 43.1% → 81.3% (Aider Polyglot leaderboard).

Terminal-Bench shows the same inflation with more trials: GPT-6 Astra (max) goes pass@1 58.18% → pass@5 71.21%, and Fable 5.1 goes 57.88% → 78.79% (Terminal-Bench). The rigorous counterpart is tau-bench’s pass^k, defined as the chance the agent succeeds on all k i.i.d. trials, expected over tasks as C(c,k)/C(n,k); pass@k by contrast means at least one of k trials succeeds, and pass^1 = pass@1 is the default headline (tau-bench, arXiv 2406.12045). gpt-4o reaches pass^1 ≈ 61% on tau-retail but pass^8 falls to as low as ~25%.

The corrected figures use single-trial rates: $3,267.18 / 191.99 successes = $17.02, and $6,243.50 / 191.00 = $32.69 (Terminal-Bench leaderboard). Reporting the pass@5 number would understate cost per resolved task by roughly 13–21 pp of resolve rate.

Prompt Caching Changes the Cost Equation

Prompt caching changes the cost equation because cached input tokens can be priced at a fraction of fresh input, and long agent sessions accumulate them heavily. OpenAI discounts cached input up to 90%; for GPT-5.6+ cache writes cost 1.25× the uncached input rate and subsequent reads cost 0.1× that rate, with a 1,024-token minimum and no guarantee of a cache hit (OpenAI prompt caching guide).

Anthropic’s multipliers are 1.25× for a 5-minute cache write, 2× for a 1-hour write, and 0.1× base input for a cache read, with caching paying off after one read at 5 minutes or two reads at 1 hour; the Batch API adds a 50% discount (Anthropic pricing). Google’s Vertex AI pricing sets Gemini 3.1 Pro Preview at $2.00 input / $0.20 cached input / $12.00 output per 1M tokens, and Gemini 3.8 Flash at an introductory $0.75 / $0.075 / $3.75 through 2026-12-31, rising to $1.50 / $7.50 from 2027-01-01.

Cache-hit rates are not theoretical: Terminal-Bench payloads show GPT-6 Astra (max) with 1,443,351,702 cached input tokens against 1,505,789,330 uncached — roughly a 49% hit rate (Terminal-Bench). Artificial Analysis breaks its Terminal-Bench “Cost per Task” tab into input, cache hit, cache write, reasoning, and answer tokens, which is the right granularity for this arithmetic (Artificial Analysis Terminal-Bench view).

Why the Scaffold Matters as Much as the Model

The scaffold matters as much as the model because the same weights in different harnesses produce different accuracy and different dollar cost. HAL reports that running Opus 4.5 with an updated scaffold that uses Claude Code “drastically outperforms the CORE-Agent scaffold,” and its CORE-Bench Hard rows show Claude Code + Opus 4.5 at 77.8% / $87.16 against CORE-Agent + Opus 4.1 at 51.1% / $412.42 (HAL).

SWE-bench Verified compounds this: top entries span different attempt counts, with Sonar Foundation Agent + Claude 4.5 Opus at 79.2% (Attempts - 1) alongside TRAE + Doubao-Seed-Code at 78.8% (Attempts - 2+) and Atlassian Rovo Dev at 76.8% (Attempts - 2), so cross-agent comparison is scaffold- and attempt-dependent. Reproducibility requires pinning dataset versions: Harbor Hub lists terminal-bench at 66 tasks and terminal-bench-2-1 at 89, within the same family (Harbor Hub).

Reliability is a separate axis. The HAL Reliability Dashboard headlines that “recent capability gains have yielded only small improvements in reliability”: Gemini 3.5 Flash posts the highest accuracy at 80.0% but weaker consistency (0.74 / 0.81) than Claude Opus 4.5 at 74.6% accuracy with 0.81 / 0.84 consistency.

Building a Pareto Frontier Instead of a Single Ranking

Building a Pareto frontier instead of a single ranking means plotting $/resolved-task against resolve rate and keeping only models no other model beats on both axes. HAL frames this explicitly, noting that “downstream developers can’t tell the difference from 1D leaderboards,” and reports accuracy and dollar cost per agent run side by side (HAL).

SWE-bench already publishes a “Resolved vs cost (scatter plot)” tab, so the visualization exists to inspect rather than invent. Take three bash-only points: MiniMax M2.5 at 75.8% / $0.097, Gemini 3 Flash at 75.8% / $0.470, and Claude 4.5 Opus at 76.8% / $0.982. MiniMax M2.5 is on the frontier — nothing is cheaper at equal or better resolve rate. Claude 4.5 Opus is dominated: Gemini 3 Flash matches it within 1.0 pp at less than half the cost per resolved task, and MiniMax M2.5 matches it at one-tenth.

The HAL paper’s three-dimensional analysis spanning models, scaffolds, and benchmarks is the honest version of this exercise, because a frontier computed on one scaffold and one dataset version is a point estimate, not a property of the model (HAL paper, arXiv 2510.11977). For teams wiring agents into production, the practical checklist is: normalize by single-trial resolve rate, report pass@1 alongside pass@k, pin the dataset version, state the scaffold, and plot the frontier rather than quoting a rank. Related normalization work on judge reliability and structured-output scoring appears in our LLM judge reliability study and structured output benchmark, while serving-side cost drivers are covered in the inference engine benchmark and the vLLM vs SGLang serving benchmark. The AI stack reference and tools index cover the surrounding components, and long-context retrieval benchmarks matter when cache-hit rates depend on prompt reuse.

FAQ

Is cost per resolved task the same as cost per task?

No. Cost per task divides total spend by attempts, while cost per resolved task divides by successes, so resolve rate is the denominator. Artificial Analysis defines its “Cost per Task” as a weighted average over completed tasks, not resolved ones (Artificial Analysis methodology), which is why the two figures diverge sharply whenever resolve rate sits well below 100%.

How many trials should I budget for when comparing agents?

Budget for at least the trial count the leaderboard reports, and always publish pass@1 next to pass@k. Aider’s pass@1 and pass@2 differ by 30+ points (Aider Polyglot leaderboard), Terminal-Bench’s pass@1 to pass@5 spans 13–21 points (Terminal-Bench), and tau-bench’s pass^k measures all-k success, not at-least-one (arXiv 2406.12045).

Does prompt caching make cheaper models always better on the Pareto frontier?

No. Caching shifts the frontier by lowering effective input cost — OpenAI reads at 0.1× and Anthropic at 0.1× base input (OpenAI, Anthropic) — but Terminal-Bench’s ~49% cache-hit rate for GPT-6 Astra (Terminal-Bench) shows hits are partial, so quality differences still decide resolve rate.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
• NiteAgent — AI agent development, frameworks, and production patterns

Cross-links automatically generated from CodeIntel Log.