Contents
Why Terminal-Bench 4.0 Scores Are Not Model-Only Measurements
Terminal-Bench 4.0 scores are joint measurements of an agent stack, not isolated model ability: the benchmark has 66 tasks, uses five trials per task, and treats harness and reasoning effort as reported configuration, according to the Terminal-Bench 4.0 definition accessed September 30, 2026.
A Terminal-Bench row is the output of an evaluation protocol, not a weights release. Tool wrapper, command harness, reasoning setting, safeguard policy, task suite, trial schedule, scorer, and publisher all remain part of the result. The benchmark definition, accessed September 30, 2026, emphasizes real-world work in containerized terminal environments, including science-adjacent and frontier engineering problems.
The same Claude Opus 5.5 ID appears at 66.4% in Anthropic’s self-report, accessed September 30, 2026, but 61.62% on the Vals mirror snapshot dated September 22, 2026. Those measurements should not be subtracted as though they were model-only scores. CodeIntel’s Terminal-Bench v2.1 benchmark study documented same-model, different-harness spreads from 2.7 points to 8.1 points, including one case that reversed direction; that is an engineering mechanism from prior published runs, not a new measurement taken here.
The suite itself is open: the task code lives in the Terminal-Bench repository, alongside a project site and a published task dataset. What is not standardized across publishers is the harness that drives those tasks — which is precisely the variable the rows below diverge on.
Inside the Anthropic Opus 5.5 Self-Report
Anthropic’s Opus 5.5 self-report is a vendor-run result, not a neutral leaderboard entry: the launch post reports 66.4% at xhigh with ±2.6 and 64.8% at max, within noise, while Section 8.5 identifies Claude Code --bare and safeguards (source, accessed September 30, 2026).
The system-card account underlying the rounded launch figure gives 66.36% and specifies Claude Code --bare, xhigh thinking, safeguards, and server-side fallback. It records fallback on 2.5% of requests and in 10% of trials across five trials for each of 66 tasks—330 trials (Anthropic launch post, accessed September 30, 2026). The surrounding change surface is documented in Anthropic’s Opus 5.5 model overview, what’s-new notes, and migration guide.
Reasoning labels are not an ordinal performance guarantee. Anthropic reports 66.4% with ±2.6 at xhigh and 64.8% at max, explicitly calling the latter within noise (launch post, accessed September 30, 2026). The appropriate conclusion is that effort belongs in the row, not that xhigh must always exceed max.
The same September 30, 2026 launch post lists Fable 5.1 at 55.8%, Opus 5 at 52.3%, GPT-6 Astra at 57.9% as reported by OpenAI’s GPT-6 launch note, and GPT-5.6 Sol at 37.3% (Anthropic comparison table, accessed September 30, 2026). That provenance qualifier matters: a vendor table can reproduce several reporting regimes without making them mutually equivalent.
Same Benchmark, Same Model ID, Different Boards
The same Opus 5.5 ID appears at 66.4% in Anthropic’s xhigh self-report but 61.62% on Vals’ September 22, 2026 snapshot, while BenchLM has no row as accessed September 30, 2026; this is source and configuration disagreement, not evidence that the model itself changed (Anthropic, Vals, BenchLM).
Table 1 — Same model ID, different board or harness
| Source and date | Model ID | Reported TB 4.0 score | Configuration / reporting note |
|---|---|---|---|
| Anthropic Opus 5.5 launch post, accessed 2026-09-30 | Claude Opus 5.5 | 66.4% ±2.6 at xhigh; 64.8% at max | Vendor-run; reasoning effort is part of the reported score |
| Anthropic system card §8.5, via the launch post, accessed 2026-09-30 | Claude Opus 5.5 | 66.36% | Claude Code --bare, xhigh, safeguards enabled with server-side fallback (2.5% of requests / 10% of trials), 5 trials × 66 tasks |
| Vals Terminal-Bench 4.0 mirror, 2026-09-22 snapshot | Claude Opus 5.5 | 61.62% | Independent mirror snapshot |
| BenchLM Terminal-Bench 4.0 board, accessed 2026-09-30 | Claude Opus 5.5 | No entry | Board tops at GPT-6 Astra 58.18% (Codex · max) |
| Artificial Analysis, accessed 2026-09-30 | GPT-6 Astra | 59.6% xhigh / 59.1% max | Independent board |
| ModelCap, accessed 2026-09-30 | GPT-6 Astra | 58.2% [55.4–61.0] | Confidence interval reported by the board |
The Anthropic rows represent the same result at different precision: the launch post’s 66.4% corresponds to the system card’s exact 66.36% (Anthropic, accessed September 30, 2026). They should not be counted as independent replications.
The remaining sources are not interchangeable leaderboards. Vals supplies a dated mirror snapshot, BenchLM has incomplete Opus 5.5 coverage, Artificial Analysis separates xhigh and max, and ModelCap publishes an interval. None is a universal denominator.
Harness and Reasoning-Effort Are First-Class Variables
Harness and reasoning effort are first-class variables because an agent’s tools, scaffolding, safeguards, and thinking setting can change terminal outcomes without changing the model ID; Anthropic discloses those controls for Opus 5.5 in its launch post accessed September 30, 2026.
A harness is the path through which the model encounters tasks: command construction, tool exposure, agent scaffolding, and execution policy all sit between the model and terminal outcome. Anthropic’s disclosure of --bare, xhigh thinking, and safeguard fallback is therefore part of the result definition, not optional implementation detail (system-card details, accessed September 30, 2026). --bare is a configuration of the Claude Code harness, and Anthropic documents the reasoning-continuity semantics behind its thinking controls in its preserved-thinking notes. This is the same coupling described in CodeIntel’s eval harness architecture analysis.
Reasoning effort also requires a qualified comparison. Artificial Analysis, accessed September 30, 2026, reports GPT-6 Astra at 59.6% with xhigh and 59.1% with max. That pair does not establish a universal ordering; it establishes that the effort label must travel with the score.
Operating conditions add further context. Anthropic’s telemetry post, covering March through September 2026, reports context per request increasing 2.6x, the input-to-output ratio moving from 189:1 to 324:1, prompts becoming 3.3x longer, and 40% more model calls. These are not Terminal-Bench scores, but they show why long-session behavior and harness state belong in an evaluation record.
Trial Protocol and Reporting Regimes Change the Decimal
Trial protocol and reporting regimes change the decimal because ModelCap’s September 30, 2026 board reports GPT-6 Astra at 58.2% with a 55.4–61.0 interval, while Vals’ September 22, 2026 snapshot reports 57.07%, exposing different uncertainty and coverage conventions for the same benchmark (ModelCap, Vals).
Reported Terminal-Bench configurations use five trials per task; Anthropic’s system card expands that protocol into 330 trials across 66 tasks (Terminal-Bench definition, Anthropic launch post, accessed September 30, 2026). A point estimate, a ± value, and a confidence interval communicate different amounts of information even when their central values are close.
Table 2 — Decision rules for reading Terminal-Bench 4.0
| Failure mode | What it misreads | Decision rule |
|---|---|---|
| Comparing Anthropic’s 66.4% to Vals’ 61.62% as a model gap (Anthropic, accessed 2026-09-30; Vals snapshot, 2026-09-22) | Treats a vendor-run config as a board-equivalent number | Cite both with source and date; label 66.4% as Anthropic’s xhigh --bare harness result, not a neutral board row |
| Treating BenchLM’s missing Opus 5.5 row, accessed 2026-09-30, as model absence or failure | Confuses archive and configuration coverage with a benchmark result | Check for missing entries and board methodology before quoting rank |
| Reading two boards that differ by 2.7–8.1 points | Assumes one board is wrong or the model changed | Compare harness, effort, trial count, and date; if not aligned, report a range |
| Quoting “Model X beats Model Y” from top rows | Ignores protocol mismatch and reversed-direction cases | Require side-by-side rows with the same harness, configuration, and date before accepting ordinal claims |
| Using a lab self-report as the board number | Creates a category error between a vendor launch measurement and an independent board | Label vendor self-reports as vendor-run; compare only like-for-like |
The decision table is not a ranking. It prevents a decimal from silently absorbing changes in harness, effort, protocol, and row coverage.
Lab Self-Reports vs. Independent Boards
Lab self-reports and independent boards answer different questions: Anthropic documents a controlled vendor configuration, while Vals and BenchLM publish independent snapshots, so neither category should be silently substituted for the other as if they were equivalent (Anthropic, Vals, BenchLM, accessed September 30, 2026).
A vendor page is the primary source for what the vendor says it ran. It is not independent validation. An independent board is the primary source for its own dated snapshot, but it is not a permanent or universally applicable leaderboard. The distinction between a pinned snapshot identifier and an evolving generation alias is spelled out in Anthropic’s model IDs and versioning and models overview.
Anthropic’s own footnote provides a useful alignment check: “The public leaderboard (5 trials/task, Claude Code harness) reports Claude Opus 5 at 51.8%; our setup reproduces it at 52.3%, within noise” (launch post, accessed September 30, 2026). The important move is matching the protocol before judging the decimal.
Conversely, mixed-news, accessed September 30, 2026, correctly distinguishes absence from failure: the public board has no Opus 5.5 entry and tops out at 58.2%. That describes coverage, not a model result.
A Like-for-Like Checklist for Terminal-Bench 4.0
A like-for-like Terminal-Bench 4.0 comparison requires the same task suite, model ID, harness, reasoning effort, safeguards, trial policy, and reporting date; the definition accessed September 30, 2026 explicitly makes harness and effort part of the score reported in published leaderboard rows.
Before accepting an ordinal claim, require a shared comparison record:
- Identity: Pin the exact model ID and board access date.
- Harness: Record the agent command, tool surface, and whether the row uses Claude Code, Codex, or
--bare. - Effort: Preserve labels such as xhigh and max rather than normalizing them to “reasoning enabled.”
- Safeguards: Record fallback behavior and any other intervention that distinguishes the run.
- Protocol: Require the same 66-task suite and five-trial schedule (definition, accessed September 30, 2026).
- Provenance: Label vendor-run, mirror, and independent results, then retain each snapshot date.
- Uncertainty: Preserve reported ± values or intervals instead of retaining only a central decimal.
- Decision: If any required field differs, report a range rather than manufacture a rank.
For production model selection, extend that record into a behavioral contract covering tool use, recovery, and required task outcomes, as described in model-upgrade contract testing.
From Public Board Ranges to Your Own Workload Measurement
Public board ranges are inputs to a workload measurement, not replacements for it: the September 30, 2026 Terminal-Bench definition describes a fixed suite, while an architecture review should measure the actual task distribution, harness, effort, and pass criteria before choosing a model.
Terminal-Bench emphasizes science-adjacent and frontier engineering work, but an architecture decision also depends on the tasks an organization actually resolves. Public results can calibrate expectations; they cannot establish workload-specific uplift.
The reviewing team should:
- Build a representative task portfolio from recurring operational and engineering objectives.
- Freeze the model ID, harness, effort, safeguards, and task versions.
- Use matched trials so model comparisons share tasks and scoring rules.
- Record resolved-task success, uncertainty, cost, latency, interventions, and fallback behavior.
- Version the configuration and preserve failures as regression cases.
Normalize economics with cost per resolved task rather than cost per model call, and evaluate session behavior with the workload assumptions in CodeIntel’s long-horizon agent benchmark study. The HAL paper on holistic agent evaluation is the methodology template for that exercise: it scores models, scaffolds, and benchmarks on three axes rather than collapsing them into one ranking. The public score is context; the paired workload result is the decision evidence.
FAQ
Terminal-Bench 4.0 results are configuration-bearing evidence: the suite has 66 tasks, and vendor, mirror, and independent-board scores can diverge because harness, effort, trials, and row coverage are not aligned even when the model label matches across publishers (definition, Anthropic, and Vals, accessed September 30, 2026).
Why does Anthropic report Opus 5.5 at 66.4% while Vals reports 61.62%?
Anthropic reports 66.4% at xhigh effort with ±2.6 and 64.8% at max, while Vals’ September 22, 2026 mirror reports 61.62%; the launch post, accessed September 30, 2026, and Vals board represent different reporting contexts, so their gap is not a clean model-to-model delta.
Is GPT-6 Astra above or below Opus 5.5 on Terminal-Bench 4.0?
The defensible answer is source-dependent: Anthropic’s September 30, 2026 launch table gives Astra 57.9% as reported by OpenAI versus Opus 5.5 at 66.4%, while Vals’ September 22, 2026 snapshot gives Astra 57.07% versus Opus 5.5 at 61.62%; BenchLM cannot settle the pair because it has no Opus 5.5 row.
Which number should I quote in an architecture review?
Quote a source-and-date-qualified range unless the comparison is demonstrably aligned: identify Opus 5.5 as 66.4% in Anthropic’s September 30, 2026 self-report or 61.62% on Vals’ September 22, 2026 snapshot, and never present either as the unqualified board score.
Why is there no Opus 5.5 row on some public boards?
Some public boards are snapshots with incomplete model coverage: BenchLM’s September 30, 2026 board omits Opus 5.5 and lists GPT-6 Astra at 58.18%, an absence also reported by mixed-news on September 30, 2026; a missing row is not a failed evaluation.
How do I benchmark my own terminal-agent workload without treating a public board as truth?
Benchmark your workload by freezing a representative task set, harness, effort, safeguards, and trial policy; log resolved-task success, uncertainty, cost, latency, and interventions; then make paired model comparisons under identical settings rather than importing a public ranking as architectural truth.
How This Guide Was Built
This analysis is derived from vendor documentation, vendor system cards, launch posts, and published independent benchmark boards — we did not run the models, the harnesses, or the benchmark ourselves. It cross-checks Anthropic’s launch post with Vals, BenchLM, Artificial Analysis, ModelCap, and mixed-news, all accessed September 30, 2026.
Source roles were kept separate. Anthropic is authoritative for Anthropic’s disclosed configuration, while each board is authoritative only for its dated row. The dated Vals snapshot was not merged into BenchLM, Artificial Analysis, or ModelCap to create a synthetic leaderboard.
The same-model spread is treated as a synthesis of published configurations, not a fresh experiment. Where pages expose different protocols, uncertainty summaries, or model coverage, this analysis preserves those differences rather than inferring missing evidence.
The Bottom Line
Terminal-Bench 4.0 leaderboards are harness-coupled evidence: Anthropic’s Opus 5.5 result leads its cited comparators, yet cross-source disagreement can erase or reverse leader margins; Anthropic itself says that “at these levels of capability … benchmark margins have become a less reliable guide to real-world differences” in its launch post accessed September 30, 2026.
The operational rule is straightforward: retain source, configuration, uncertainty, and date; align those fields before accepting a rank; report a range when they do not align. No public board is the universal truth. Model selection should ultimately rest on paired, reproducible evidence from the workload the architecture must solve.
📖 Related Reads
- ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
- NiteAgent — AI agent development, frameworks, and production patterns
Cross-links automatically generated from CodeIntel Log.
