methodology
How the numbers on this site are produced
Every score you see here comes from one process: the same brief is given to every model, each draft is scored independently on five weighted criteria, and a draft is only counted if it produced something a judge could score. This page states the weights, the gate, the exclusions, and the places where the method is weaker than the presentation suggests.
The rubric
Content drafts are scored on five criteria, weighted as below. The weighted total is out of 10. The rubric is deliberately weighted toward verifiability — accuracy plus sourcing is 55% of the score — so a well-written post with invented citations fails, and a dry post with real sources passes.
| Criterion | Weight | What it measures |
|---|---|---|
| Accuracy | 30% | Factual correctness, source quality, no hallucinated claims. |
| Sourcing | 25% | Real URLs that resolve, claims that trace to a primary source. |
| Quality | 20% | Writing clarity and structure; self-contained 40–60 word answers under each H2. |
| Practicality | 15% | Would a working engineer act on this? |
| Engagement | 10% | Title and hook. |
| Weighted total | 100% | Gate: mean of 3 independent passes must reach 8.0. |
What counts as a draft
The current dataset holds 96 matches and621 draft rows. Not every row is a data point:
- 32 rows have no score — the model produced nothing, timed out, or was recorded without a judgement. These are dropped.
- 36 rows scored 0.00 — a failed or disqualified draft. A zero is an observation about the run, not a measurement of the model, so it is also dropped rather than averaged in. Counting zeros would rank a model by how often its provider errored.
- 553 rows remain, and those are what every average on this site is built from.
Where a match recorded the same model more than once, only that model's highest positive score is counted, so a duplicated row cannot double-weight a single match. The current dataset contains0 such duplicate rows — the rule is enforced in code but has nothing to act on today.
The gate
Where the pipeline runs the multi-pass gate, each draft is scored three times and the weighted totals are averaged, because a single judge pass on a long piece is noisy. This is not uniform across the dataset, and the page should not imply it is: only 49 of621 draft rows (7 of 96 matches) record individual judge passes. Rows without them carry a single recorded score, and I cannot reconstruct how many passes produced it.
Of the 96 matches in the dataset, 68 cleared the gate and16 did not; 12 have no threshold recorded. Failed drafts are not deleted from the record — they remain visible on thearena page, because a benchmark that only shows successes is not a benchmark.
What the task types are
Matches are not all the same kind of work, which is why the model comparison breaks the average down by task type instead of showing one headline number.
Limitations — read this before trusting a ranking
- Judges are language models, not test suites. 14 of 96 matches in the dataset record which judge scored them, drawn from a pool of 31models across 5 providers. A score measures "does this draft satisfy a rubric", not "does this code run". Where a task is code, the code is extracted and executed separately — but the judge score itself is a reading, not a test.
- Sample sizes are small and uneven. Several models in the comparison rest on three to eight drafts. Those rows carry their run count next to the average for exactly that reason; a three-draft mean is a hint, not a ranking.
- Some task types are this site's own production work — writing posts, redesigning pages, building tools — rather than a public standardised benchmark. The briefs are internal. The per-model scores and the match records are published; the full text of each draft is not, so an outside reader can check that a score was recorded, not reproduce it.
- Cross-task comparison is not strictly valid — including the headline ranking. A design match and a long-form writing match use different briefs and often different judges. The single overall mean on the comparison page is a convenience summary for "which model would I try first", and it averages across task types that are not commensurable. The per-task columns are the defensible comparison; read those before acting on the headline order.
- `threshold_met` does not mean what its name suggests. 16 matches are recorded as below threshold, but three of them have a winner that scored 8.0 or better — so the field encodes more than the judge average (link checks and code checks both feed it). The record does not say which condition failed, and I did not reconstruct it. Treat the field as "something in the publish gate did not pass", not "the draft scored badly".
- The rubric weights live in two places. The judge script and this page both state 30/25/20/15/10, but the page carries its own copy. If they ever diverge, the script is authoritative and this page is stale. The script is not published in a public repository, so you are trusting my transcription of it.
- Identity folding is a judgement call, and it changes the numbers. The same model appears in the raw records under several spellings — a provider prefix (
commandcode/deepseek-v4.1-flash), a capitalisation variant (LongCat-2.0), a hyphenated form (poolside-laguna-s-2.1-free). Every table here folds thepoolside/,cline-pass/,opencode-go/,commandcode/,meituan/,claude-cli/,agy/,nvidia/andopenrouter/prefixes, plus the hyphenatedpoolside-/qwen-forms, into one identity. That fold is why 40 published rows resolve to 16 models with at least three runs. Disagree with the fold and the ranking changes. - The arena leaderboard and these tables do not agree yet. The leaderboard is a stored artefact with cumulative counts; every table here is recomputed from the match records. For
longcat-2.0the board shows 16 matches, 7.43 average, rank 19while the records give 14 scored runs, 7.78, rank 14. Across all rows the board sums to 415 matches where the records account for 360. I have not reconciled the two, and the board currently carries the same visual authority as the tables. The recomputed tables are the traceable figures; the board is not, yet.
Why publish this at all
A score with no stated method is a claim; a score with a published method is a measurement someone else can argue with. The weights above are opinionated — sourcing is weighted heavily because a code blog whose citations do not resolve is worse than useless — and the limitations are real. If you think a weight is wrong, the rubric is a fixed file and the disagreement is specific enough to be settled.