Hey, I'm Crypto Nite. I build AI agent systems for a living, and I got tired of model comparisons that are a paragraph of vibes and a sponsored table.

So CodeIntel works differently: I write one brief, give the exact same brief to every model in the pool, have a judge score each draft against a fixed rubric, and publish the result — including the drafts that failed. That is the whole idea. The benchmark is why this site exists — though I'll be straight with you: most of the archive predates it and is ordinary reference writing. The posts that come out of the arena are the ones worth your time.

What I actually run

The pipeline is mine and it runs on this machine. At the time of writing it holds96 matches, 31 models that have at least one usable score out of 32 in the pool, and a judge pool of31. Every match follows the same shape:

1. A brief is written with an explicit contract — required sections, word floors, citation rules.
2. Every model in the pool gets that identical brief, in parallel, with no follow-up turns.
3. A judge model scores each draft on five weighted criteria — three passes where the multi-pass gate runs; the share of drafts that actually record passes is stated in the methodology.
4. Drafts below the gate are recorded as failures rather than discarded.
5. The winner becomes a post, and its code is extracted and executed before publishing.

The weights, the gate, the exclusions and the places where this method is weaker than it looks are all stated in the methodology. I'd rather you read the caveats than take a score on faith.

What I test, and what I don't

What I test: whether a model can follow a written contract, cite sources that resolve, hold structure across several thousand words, and produce code that runs. Those are the things that decide whether an agent is usable, and they are measurable.

What I don't test: your workload. A model that scores well on a long-form engineering brief may be the wrong choice for a 200-line refactor, and no benchmark on this site will tell you that. Treat the scores as a filter for what to try, not a verdict on what to buy. I also don't publish pricing or latency — not because they don't matter, but because they change faster than I can keep a page honest about them.

Where the code comes from

The fix database and the posts are not written from memory. They start from a real merged pull request or a real failure I hit, and the snippets are extracted and executed by CI before publishing. That pipeline is a script on the same machine as the benchmark, not something you can audit from here — so treat "the code runs" as my claim, not a proof you were handed.

The trending tracker pulls GitHub's API on a schedule and records star movement over time, so you can see a repo's trajectory rather than a single snapshot. Thestack reference is the set of things I actually use to build this, with the reasons I picked them and what I'd skip.

Why should you believe any of this

You shouldn't, yet. You should check. Every number on the model comparison traces back to a specific match record, and each row lists its drafts with their dates and scores. If a figure looks wrong, it is probably wrong, and I want to know — the corrections I've made to this site in a single afternoon are documented rather than quietly overwritten.

Built with Hermes Agent and Astro. If you want to argue about a weight in the rubric, that's a specific enough disagreement to be worth having.