caveman: token compression for agents is a recovery-first game

Opening: The Honesty Tension

The repo tagline still promises caveman “cuts 65% of tokens,” yet the project’s own HONEST-NUMBERS doc and JetBrains’ third-party A/B both land in the high single digits on real agentic work — and the gap between those two numbers is the whole story.

That gap is not pure marketing drift. It is a category error the repo itself slowly stopped making: “65%” describes stylizing what the agent writes, while the durable savings come from what the agent reads. Those are different surfaces with different physics, and only one of them has a guaranteed undo path. An engineer choosing a compression layer needs to know which surface a number belongs to before believing it.

The repository sits at 108,379 stars and 6,280 forks, as measured via the GitHub API this run — you can see it on our GitHub trending tracker. That level of adoption makes the honesty discipline more interesting than the compression ratio: a repo that keeps a losing row in its own benchmark is teaching something rarer than a new tokenizer trick.

How This Review Was Researched

This review is based on the repository source, documentation, and issue tracker — we did not run it hands-on. Sources: the repository, the official docs.caveman.so quickstart, the committed benchmark docs, and the issue tracker; star and fork counts were verified against the GitHub API this run. Last reviewed: September 2026.

Every figure below is labelled: repo numbers are self-reported, JetBrains and Adobe Research numbers are third-party. Where a measurement is absent, the row says so rather than guessing. We also avoid the phrase “we benchmarked” throughout, because we did not.

Architecture: Three Layers, One Compression Engine

Three adoption surfaces — SKILL, PROXY, and MIDDLEWARE — route through one shared compression engine, so behavior stays consistent whether the agent is writing, reading, or your app is calling a provider docs. The layers differ only in where they sit in the request path.

This is the architectural argument worth stealing regardless of whether you adopt the tool: put the policy in one engine, and vary only the attachment point. A compression rule that behaves differently when invoked from a skill versus a proxy is a rule you cannot reason about.

Layer 1: SKILL (output side)

The SKILL is instructions injected into the agent’s own context that change how it writes: shorter prose, denser tool-call arguments, no restating. It ships 30+ agents with intensity levels lite, full, ultra, and wenyan, and the full skill rides along as roughly 1,000 estimated input tokens per call — a repo self-estimate from the quickstart.

Because SKILL only restyles output, nothing needs to be recovered: there is no original to preserve, since the agent never had the long version in the first place. That is why this layer is safe, and also why it is capped. JetBrains’ third-party test — skill-only, no proxy, Claude Code 2.1.200 on claude-sonnet-5, 86 SkillsBench coding tasks — measured 8.5% fewer output tokens, roughly 10% cost saving, with a sign test of p = 0.82 meaning no detectable quality change. They explicitly flag that activation was forced, so 8.5% is a ceiling rather than a floor.

Layer 2: PROXY (input side)

The PROXY is a local Go process on 127.0.0.1:8787 sitting between the agent and the provider, per the quickstart. Provider authentication — including Claude Pro/Max login — passes through untouched, and no Caveman server sits anywhere in the path. Everything that leaves your machine is your existing provider traffic, compressed.

This is the layer where the money is, because agent transcripts are dominated by what gets re-read: tool outputs, retrieved files, prior turns. It is also the layer where an unqualified compressor is dangerous, since a compressed tool result that cannot be expanded is a corrupted context. Adobe’s third-party CAVEWOMAN study makes the danger concrete: compressing the user’s prompt produced roughly 1.15× higher net cost on the mean, up to 1.8×/2.7×, with accuracy collapsing as models compensated through longer responses. That result is precisely why the repo never rewrites the user’s prompt.

Layer 3: MIDDLEWARE (your app’s LLM call)

The MIDDLEWARE layer attaches the same engine to your application’s own LLM call — the place where you assemble context from retrieval, tool schemas, and history before sending. It shares detect() and the CCR store with the other two layers, so the recovery guarantees are identical; only the injection point changes.

Practically, this is the layer for teams that do not want to modify agent behavior at all. You keep your prompts, your agent stays verbose, and the compression happens at the boundary where you already have code, logs, and schemas flowing through a single chokepoint. Our context engineering notes cover why a single chokepoint is easier to audit than scattered prompt edits.

Inside the Compression Engine

Compression starts with detect(), which types the payload and routes it to a per-type compressor, then contextwindow.Pack() fits survivors into a token budget — and CCR stores byte-exact originals so any lossy result can be undone before it reaches the model README.

detect() and the per-type compressors

detect() payload typing routes to per-type compressors with documented target savings: json 70–90%, log 85–95%, code 40–70%, diff 60–80%, search-result 80–95%, text/HTML 50–80% (README). These are targets, not guarantees — the recovery-first path means a payload that cannot hit them is returned unchanged.

The same README lists items that are never compressed: code, commands, file paths, exact error strings, and security warnings or irreversible confirmations (README). Read together, the two lists describe a boundary rather than a contradiction: savings come from compressible structure, while anything the agent must reproduce byte-for-byte — a shell command, a stack trace line, an approval prompt — is exempt. In harness design terms, compression is allowed to be lossy on bulk and lossless on meaning-bearing tokens.

contextwindow.Pack() and the token budget

contextwindow.Pack() fits candidate context to a token budget by BM25 relevance + recency + error signal, re-returning items in original order so chronology survives (README). Two details matter. First, the ranking mixes a lexical signal (BM25) with time and with an explicit error signal, so failures are not evicted merely for being old. Second, and more subtle: re-ordering back to original sequence means downstream reasoning about “what happened when” is not silently invalidated by the packer’s priorities.

CCR: the recovery store that makes it safe

CCR is a local SQLite file of byte-exact originals plus a recovery handle, default budget 512 MiB. When it is full, new writes fail and lossy transforms fall back to pass-through (HONEST-NUMBERS).

The recovery-first rule set is the design’s real contribution: a parse failure, an unsafe transform, absent durable recovery, a storage failure, or a result that is “not smaller” all return the original bytes (HONEST-NUMBERS). Compression therefore degrades toward doing nothing rather than toward corrupting context. For a production harness, that failure direction is the entire safety argument — the worst case costs you tokens, not correctness.

TOON, MCP toolkit, and pixel mode

The MCP toolkit exposes five tools: caveman_compress, caveman_retrieve, caveman_stats, caveman_toon_encode, caveman_toon_decode (README). caveman_retrieve is the recovery handle made addressable from inside the agent loop, which is what lets a model ask for the bytes it just lost.

TOON is a standalone structured-data re-encoder — an alternative serialization for JSON-shaped payloads rather than a summarizer. Pixel mode is the more provocative experiment: it renders skill bodies to PNG pages the model reads as an image, self-reported 1,069 → 415 estimated tokens, a 61% cut (README). Note the label — estimated, local tokenizer arithmetic, not provider-reported billing.

The Measurements: Third-Party vs Self-Reported, and the Red Row

The repo’s self-reported 54-run Claude Code wrap suite reports 885,793 → 591,673 provider-reported input tokens = −33.2%, with 18/18 exact-answer checks and a case-clustered 95% CI of 14.6–48.5% — but the raw harness artifacts are not in the checkout, so it is a pinned report, not a public reproduction (WRAP-BENCHMARK).

The CI is the honest part. A 95% interval from 14.6% to 48.5% says the point estimate is not a promise; case mix dominates. The suite’s other self-reported findings reinforce that: Headroom saved 6.7% and answered 15-of-18 right, browser pages came in at 121 vs 15,704 tokens (129.8×), memory-file compress averaged 46%, and a subagent-tax check found 219k of a 267k-char request was tool schemas (WRAP-BENCHMARK). That last one is the real prize: schema bloat is pure overhead that no amount of terse agent prose addresses.

Then the red row. Per-case results include a deliberately-kept RED ROW: dashboard HTML alert +9.9%, alongside CSV −55.1%, log −50.2%, YAML −46.2%, test output −27.8%, and deployment JSON −26.4% — all self-reported (WRAP-BENCHMARK). A negative result preserved in the flagship table is unusual enough to be worth more than the positive rows: it tells you the benchmark is measuring something rather than advertising.

Third-party results frame it. JetBrains (third-party) measured 8.5% fewer output tokens, not the advertised 65%. Adobe Research’s CAVEWOMAN (third-party) found output-side caveman style cuts realized cost 1.4–2.4× per model, up to 3× best case, while input-side compression of the user’s prompt is lose-lose. Our benchmark arena collects similar cross-lab discrepancies when harnesses disagree.

Finally, cloud attribution tiers: inferred (local tokenizer estimates), benchmark_counterfactual (pinned benchmarks), and verified (live traffic behind eval gates with signed receipts). The repo states that neither of the first two is an invoice (HONEST-NUMBERS) — a distinction most compression vendors blur.

Production Tradeoffs: When It Loses

caveman wins on token-billed input and loses on request-billed APIs, verbose HTML with no transform available, and pure code generation where there is barely any prose to cut (HONEST-NUMBERS). Telemetry is on by default but excludes prompts, code, and paths (SECURITY).

Three loss cases deserve explicit planning. No-transform HTML is a documented loss case: when a payload has no safe re-encoder, caveman returns it as-is while still paying the detection overhead — this is the +9.9% red row’s mechanism. Per-request billing is structurally unbeatable: a shorter answer on GitHub Copilot premium requests is still one request, so output terseness buys nothing on that pricing model. Pure codegen workloads contain almost no prose to remove, so the ratio approaches zero while the machinery still runs.

Operational footnotes for procurement: telemetry can be disabled with caveman telemetry off or DO_NOT_TRACK=1, and it never sends prompts, code, or file paths (SECURITY). The license is split — MIT for adoption surfaces, BSL-1.1 for the engine, converting to Apache-2.0 on min(2030-06-21, 4 years post-ship); self-hosting your own first-party traffic is free, while third-party hosting needs a commercial license (LICENSE). If you plan to offer compression as a service, that second clause is a budget line, not a footnote.

caveman vs the Field

None of these tools shrink the same slice of the request, so the honest comparison is what each one compresses, whether the original survives, and who published the number for the same workload WRAP-BENCHMARK. Rows below separate third-party measurements from repo self-reported figures.

The two “no public measurement” rows are the point: absence of a number is information, and we would rather print it than synthesize one.

FAQ

Three questions decide most adoptions: whether caveman rewrites your prompt, what happens when the local recovery store fills, and whether shorter output helps under per-request billing (HONEST-NUMBERS). The short answers are no, it falls back to pass-through, and generally no.

Does caveman rewrite my prompt?

No. The proxy and middleware layers compress tool output, retrieved context, and prior turns on the input side, and the skill restyles what the agent generates. The user’s own prompt is left alone — a deliberate choice driven by Adobe Research’s CAVEWOMAN result, which found input-side prompt compression raised net cost by roughly 1.15× on the mean while accuracy collapsed.

Is the input-side proxy safe if the recovery store fills up?

Yes, in the sense that failure is lossless. The CCR store defaults to a 512 MiB budget; once full, new writes fail and lossy transforms fall back to pass-through rather than dropping data (HONEST-NUMBERS). Storage pressure costs you compression ratio, never context integrity.

Do I lose on per-request billing?

Usually yes. On premium-request pricing such as GitHub Copilot premium requests, a shorter answer is still one request, so output-side terseness buys nothing (HONEST-NUMBERS). Savings show up where you pay per input token — long agent transcripts with heavy tool output — which is exactly the PROXY layer’s territory.

The Bottom Line

Treat caveman as an input-side, recovery-first middleware bet: modest, honest savings on agentic transcripts when byte-exact originals are guaranteed, and a marginal gain when it only styles what the agent writes (WRAP-BENCHMARK). Judge it on the red row, not the tagline.

The engineering lesson generalizes past this repo. Output-side terseness is a marginal win on real agentic work: JetBrains got 8.5% under forced activation, and no amount of caveman prose style reaches the tagline’s 65% (GitHub). Input-side compression is the real lever — a −33.2% self-reported swing on provider input tokens — but it is only safe when recovery is guaranteed, because the alternative is the CAVEWOMAN failure mode where compression silently raises cost and destroys accuracy.

Adopt accordingly. If your workload is token-billed, context-heavy, and full of tool output, the PROXY layer with a watched CCR budget is the defensible choice. If your workload is request-billed, code-only, or HTML-heavy, the honest numbers say you are paying for machinery that will not pay you back — and the repo’s own benchmark table would tell you the same thing before we did.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides

Cross-links automatically generated from CodeIntel Log.