LLM Caching at Scale: Production Architecture Patterns

LLM caching is no longer an optimization—it’s a cost-control necessity. As inference volumes scale past millions of daily requests, the gap between a naive deployment and a cache-aware architecture can amount to an order of magnitude in cost and a multiple on p95 latency, without changing a single model weight. This analysis maps the three caching tiers—provider prompt caching, self-hosted KV-cache reuse, and semantic caching—and shows how to compose them into a coherent production layer. We cover where each tier sits in the AI stack reference, how they interact, and the engineering tradeoffs that determine whether your cache saves money or silently corrupts outputs.

How This Was Researched

This analysis is based on official vendor documentation verified as of August 2026: Anthropic’s prompt caching docs, OpenAI’s prompt caching guide, DeepSeek’s KV cache guide, and vLLM’s automatic prefix caching docs. Pricing data comes from Anthropic’s official pricing page. We compared caching tiers by mechanism, cost structure, and correctness risk. Not covered: framework-specific client caching wrappers and provider-specific volume discounts.

Tier 1: Provider Prompt Caching — The Zero-Engineering Baseline

Provider prompt caching is the first tier you should enable because it requires no infrastructure changes, only prompt discipline. Anthropic’s implementation caches prompts on the first request and serves subsequent requests with a stable prefix at reduced cost, with cache writes priced at 1.25x base input tokens and cache reads at 0.1x base input tokens per Anthropic’s pricing documentation. OpenAI’s equivalent automatically caches prompts with 1024+ tokens and applies a 50% discount on cached input tokens per OpenAI’s prompt caching guide. DeepSeek exposes an analogous context cache with discounted hit tokens per DeepSeek’s KV cache guide.

The catch: this tier only works when your prompt prefix is byte-identical across requests. Any reordering, timestamp injection, or dynamic content before your static context breaks the cache. You’re not designing a cache—you’re designing a stable prompt prefix discipline.

Tier 2: Self-Hosted KV-Cache Reuse — Shared Prefixes, Not Just Full Matches

When you self-host with vLLM, automatic prefix caching (APC) reuses KV cache blocks across requests that share any prefix, not just full-prompt matches, via hash-based block matching per vLLM’s APC documentation. This is a superset of provider caching: it catches repeated system prompts, few-shot exemplars, and tool definitions even when the tail of the prompt varies. DeepSeek documents a comparable mechanism for its hosted API in its KV cache guide.

The engineering cost is real: you need sufficient GPU memory to hold KV blocks, and you must configure --enable-prefix-caching explicitly. But for high-QPS self-hosted deployments, this tier often delivers substantial effective cost reduction because it amortizes the most expensive part of generation—the prefill—across requests.

# vLLM server with APC enabled
vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --enable-prefix-caching \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.9

Tier 3: Semantic Caching — Short-Circuiting Before the Model

Semantic caching uses embedding similarity to return cached responses for near-duplicate requests without invoking the model at all. The architecture: embed the incoming request, query a vector store for a similar cached request above a threshold, and return the stored response if found. DataStax’s semantic caching guide describes this pattern in detail, and GPTCache is the reference open-source implementation.

The critical design decisions are threshold calibration and eviction policy. A similarity threshold that’s too loose returns wrong answers for requests that merely look alike; too tight and your hit rate collapses. You also need TTLs and tenant isolation—a cached response for one customer context can poison another’s.

def semantic_cache_lookup(request: str, threshold: float = 0.92) -> str | None:
    query_embedding = embed(request)
    hits = vector_store.query(query_embedding, top_k=1, namespace=request.tenant)
    if hits and hits[0].score >= threshold:
        if not is_expired(hits[0].ttl):
            return hits[0].response
    return None

The correctness risk is non-trivial: near-duplicate inputs can have different correct answers. A request for “weather in Berlin” and “weather in Berlin tomorrow” are semantically close but temporally distinct. Bound false hits with strict thresholds and never cache time-sensitive or user-specific data.

How do you design an LLM caching layer for production systems?

Design an LLM caching layer by stacking all three tiers in order of increasing correctness risk: provider prompt caching first (zero risk, requires prompt discipline), self-hosted KV reuse second (no correctness risk, requires memory headroom), and semantic caching last (highest risk, requires strict threshold and TTL governance). Each tier has different invalidation semantics and cost profiles, and you must instrument each separately to know which is actually saving money.

Cache-Aware System Design: Prompt Assembly Discipline

Cache hit rates are determined at prompt assembly time, not at cache lookup time. The rule is simple: static content first, dynamic content last. Your system prompt, tool definitions, and few-shot exemplars go at the top; user-specific context, timestamps, and retrieved documents go at the bottom. This ordering maximizes the shared prefix across requests, which is what both provider caching and vLLM APC exploit.

Cache keys matter too. For semantic caching, the key is the embedding of the normalized request—strip whitespace, canonicalize entity names, and include a namespace for tenant isolation. For provider caching, the key is implicit: the prompt prefix itself. Any prompt-assembly logic that reorders sections will silently destroy your hit rate.

def assemble_prompt(system_prompt: str, tools: list, dynamic_context: str) -> str:
    # Static prefix first — this is what gets cached
    static_prefix = f"{system_prompt}\n\nTOOLS:\n{json.dumps(tools, sort_keys=True)}"
    # Dynamic tail last — this is what varies per request
    return f"{static_prefix}\n\nCONTEXT:\n{dynamic_context}"

This discipline is the difference between a high cache hit rate and a negligible one on identical workloads. It’s also the cheapest optimization you can make—it costs zero infrastructure and only requires code review discipline.

Correctness & Invalidation in Agentic Systems

Agentic systems break naive caching because tool outputs change over time. A cached response that references a stale tool result is worse than no cache at all—it’s confidently wrong. The invalidation strategy must be explicit: TTLs for time-sensitive data, versioned cache keys for tool definition changes, and a hard rule against caching any response that includes tool outputs with timestamps or mutable state.

Cache poisoning is a real threat in multi-tenant systems. If one tenant’s request pollutes a shared cache namespace, other tenants receive that tenant’s data. This is why namespace isolation is non-negotiable in the semantic tier, and why provider and KV-cache tiers are safer—they cache only the prompt, not the response.

When NOT to cache: non-deterministic tasks (creative writing, brainstorming), time-sensitive data (stock prices, news), and multi-turn agent state where each turn depends on prior turns. In these cases, the cache saves pennies but costs correctness. Our determinism budget analysis covers this tradeoff in detail.

Measuring ROI: Instrumenting Each Tier

You cannot optimize what you don’t measure. Each caching tier needs separate instrumentation: cache hit rate, cache read vs. write token mix, effective cost per request, and latency percentiles. Provider APIs expose cache read/write tokens in their usage responses—log them. vLLM exposes APC hit rates via its metrics endpoint. Your semantic cache should log threshold scores and hit/miss decisions.

The key metric is effective cost per request, not raw hit rate. A sky-high semantic cache hit rate on a workload with a handful of tokens per request saves little; a modest provider cache hit rate on a multi-thousand-token system prompt saves a great deal. Measure cost per request before and after each tier, and you’ll know which tier is paying for itself.

This instrumentation also feeds capacity planning. If provider cache reads dominate your token mix, your cost ceiling is set by output tokens. If semantic cache hits dominate, your cost floor approaches zero for repetitive workloads. The scaling production AI systems analysis covers the capacity implications in depth.

The three tiers are complementary, not competing. Provider caching handles the bulk of your static prompt cost with zero engineering. vLLM APC catches shared prefixes that cross request boundaries. Semantic caching short-circuits the most repetitive workloads entirely. Deploy all three, instrument each, and tune the semantic threshold until false hits approach zero. Your tools directory should reflect this stack.

FAQ

This section answers the recurring questions engineering teams raise when adopting LLM caching in production. Each answer is grounded in the vendor documentation cited above and reflects the tradeoffs between cost savings and correctness risk that determine whether a given cache tier is safe to enable.

Is prompt caching automatic with Anthropic and OpenAI?

No. Anthropic requires explicit cache_control breakpoints in your prompt per their prompt caching docs. OpenAI’s caching is automatic for prompts with 1024+ tokens per their guide, but you must ensure your prompt prefix is stable or the cache will never hit.

When should I use semantic caching instead of prompt caching?

Use semantic caching for high-volume, near-duplicate requests where the cost of an occasional false hit is acceptable. Use prompt caching for everything else—it carries no correctness risk. Semantic caching’s embedding-based lookup adds latency and risk, so only justify it when repetitive traffic dominates your spend.

How much can LLM caching cut inference cost?

Anthropic’s pricing shows cache reads at a tenth of base input token cost per their pricing page, and OpenAI offers half off cached input tokens per their guide. Combined with semantic caching short-circuiting, effective cost per request can drop dramatically on cache-friendly workloads.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides

Cross-links automatically generated from CodeIntel Log.