Prompt Caching in Production: Engineering Patterns Across Four Providers

A comparative engineering analysis of prompt caching across OpenAI, Anthropic, Google, and DeepSeek — provider mechanics, cache-aware architecture patterns, cost modeling, and empirical latency/cost data from production deployments.

Production EngineeringAI EngineeringLLM Serving

Prompt caching is the single highest-leverage cost and latency optimization available to production AI systems in 2026 — and the most fragmented one. Every major provider ships a different implementation with different cache semantics, discount structures, and invalidation models. Choosing the wrong caching strategy for your workload can mean leaving 50-90% of potential savings on the table.

This post maps the caching mechanics of the four dominant providers — OpenAI, Anthropic, Google Gemini, and DeepSeek — then derives three production engineering patterns that work regardless of which provider you use: static prefix separation, agent loop cache orchestration, and provider-arbitrated caching. We include real cost models and empirical performance data.

Provider Cache Mechanics

Each provider implements prompt caching differently. The fundamental tradeoff is between transparency (cache hits happen automatically, no code changes) and control (developers choose exactly what gets cached via explicit breakpoints).

OpenAI — Automatic Prefix Caching

OpenAI enables prompt caching by default on GPT-4o and all newer models, with no code changes required. The cache key is the request’s token prefix — any request whose first N tokens match a recent prior request will return cached KV computations for those tokens [1].

The pricing structure has evolved significantly:

Model Family Cache Write (per 1M tokens) Cache Read (per 1M tokens) Discount
GPT-4o / 4o-mini $3.75 / $0.164 $1.875 / $0.082 50%
GPT-5 family 1.25× base 0.1× base 90%
GPT-5.6 1.25× base 0.1× base 90%

Data from OpenAI’s prompt caching documentation and the Artificial Analysis caching comparison [1][2].

The cache TTL is implicit — OpenAI does not document a specific expiration window. In practice, cache persistence correlates with request volume on the same prefix: frequently-repeated prefixes stay cached for minutes to hours, while single-use prefixes expire quickly. Independent testing by Portkey found that cache hits persist for 5–8 minutes under continuous traffic, dropping to 60–90 seconds during low-traffic periods [3].

Key constraint: OpenAI caches the exact prefix. Any change to the system prompt, even a single character, invalidates the cache for all subsequent tokens. This is the single most important engineering implication — it dictates the cache-breakpoint architecture pattern below.

Anthropic — Explicit Cache Breakpoints

Anthropic requires explicit cache_control markers to define cache boundaries. You insert a {"type": "ephemeral", "cache_control": {"type": "ephemeral"}} entry in the messages array at the point where the cache segment ends. Everything before that marker is cached as one segment [4].

Key parameters:

  • Maximum breakpoints: 4 per request (more than 4 is a validation error)
  • Cache write cost: 1.25× the uncached input token rate
  • Cache read cost: 0.1× the uncached input token rate (90% discount)
  • Standard TTL: 5 minutes after last access
  • Extended TTL: 1 hour (available on some models, slight cost premium)

The breakpoint model gives Anthropic an important advantage for agentic workloads: you can cache the system prompt and tool definitions as one segment, and the conversation history as another, while placing the current turn’s user message outside the cache. If the user message is the only thing that changes, both cache segments remain valid across requests [5].

Google Gemini — Implicit + Explicit Context Caching

Google offers two caching mechanisms in parallel:

Implicit caching (enabled by default since May 2025, Gemini 2.5+): The provider automatically detects repeated prefixes and returns a 90% discount on cached input tokens. No code changes required. For Gemini 2.0 models, the discount is 75% [6].

Explicit Context Caching (via the Context Cache API): Developers create a named cache object with a configurable TTL (from minutes to hours). The cache is populated explicitly via a separate API call and referenced by cache name in subsequent generation requests. This gives deterministic control over cache lifetime and content but adds API complexity [7].

Cache Type Discount TTL Control Code Changes
Implicit (2.5+) 90% Automatic (opaque) None
Explicit (Context Cache API) 90% Configurable (minutes to hours) Cache creation + reference

Google’s implicit caching is the most transparent of the four providers — it requires zero developer effort — but the opaque invalidation model means you can’t predict whether a given request will hit or miss the cache.

DeepSeek — Context Caching on Disk

DeepSeek uses a fundamentally different approach: disk-based KV cache rather than in-memory caching. When a request prefix matches a previously processed prefix, the cached KV computation is loaded from disk rather than recomputed [8].

Key characteristics:

  • Enabled by default: No code changes, no cache_control markers
  • Cache mechanism: Disk-based storage of KV tensors, not in-memory
  • Pricing: Cache hits are billed at approximately 10% of uncached input cost (90% savings, exact discount varies by model variant)
  • TTL: Opaque but reported to be 30–60 minutes based on access patterns [9]

The disk-based approach is a competitive advantage for DeepSeek on costs (disk storage is cheaper than GPU memory) but may introduce higher latency on cache hits compared to in-memory approaches, depending on the storage backend and I/O bandwidth.

A June 2026 analysis on ofox.ai found that DeepSeek V3.2 achieves cache hit rates of 60–80% on typical chat workloads with repeated system prompts, without any developer effort [9].

Cost Model: What These Discounts Actually Mean

To make the comparison concrete, model a production agent workload: 10,000 requests/day, each with a 4,000-token system prompt + tool definitions, 1,000-token user input, and 500-token output. Assume 70% cache hit rate on the prefix (typical for agent systems where system prompts are static across user sessions).

Provider Uncached Daily Cost Cached Daily Cost Savings Annual Savings
OpenAI GPT-4o $15.60 $10.53 (50% cache read discount) 32.5% $1,851
OpenAI GPT-5 family $8.40 $2.10 (90% cache read discount) 75% $2,300
Anthropic Claude 3.5 Sonnet $12.60 $4.41 (90% cache read, 1.25× write) 65% $2,990
Google Gemini 2.5 Pro $5.40 $1.62 (90% implicit) 70% $1,381
DeepSeek V4 $1.20 $0.36 (90% disk cache hit) 70% $307

Assumptions: pricing from each provider’s published rates as of July 2026. Cache write overhead (1.25× for first request to populate cache) amortized across the 70% hit rate. DeepSeek pricing before caching is substantially lower, which is why absolute savings are lower despite similar discount percentages.

The key takeaway: Anthropic and Google offer the deepest marginal discounts on cached tokens, but OpenAI GPT-5 family’s 90% read discount combined with improved base pricing makes it competitive. DeepSeek’s absolute pricing is so low that caching savings, while proportionally large, produce smaller absolute dollar impact.

Engineering Pattern 1: Static Prefix Separation

The fundamental architectural pattern for maximizing cache hits is separating static from dynamic content at a cache boundary.

For OpenAI (which caches the exact prefix), this means every per-request variable must go after the static prefix. The safe structure is:

[System Prompt - static] [Tool Definitions - static]
[User Message - dynamic]

If you interleave dynamic content — even a user name or timestamp — anywhere in the first N tokens, you break the prefix match and lose the cache for the entire request.

For Anthropic, the pattern uses explicit breakpoints:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=2048,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,  # Long, static system prompt
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": CONVERSATION_HISTORY,  # Previous turns — cached
                    "cache_control": {"type": "ephemeral"}
                },
                {
                    "type": "text",
                    "text": CURRENT_USER_MESSAGE  # Not cached — unique per request
                }
            ]
        }
    ]
)

This achieves two cache segments: the system prompt and the conversation history. As long as the system prompt and prior turns are identical, only the current user message incurs KV computation — everything else is a cache read at 10% cost.

For Google Gemini, implicit caching handles this automatically if your static prefix is long enough and repeated frequently. The Gemini team recommends keeping static content at the beginning of the system instruction and placing dynamic content at the end to maximize implicit cache hits [7].

Engineering Pattern 2: Agent Loop Cache Orchestration

Agent loops are where prompt caching produces the most dramatic savings, because each iteration typically passes the entire conversation history back to the model. Without caching, a 10-turn agent loop would recompute KV for the full system prompt and all prior turns on every iteration.

The optimal strategy for Anthropic uses multiple cache breakpoints to segment the conversation:

import anthropic
from typing import List, Dict, Any

class CacheAwareAgent:
    """Agent loop that uses cache breakpoints for conversation history."""

    SYSTEM_CACHE = {
        "type": "text",
        "text": SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"}
    }

    def __init__(self):
        self.client = anthropic.Anthropic()
        self.conversation: List[Dict[str, Any]] = []

    def _build_messages(self, user_input: str) -> List[Dict[str, Any]]:
        """Build message list with cache breakpoints at history boundary."""
        if not self.conversation:
            return [{"role": "user", "content": [{"type": "text", "text": user_input}]}]

        return [{"role": "user", "content": [
            # Cache the entire prior conversation as one segment
            {
                "type": "text",
                "text": self._serialize_history(),
                "cache_control": {"type": "ephemeral"}
            },
            # Current turn — never cached, but small
            {"type": "text", "text": user_input}
        ]}]

    def step(self, user_input: str) -> str:
        messages = self._build_messages(user_input)
        response = self.client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=4096,
            system=[self.SYSTEM_CACHE],
            messages=messages
        )
        self.conversation.append({"role": "user", "content": user_input})
        self.conversation.append({"role": "assistant", "content": response.content})
        return response.content[0].text

For this pattern, the first turn (cold start) pays 1.25× for the system prompt write plus full KV computation for the user message. Turn 2 pays 0.1× for the system prompt read plus full KV for turn 1’s history write. By turn 5, the history write cost amortizes to roughly 20% of the per-turn cost of an uncached loop.

Empirical data from a May 2026 analysis by OfoxAI found that this pattern reduces per-turn costs by 55-75% on Anthropic Claude Sonnet, with the savings increasing with conversation length [10].

Engineering Pattern 3: Provider-Arbitrated Caching

For multi-provider systems, or for systems that need deterministic cache control beyond what any single provider offers, the provider-arbitrated pattern uses an external KV cache at the application layer and routes requests based on cache state:

from dataclasses import dataclass
from typing import Optional

@dataclass
class CacheState:
    prefix: str
    provider: str
    last_hit: float
    expiry: float

class ProviderCacheArbiter:
    """Routes requests to the provider-best-suited for current cache state."""

    def __init__(self):
        self.cache_registry: dict[str, CacheState] = {}

    def select_provider(self, system_prompt: str, user_input: str) -> str:
        prefix = system_prompt[:1024]  # Use first 1K as cache key

        if prefix in self.cache_registry:
            state = self.cache_registry[prefix]
            if time.monotonic() < state.expiry:
                return state.provider  # Route to warm provider
            else:
                del self.cache_registry[prefix]

        # Cold: route to cheapest provider for first write
        provider = "deepseek"  # Lowest absolute cost for cache writes
        self.cache_registry[prefix] = CacheState(
            prefix=prefix,
            provider=provider,
            last_hit=time.monotonic(),
            expiry=time.monotonic() + 300  # 5 min TTL
        )
        return provider

This pattern is useful in high-volume systems where cache state is a deployment concern: you want to route requests to whatever provider currently has a warm cache for that prefix, rather than paying cold-start cost on every provider simultaneously.

Note: this pattern requires careful monitoring. If you route to a different provider mid-session, the new provider has no cache for that prefix, and you pay full uncached cost for the first request. The pattern works best when traffic is large enough that individual sessions are statistically irrelevant — think background batch processing, not interactive chat.

When Prompt Caching Breaks

Prompt caching is not free. The engineering costs to consider:

  1. Cache invalidation tax on system prompt updates: Every time you update a cached system prompt, all warm caches across all providers become cold. For Anthropic, the 1.25× write cost for re-population applies to every request until the new prefix is cached. If you deploy system prompt updates multiple times per day, you may spend more on cache writes than you save on reads. A June 2026 study across 23 production deployments found that teams updating system prompts more than 3×/day had negative net caching ROI [11].

  2. Anthropic’s 4-breakpoint limit: Complex agent systems with multiple tools, few-shot examples, instruction hierarchies, and conversation history can easily exceed 4 breakpoints. Each additional cache segment requires careful design to fit within the limit.

  3. DeepSeek’s disk I/O latency: Disk-based KV cache retrieval can add 100-500ms to cache-hit requests compared to in-memory approaches. For latency-sensitive applications, this may offset the cost benefit [9].

  4. Provider lock-in: Once your architecture is optimized for one provider’s caching semantics (e.g., Anthropic’s explicit breakpoints), switching providers requires redesigning the cache orchestration layer. The provider-arbitrated pattern mitigates this but adds complexity.

Decision Framework

If your workload is… Use… Because…
Single-provider agent loops Anthropic (explicit breakpoints) Best cache segmentation for conversation history
High-throughput, simple schema OpenAI GPT-5 family (automatic) 90% read discount, zero code changes
Cost-sensitive, latency-flexible DeepSeek (disk cache) Lowest absolute pricing, 90% cache savings
Multi-model orchestration Google Gemini (implicit) Works across Google’s model family, no code changes
System prompts change weekly+ Any provider’s automatic caching Explicit breakpoints add overhead without benefit

Key Takeaways

  1. All four providers now offer 90% read discounts on cached tokens — the base economics are converging. The differentiator is control over cache segmentation (Anthropic > Google > OpenAI > DeepSeek).

  2. The static prefix separation pattern is universal — regardless of provider, placing all dynamic content at the end of your request maximizes cache hits. Every character of interleaved dynamic content is a broken cache.

  3. Agent loops benefit disproportionately from prompt caching, with 55-75% per-turn cost reductions on Anthropic and 50-70% on OpenAI when conversation history is properly segmented.

  4. Cache invalidation is the hidden cost — frequent system prompt updates (3+ per day) can erase caching ROI. Design your prompt update cadence around your provider’s cache TTL.

As provider pricing converges toward commodity token rates, prompt caching engineering — not model choice — is increasingly the variable that determines production AI costs.

References

[1] https://developers.openai.com/api/docs/guides/prompt-caching [2] https://artificialanalysis.ai/models/caching [3] https://portkey.ai/blog/openais-prompt-caching-a-deep-dive [4] https://platform.claude.com/docs/en/build-with-claude/prompt-caching [5] https://www.respan.ai/articles/claude-prompt-caching [6] https://developers.googleblog.com/gemini-2-5-models-now-support-implicit-caching/ [7] https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/context-cache/context-cache-overview [8] https://api-docs.deepseek.com/guides/kv_cache/ [9] https://ofox.ai/blog/deepseek-v3-2-prompt-caching-setup-2026/ [10] https://ofox.ai/blog/prompt-caching-cost-math-anthropic-vs-openai-2026/ [11] https://www.digitalapplied.com/blog/prompt-caching-2026-cut-llm-costs-engineering-guide

📖 Related Reads

  • ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
  • NiteAgent — AI agent development, frameworks, and production patterns

Cross-links automatically generated from CodeIntel Log.