Kimi K3 + vLLM: Inside the Day-0 Production Integration of a 2.8T Hybrid Recurrent MoE

The Bottom Line

Kimi K3 is not a larger K2. At 2.8T parameters with Kimi Delta Attention (KDA), Attention Residuals (AttnRes), 896 routed MoE experts, native MXFP4 weights, and a 1M-token context window, it represents a fundamentally different serving problem than any open-weight model before it. vLLM’s day-0 support — shipping the same day weights dropped — required a redesign of prefix caching for recurrent state, fused depth-wise attention residual kernels, custom sequence-parallelism kernels, and DSpark speculative decoding at 370 tok/s [1][2]. This post breaks down every major engineering decision in the integration.

The Release-Process Precedent

Moonshot AI announced Kimi K3 on July 16, 2026 and released weights on July 27 — an 11-day gap [1][3]. vLLM proposed this separation and Moonshot AI agreed. The reasoning: a frontier-model announcement carries last-mile uncertainty around product stabilization and commercial rollout. Tying open-source release to the same deadline forces inference engines to chase a moving target. Separating announcement from artifact release gives the model vendor room to freeze its checkpoint while the inference team gets a stable integration window for correctness testing, performance tuning, and recipe validation [1].

Kimi Delta Attention: When Prefix Caching Meets Recurrent State

KDA is a linear-time attention mechanism that maintains a fixed-size matrix-like recurrent state per layer, combined with a short causal convolution state, instead of materializing a per-token KV pair [3]. Roughly 3 out of every 4 layers in K3 use KDA, interleaved with periodic full-attention (MLA) layers for exact global recall. This hybrid design is what makes a 1M-token context window feasible — but it breaks nearly every assumption in conventional prefix caching.

The Cache-Mismatch Problem

Conventional prefix caching hashes token blocks into paged KV. KDA is fundamentally different: its recurrent state updates at every token, but storing a snapshot at every boundary is prohibitive — each KDA state is orders of magnitude larger than one token’s KV entry. Prior implementations used large state blocks (thousands of tokens) to amortize storage, forcing prefix-match granularity to the same alignment. Two requests sharing almost their entire prompt could still miss a cache hit because their common boundary fell mid-block.

Separating Physical Blocks, Scheduler Alignment, and Prefix Matching

vLLM’s solution decouples three concepts that previously moved together: physical block size (KDA state and KV allocation), scheduler alignment (where execution must stop for cache consistency), and prefix-match unit (the finer token interval at which shared prefixes are hashed and matched) [1]. This allows registering a valid KDA state at a fine-grained boundary inside a larger physical block. When a later request hits that partial block, the cached state is copied into a private destination via copy-on-write before extension — preserving the shared prefix while allowing safe continued generation.

The implementation handles subtle invariants: scheduler boundaries aligned to both block and hash boundaries, unified num_computed_tokens across KDA and full-attention groups, chained fine-grained hashes, deferred same-step reuse, and cross-worker cache transfer [1][2]. This is now core vLLM infrastructure for any hybrid linear attention model.

Interval-Based and Marconi-Style Retention

Even with fine-grained matching, caching every KDA state is wasteful. vLLM supports two policies [2]: interval-based retention (checkpoints at configurable intervals, e.g., every 32K tokens, with prompt boundaries always retained) and Marconi-style selective retention (cache on the second hit — the first observation shows the prefix exists, the second confirms it’s shared). Controlled via VLLM_PREFIX_CACHE_RETENTION_INTERVAL, these policies cover both predictable and emergent reuse patterns. The scheduler compares reusable token lengths from local GPU and external stores (e.g., Mooncake), selecting the longer prefix and reconciling cache groups when a remote hit wins.

Attention Residuals: Fused Depth-Wise Mixing

Kimi K3’s Attention Residuals (AttnRes) replace ordinary residual accumulation with depth-wise attention across preceding layer blocks [3]. Each Transformer sublayer uses a learned pseudo-query to weight RMS-normalized residual states from up to eight prior blocks, then receives the weighted combination as its input. This is not the standard residual stream — it’s a learned retrieval across depth.

The Kernel Challenge

Naive AttnRes means: for every token in every KDA layer, read N prior residual states, compute logits, softmax, aggregate, normalize — a crippling chain of small launches across 93 layers. vLLM’s solution fuses depth-wise logit computation, online softmax (across depth rather than sequence), weighted aggregation, residual update, and optional output RMSNorm into a single kernel pass [1][2]. A specialized CUDA kernel covers Blackwell; a portable Triton fallback covers other configurations [2].

Sequence Parallelism for TEP Prefill

AttnRes also drove a TEP prefill redesign [2]. Naive TEP performs two all-reduces per layer (attention output, then MoE), forcing every rank to materialize the full batch and redundantly apply AttnRes. vLLM replaces this with sequence parallelism: the all-reduce after o_proj becomes a reduce-scatter — each rank owns a token shard, applies AttnRes per shard, runs MoE dispatch/combine, and a single all-gather restores the full batch. Since NCCL’s reduce-scatter/all-gather are not optimized for prefill message sizes, vLLM implemented custom kernels that are 1.7×–4.5× faster than NCCL [2].

MXFP4 MoE: Native 4-Bit at 896-Expert Scale

Kimi K3 uses MXFP4 weights (4-bit) with FP8 activations, baked in via quantization-aware training [3][4]. Stable LatentMoE dispatches tokens to 16 of 896 experts (~1.8% sparsity), projecting dispatched activations into a narrower latent dimension before expert computation to reduce weight bandwidth and all-to-all traffic [2][3].

K3’s SiTU activation was not supported in prior MXFP4 TRTLLM-Gen paths. vLLM mapped SiTU parameters into the optimized FP4 expert path with safe workload chunking for large token-by-top-k launch grids, validated on 16-GPU DP16+EP16 [1]. On AMD, FlyDSL’s MLIR kernel stack provides hardware-tuned A16W4/A8W4 fused operators with SiTU [1]. For production, use deep_gemm_mega_moe for DEP and flashinfer_trtllm for TP > 1 [2].

Separate Prefill/Decode Code Paths

Kimi K3 uses MLA attention every four layers. Previous vLLM releases relied on torch.compile custom-fusion for MLA, which slowed startup and left kernels unfused. The K3 integration implements a hand-fused MLA module with two explicitly different code paths [1]: the prefill path fuses gate-projection elementwise ops into the projection epilogue, while the decode path uses multi-stream support to run gate projection in parallel with attention. The fused KDA decode kernel folds causal convolution, recurrent state update, gating, and RMSNorm into a single launch — critical for minimizing per-layer launch overhead across K3’s many KDA layers [1][2].

During DSpark bring-up, KDA metadata preparation was identified as a bottleneck: the generic GDN builder prepared unused FLA metadata and used sequences of small eager PyTorch ops. vLLM built a dedicated K3 KDA metadata builder using fused Triton kernels, reducing metadata-preparation latency from 870 μs to 34 μs (96% reduction) at batch size 1, improving end-to-end DSpark latency by 6% [2].

DSpark Speculative Decoding: 3.14× Throughput

vLLM supports DSpark — a block-diffusion speculative decoding algorithm — from day 0 for K3, with Inferact training and open-sourcing a dedicated draft model [2]. The draft generates multiple speculative tokens in one parallel pass from K3’s intermediate states, with a low-rank Markov head for intra-block dependency and a confidence head for acceptance prediction. The draft is MLA-native, mirroring K3’s own attention layout for KV-cache compatibility. On GB300 NVL72 (16 GPUs), vLLM achieves 118 tok/s without speculation and 370 tok/s with DSpark (3.14×), with ~4.73 accepted tokens per step on coding workloads and ~2.61 on creative writing [2]. The draft model is open-sourced at HuggingFace.

Agentic Serving: Cache Retention Across Turns

KDA’s constant-size state (not growing with sequence length) is memory-efficient for long contexts but complicates multi-turn reuse. vLLM’s interval-based and Marconi-style retention policies (described above) cover both predictable (prompt-end) and emergent (second-hit promotion) reuse patterns, with the scheduler comparing reusable token lengths from local GPU and external stores (e.g., Mooncake) and selecting the longer prefix [2].

Deployment Tips

Based on vLLM’s validated recipes [2]: Minimum hardware is 8× B300 or 16× B200 GPUs. Prefix caching is not enabled by default for K3 — pass --enable-prefix-caching. For MoE backend, use deep_gemm_mega_moe for DEP and flashinfer_trtllm for TP > 1. For all-to-all, use flashinfer_nvlink_one_sided for NVLink and deepep_v2 for RDMA. Tool calling should be validated on your own traffic — K3 can emit formats its parser doesn’t expect; consider XGrammar-constrained structured calling for production. Set VLLM_USE_RUST_FRONTEND=1 for the Rust frontend, which fully supports this model [2].

Sources

[1] vLLM Team, “A Preview of Production-Scale Kimi K3 Support on vLLM,” vLLM Blog, Jul. 22, 2026. https://vllm.ai/blog/2026-07-22-kimi-k3-preview

[2] vLLM Team and Inferact, “Kimi K3 Is Here: Efficient Day-0 Support on vLLM,” vLLM Blog, Jul. 27, 2026. https://vllm.ai/blog/2026-07-27-k3

[3] Moonshot AI, “Kimi K3 Tech Blog: Open Frontier Intelligence,” Jul. 16, 2026. https://www.kimi.com/blog/kimi-k3

[4] Moonshot AI, “moonshotai/Kimi-K3,” Hugging Face Model Hub, Jul. 27, 2026. https://huggingface.co/moonshotai/Kimi-K3

ToolBrain — tool reviews, LLM comparisons, and AI workflow guides

Cross-links automatically generated from CodeIntel Log.