LLM Inference Engines Benchmark: vLLM vs SGLang vs TensorRT-LLM

The vLLM vs SGLang vs TensorRT-LLM comparison is the most consequential decision for production LLM serving in 2026. Engine choice determines your token economics, latency SLOs, and hardware utilization. This post synthesizes the latest vendor-published benchmarks to give you a grounded picture of where each engine stands—and where vendor numbers mislead.

How This Was Researched

This analysis synthesizes vendor-published benchmark results from official sources; we did not run these workloads hands-on. We pulled live data from the vLLM blog, LMSYS/SGLang blog, NVIDIA Technical Blog, and the TensorRT-LLM GitHub repository on August 5, 2026. We excluded community benchmarks and anecdotal reports. We did not cover CPU-only serving or non-NVFP4 quantization. Last researched: August 2026.

Why Engine Choice Matters for Production

The serving engine is the control plane for your inference fleet—it sets throughput ceilings, latency tails, and accelerator efficiency, and the gap between engines is not cosmetic: vLLM’s V1 rewrite delivered up to 1.7x higher throughput than V0 on Llama-8B/H100, per vLLM’s V1 release post.

A 10% throughput gain on a 24-GPU cluster translates to roughly one extra GPU of free capacity. SGLang hit 500 TPS on an 8x B300 agentic workload SGLang GLM-5.2. Engine choice also shapes your team’s debugging surface, kernel compatibility, and ability to adopt speculative decoding or disaggregated serving.

vLLM V1: Throughput-First Architecture

vLLM V1’s throughput gains come from reduced CPU overhead, not new kernels—the kernels are nearly identical between V0 and V1, per the vLLM V1 announcement. The wins live in scheduling, batching, and memory management, and prefix caching costs less than 1% throughput even at a 0% cache hit rate.

In production, vLLM served GLM-5.2-NVFP4 on 24x B300 with a disaggregated 4-Prefill + 1-Decode topology, hitting mean TPOT of 17ms against a 20ms SLA vLLM GLM-5.2. The optimization journey from 40ms to 17ms TPOT shows that engine tuning is a multi-week effort, not a one-shot config.

SGLang: Latency-Optimized with Cross-Engine Kernels

SGLang’s headline result is over 500 tokens/s (bs=1) on 8x B300 for the GLM-5.2 NVFP4 agentic workload, reached in two weeks of optimization per the SGLang team’s post. Its edge comes from sync-free speculative decoding and trtllm-gen kernels for sparse attention and MoE.

The critical architectural detail is cross-engine convergence: SGLang now uses trtllm-gen kernels for both sparse attention and MoE, the same kernel-generation path as TensorRT-LLM. Removing GPU bubbles gave an 11% end-to-end TPS speedup, and IndexShare MTP + Spec V2 power the speculative path. For single-stream latency, SGLang is the current leader—but its advantage narrows at high batch sizes where vLLM’s scheduler shines.

TensorRT-LLM: Kernel-Level Control on NVIDIA

TensorRT-LLM remains the kernel-level performance ceiling, demonstrated by the world-record DeepSeek-R1 inference of over 30,000 tokens/s on the 671B-parameter model across eight NVIDIA Blackwell GPUs, per NVIDIA’s Technical Blog. The current release line is 1.3.0, with ongoing DeepSeek-V4/V3.2 Blackwell optimizations TensorRT-LLM GitHub.

However, this performance comes at a cost: TensorRT-LLM requires model compilation, graph optimization, and deeper NVIDIA-specific expertise. It is the right choice when you need absolute peak performance on a fixed model and can invest in the compilation pipeline.

Why Vendor Numbers Are Not Directly Comparable

The three engines’ headline numbers are not apples-to-apples: vLLM’s GLM-5.2 result used 24x B300 with disaggregated prefill/decode, SGLang’s used 8x B300 on agentic workload characteristics, and TensorRT-LLM’s used 8x Blackwell with DeepSeek-R1 vLLM GLM-5.2. Hardware counts, model sizes, quantization, and workload mixes all differ, so treat vendor numbers as directional evidence, not a ranking table.

Agentic traffic has different prefix-cache and speculative-decoding characteristics than bulk chat. Quantization (NVFP4 vs FP8 vs BF16) changes memory bandwidth and compute ratios. Even the metric definitions vary: TPS with bs=1 is not the same as throughput under load.

How to Benchmark Engines Yourself

Run your own load tests with LLMPerf-style harnesses, not ad-hoc curl loops, fixing hardware, model, and quantization across all engines—a single variable change invalidates the comparison, per the LLMPerf README. Measure TTFT, ITL, TPOT, and goodput, warm up for at least 100 requests before measuring, and report median and p95, not the mean alone.

Here is a minimal harness based on LLMPerf’s documented CLI:

# LLMPerf-style load test (schematic — flags per the LLMPerf README)
export OPENAI_API_BASE="http://localhost:8000/v1"
export OPENAI_API_KEY="your-key"

python token_benchmark_ray.py \
  --model "meta-llama/Llama-3.1-8B-Instruct" \
  --mean-input-tokens 1024 \
  --stddev-input-tokens 256 \
  --mean-output-tokens 256 \
  --stddev-output-tokens 64 \
  --max-num-completed-requests 500 \
  --timeout 600 \
  --num-concurrent-requests 32 \
  --results-dir "bench_results" \
  --llm-api openai

Run the identical config against each engine on identical hardware, then compare TTFT/TPOT p50/p95 and goodput.

Decision Guidance by Workload

For high-concurrency chat serving with mixed traffic, vLLM’s scheduler and prefix caching make it the safest default—the under-1% prefix-cache overhead is negligible insurance, per vLLM’s V1 release post. For agentic workloads with long single-stream generations, SGLang’s 500 TPS result and speculative decoding make it the latency leader SGLang GLM-5.2.

For fixed-model, peak-throughput scenarios on NVIDIA hardware, TensorRT-LLM’s compiled kernels deliver the raw ceiling NVIDIA Blackwell. Consider your team’s expertise: TensorRT-LLM demands NVIDIA-specific skills, while vLLM and SGLang are closer to drop-in replacements. For a deeper look at how engines integrate with your stack, see our AI stack reference and tools page. If you are adopting new models fast, our Kimi K3 day-0 vLLM integration analysis shows the operational cost of engine churn, and our speculative decoding library analysis covers the acceleration layer above engines.

How do vLLM, SGLang, and TensorRT-LLM compare for production LLM serving?

The three engines trade off throughput ceiling, latency optimization, and operational complexity: vLLM leads on scheduler efficiency with up to 1.7x V1 throughput over V0 vLLM V1, SGLang leads on single-stream latency with over 500 TPS on 8x B300 SGLang GLM-5.2, and TensorRT-LLM leads on raw kernel performance at over 30,000 tokens/s on DeepSeek-R1 NVIDIA Blackwell.

Your choice depends on whether you optimize for aggregate throughput, tail latency, or peak single-model performance, and on your team’s willingness to manage compilation pipelines.

FAQ

Which engine has the lowest latency for single-stream agentic workloads?

SGLang currently leads on single-stream latency, demonstrated by its 500+ tokens/s result on 8x B300 for GLM-5.2 NVFP4 agentic workloads SGLang GLM-5.2. Its sync-free speculative decoding and trtllm-gen kernels for sparse attention and MoE deliver this edge. vLLM’s disaggregated serving achieves 17ms TPOT but with a larger hardware footprint vLLM GLM-5.2.

Is TensorRT-LLM worth the extra engineering effort?

TensorRT-LLM is worth it when you need absolute peak throughput on a fixed model and have NVIDIA-specific expertise—its 30,000+ tokens/s on DeepSeek-R1 across eight Blackwell GPUs is unmatched NVIDIA Blackwell. The cost is model compilation and graph optimization, which slows iteration. For teams that change models frequently, vLLM or SGLang offer faster deployment cycles with acceptable performance tradeoffs.