LLM Traffic Prioritization: Scheduler-Level QoS

Interactive chat calls and batch backfills sharing one GPU fleet collide in the serving scheduler, not the API gateway. Prioritizing interactive vs batch LLM traffic requires five QoS levers: traffic-class separation, queue policy, preemption and admission control, batching delay tuning, and prefill/decode disaggregation. Each lever operates at a different layer of the inference stack, and choosing among them depends on your workload mix and latency SLOs.

How do you prioritize interactive LLM traffic over batch workloads?

Prioritize at the scheduler by separating traffic classes, applying priority queue ordering, and gating admission under KV-cache pressure. The five levers are traffic-class separation, queue policy, preemption and admission control, batching delay knobs, and prefill/decode disaggregation. Each maps to concrete configuration in vLLM or Triton.

The failure mode of treating this as a gateway problem is that batch jobs already past the gateway consume KV-cache blocks and decode slots inside the engine. our LLM API rate-limiting gateway architecture handles request volume, not scheduler-level resource contention. Rate limiting rejects excess traffic; it does not reorder the requests that were admitted.

How This Was Researched

This post is based on documentation and source-code analysis plus published benchmark results — no hands-on testing was performed. Sources consulted: official OpenAI and Anthropic batch API documentation; vLLM v1 scheduler source code and architecture docs; NVIDIA Triton Inference Server rate-limiter, model-configuration, and batcher documentation; and two peer-reviewed papers — PARS (arXiv 2510.03243, ISC High Performance 2026) and EcoServe (USENIX OSDI 2026). What was NOT covered: cloud-managed serving platforms’ internal scheduling (e.g., Azure OpenAI, AWS Bedrock), non-GPU inference stacks, or client-side API gateway rate limiting (covered in our LLM API rate-limiting gateway architecture). Last researched: September 2026.

Traffic-Class Separation: Separate Endpoints, Separate Pricing

Separate endpoints with different pricing and rate-limit pools are the first isolation layer. According to OpenAI’s Batch API docs, batch requests receive a 50% cost discount versus synchronous APIs, draw from a separate higher rate-limit pool, and complete within a 24-hour window. Anthropic’s documentation states Message Batches offer a 50% cost reduction, most batches finish in under 1 hour, expire after 24 hours, and cap at 100,000 requests or 256 MB per batch.

This pattern works because batch traffic accepts hours of delay, so the provider can schedule it in idle capacity. Self-hosted fleets can mirror this by routing batch jobs to separate engine instances or by tagging requests for scheduler-level priority — but the pricing lever is a vendor construct. On your own GPU fleet, the equivalent is capacity planning: reserve a fraction of instances for batch and accept that they idle during low load.

Queue Policy: FCFS, Priority, and Shortest-Job-First

Queue policy determines request ordering within a single engine. vLLM’s v1 request queue supports two policies: FCFS using a deque, and PRIORITY using a heap ordered by (priority, arrival_time) where lower priority numbers process first. The vLLM GitHub README lists continuous batching, chunked prefill, prefix caching, and PagedAttention KV-cache management as core features that make these queue policies effective.

FCFS head-of-line blocking causes starvation when short interactive requests wait behind long batch generations. The PARS paper (ISC High Performance 2026) demonstrates this: ranking requests by predicted response length approximates shortest-job-first and reports up to 15.7× lower latency versus vLLM’s default on chat, math, and code workloads. The vLLM v1 architecture runs one engine-core process per data-parallel rank that executes the scheduler in a continuous busy loop, as described in the vLLM architecture overview — so queue policy decisions happen at nanosecond scale per scheduling step.

# vLLM v1 PriorityQueue ordering (conceptual, from request_queue.py)
# Lower priority number = processed first. Ties broken by arrival time.
def pop(self):
    if self.policy == SchedulingPolicy.FCFS:
        return self.deque.popleft()
    elif self.policy == SchedulingPolicy.PRIORITY:
        priority, arrival_time, seq = heapq.heappop(self.heap)
        return seq

Preemption and Admission Control Under KV-Cache Pressure

When KV-cache block allocation fails, the scheduler must reclaim memory from running requests. vLLM’s v1 scheduler preempts running requests, choosing the victim by priority ordering under the PRIORITY policy; preempted requests are reset and requeued. The same source caps concurrency via max_num_seqs and per-step tokens via max_num_batched_tokens (with max_num_scheduled_tokens as fallback).

Two structural failure modes emerge. Starvation occurs when a constant stream of high-priority requests preempts low-priority batch work indefinitely — the batch job never completes. Preemption thrash occurs when preempted requests are requeued and immediately re-admitted, consuming scheduler cycles without making progress. Admission control via max_num_seqs prevents the first condition by bounding how many requests compete for KV-cache blocks. The second requires careful priority tuning: if interactive and batch priorities are too close, the heap ordering produces frequent preemption.

Batching Delay Knobs and Cross-Model Fair Share

Triton offers different mechanisms at the model-instance level. The Triton dynamic batcher documentation explains that a configurable maximum delay time lets requests join a batch; the tuning guide recommends raising max batch size or setting non-zero batch delay to trade increased latency for increased throughput. This is a latency-for-throughput knob: interactive requests waiting for batch fill incur added TTFT.

The Triton rate limiter documentation describes a different mechanism: with rate limiting off, requests schedule when a model instance is available; with it on, instances declare scarce resources (e.g., memory) and model priorities decide execution order under contention. Triton’s model configuration docs state model priority is a fair-share weighting: “an instance with priority 2 will be given 1/2 the number of scheduling chances as an instance with priority 1.” This is cross-model fair share, not per-request preemption — do not conflate the two.

# Triton model config: dynamic batching with delay (documented fields only)
name: "interactive-llm"
max_batch_size: 64
dynamic_batching:
  max_queue_delay_microseconds: 500
  preferred_batch_size: [32, 64]

Prefill/Decode Disaggregation as Latency Isolation

PD disaggregation separates prefill and decode onto different instances, breaking the coupling between TTFT and TPOT. vLLM’s experimental disaggregated prefilling documentation states prefill and decode run on different instances so operators can assign different parallel strategies to tune TTFT without affecting inter-token latency (ITL). The EcoServe paper (USENIX OSDI 2026) formalizes this: prefill and decode have different latency SLOs (TTFT vs. TPOT) forming a performance trade-off triangle with throughput, and time-dimension separation yields 1.96–2.51× goodput gains over vLLM/Sarathi/DistServe/MoonCake on a 32-GPU cluster.

This is the strongest isolation lever but the most operationally complex — it requires two instance pools, a router between them, and careful capacity planning. For mixed traffic, it decouples batch prefill bursts from interactive decode streams. For production streaming considerations, see our production streaming architecture patterns, which covers token-level latency management. The AI stack reference catalogs the components involved.

Decision Framework: Workload Mix → Pattern

Shared engine + priority queue. Use when interactive traffic is a small fraction of total requests and batch jobs tolerate preemption. Configure vLLM’s PRIORITY policy with a wide priority gap (e.g., interactive at -10, batch at 0) to minimize preemption thrash. Failure mode: starvation of batch work under sustained interactive load.

Separate GPU pools per class. Use when both classes have strict completion requirements — batch jobs must finish, interactive calls must stay fast. Dedicate instances per class and size each independently. Failure mode: stranded capacity when one pool idles while the other queues; also the most expensive option. This mirrors the OpenAI/Anthropic batch pattern at the infrastructure level.

PD disaggregation. Use when TTFT and TPOT SLOs are both strict and workload is large enough to justify two pools. Prefill instances absorb batch bursts; decode instances maintain steady TPOT. Failure mode: router overhead and instance-pool coordination complexity; prefill pool becomes the bottleneck if under-provisioned.

FAQ

Can I set per-request priority through the vLLM OpenAI-compatible API?

No. vLLM’s request.py shows requests carry a per-request integer priority field (default 0), but this is engine-internal. No built-in OpenAI-compatible priority parameter is documented in the cited sources. Mapping priority to an API entrypoint requires custom middleware that injects the field into engine requests before scheduling.

Is separate GPU hardware always required to isolate interactive traffic?

No. Priority queueing and admission control within a single engine provide partial isolation, and PD disaggregation can run on the same physical GPUs partitioned into logical pools. Separate hardware guarantees isolation but strands capacity. The right choice depends on whether your batch workload can tolerate preemption and requeueing without violating its own completion SLO.

How does KV-cache preemption differ from request-level rate limiting?

Rate limiting rejects requests before they enter the engine, protecting the gateway and bounding queue depth. KV-cache preemption reclaims memory from requests already admitted and running, protecting the engine’s memory footprint. Rate limiting cannot reorder admitted requests; preemption cannot reject new traffic. our multi-provider failover architecture and our LLM response-caching architecture at scale address the gateway layer; the AI tools reference lists scheduler-level tooling.

The decision framework reduces to your batch workload’s tolerance for preemption and your interactive workload’s strictness on TTFT versus TPOT. Priority queueing handles moderate mixes; separate pools handle strict SLOs on both sides; PD disaggregation buys the last increment of latency isolation at the cost of operational complexity.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides

Cross-links automatically generated from CodeIntel Log.