Why Dynamic Batch Tuning Failed to Beat Static Scheduling: An Honest Failure Analysis of a Multi-Tenant vLLM Serving Control Loop

If you operate a vLLM serving cluster on Kubernetes where multiple tenants share high-end GPUs such as the H200, and you have assumed that Kueue’s static admission control is “good enough,” this post is for you. It is even more worth reading if you have been assuming that “an LLM agent adjusting serving parameters in real time is always better.” This paper shows that assumption is not always true, and it does so with quite concrete reasons.

The Problem: Admission Is Static, but Load Is Real-Time

Serving LLM inference for multiple tenants on shared GPUs has become a common operational challenge. A single high-capacity accelerator such as the H200 hosts several tenants at once, each with different traffic burstiness, sequence lengths, and SLOs. On Kubernetes-based platforms, a quota- and queue-based scheduler such as Kueue typically decides which jobs are admitted into the pool, but the serving-time parameters that determine how already-admitted requests actually share the engine’s finite capacity, such as batch size and the maximum number of concurrent sequences, are usually fixed statically at deployment time.

The paper raises the question that naturally follows from this setup. In a multi-tenant H200 GPU pool managed by Kueue, when multiple tenants share a vLLM inference server, can an LLM agent that observes real-time GPU memory and latency telemetry and re-tunes each tenant’s batch size and maximum concurrent sequence count online empirically improve the throughput, p99 latency, and cost Pareto frontier over static Kueue admission control? To answer this question, the authors designed a complete empirical protocol meant to run on a single H200, but they state up front that execution itself was called off because access to the target Kubernetes cluster context was unavailable. As a result, not a single throughput or latency number measured on real hardware appears anywhere in this paper. Instead, the paper delivers three things: a control-loop formalization of the online batch and concurrency tuning problem, a reproducible protocol that can be executed the moment cluster access is restored, and a deterministic queuing simulation that stress-tests the structure of three control policies.

Formalizing the Tuning Problem as a Control Loop

The paper formalizes online per-tenant batch and concurrency tuning as a discrete-time closed-loop control problem with a structure similar to a Markov decision process. The state consists of per-tenant queue depth, GPU memory utilization, recent latency percentiles (p50/p95/p99), and the current concurrency cap. The action is a bounded-width per-tenant increment or decrement to concurrency. The reward function subtracts a penalty for p99 latency that exceeds the SLO and a cost term from throughput, and is designed to jointly optimize throughput, latency, and cost within a safe range. Building on this formalization, the authors also propose how the agent would integrate into an actual deployment. The agent does not replace Kueue’s job admission; it operates as a sidecar inside it, adjusting only how already-admitted jobs divide the engine’s capacity. The key point is that the lever being pulled is not the vLLM engine’s internal max_num_seqs, but client-side per-tenant admission concurrency. Production vLLM does not expose engine parameters as a knob that can be changed in real time without a restart, so this design reflects the practical constraint that the only point actually adjustable is the number of concurrent requests arriving at the server.

The control-loop structure in which an LLM agent observes GPU telemetry and per-tenant queue depth, produces a bounded-width concurrency adjustment, and thereby re-tunes the batch configuration. This is a conceptual architecture diagram, not a result measured on hardware.

The Warning the Simulation Raised: Naive Dynamic Control Was Worse Than Static

To fill the gap left by the unexecuted empirical protocol, the authors built a seed-fixed, discrete-time queuing simulation written entirely with Python’s standard library. On a simplified model where two tenants share a finite pool of slots, they ran three policies for 20 seeds of 1,800 seconds each and averaged the results. The static policy fixes each tenant’s concurrency cap at 4. The naive dynamic policy observes GPU memory utilization and 6-second-window p99 latency every 3 seconds and adjusts both tenants together by plus or minus 1. The differentiated agent policy is a surrogate model that mimics more sophisticated agent reasoning by independently reflecting each tenant’s backlog ratio.

The results defied expectations. The static policy was best on both throughput (0.686 req/s) and drop count (an average of 1.9 requests), while the naive dynamic policy’s throughput fell to 0.515 req/s and it dropped an average of 309.5 requests. The differentiated agent surrogate model did better than the naive dynamic policy (throughput 0.601 req/s, 154.7 drops) but still could not catch up to the static policy, and p99 latency was also higher for both dynamic variants than for the static policy (11.75 seconds), at 12.79 and 13.25 seconds respectively.

The static policy recorded the highest throughput and fewest drops, while both dynamic variants underperformed in the simulation. These are results from a deterministic queuing simulation averaged over 20 seeds, not values measured on an actual GPU.

The static policy had the lowest p99 latency, and neither dynamic controller reduced tail latency in this simulation regime. These are simulation results averaged over 20 seeds, not figures measured on hardware.

The authors verify that this result is the model’s actual dynamics rather than a simple bug, and trace the cause through four steps. First, because service time follows an exponential distribution, the tail is inherently heavy. An exponential distribution with a mean of 2.5 seconds forms a p99 around 11.5 seconds even with zero queuing at all, which is barely different from the 11.75 seconds the static policy recorded. In other words, most of the observed tail latency comes not from congestion but from the variance of the service time itself. Second, the throttle threshold is set at 6 seconds, twice the 3-second window, which is far below this intrinsic p99 of 11.5 seconds. The short-window p99, estimated from only a handful of completed requests, is noisy and biased toward this intrinsic tail, so it frequently crosses the threshold even under genuinely light load. Third, because the resulting false-positive throttles lower both tenants together, even a non-congested tenant gets restricted along with the congested one, and under a 3-second hard timeout, every unnecessary throttle converts directly into a drop. Fourth, while the scaling rule does allow recovery, the throttle-drop cost is asymmetrically larger than the scaling gain, so net throughput falls.

Design Requirements Drawn from the Failure

What this diagnosis tells us is not that dynamic tuning itself is useless. It is that when a noisy, tail-biased signal, throttling that couples tenants together, and a hard timeout combine, naive fixed-threshold control can actually turn out worse than static control. From here, the authors derive four design requirements that an effective agent controller must satisfy. The congestion signal must be robustly estimated from longer telemetry history and confidence estimates rather than raw short-window percentiles. Decisions must be differentiated per tenant rather than driven by a global signal that lumps tenants together. The asymmetric cost between throttling and scaling must be explicitly accounted for, and additional context such as burst-pattern recognition or per-tenant SLA metadata, which cannot be expressed with a fixed numeric threshold, must also be leveraged. The fact that the differentiated agent surrogate model actually recovered roughly half the loss relative to the naive controller suggests this direction may well be valid. However, the authors are clear that because both the coupling and the signal type were changed at the same time, this experiment alone cannot isolate which of the two contributed to the recovery.

What This Leaves for the Company, Society, and Science

For ThakiCloud, what this research leaves us right now is not a verified cost saving. It is a reproducible empirical protocol directly applicable to our multi-tenant H200 cluster, and a concrete warning that a naive approach could actually make things worse. The harness itself is already complete, so that once cluster access is restored, all that needs to be substituted into this protocol is an actual LLM agent decision function. More broadly, as methodologies for reducing resource waste in shared GPU pools are refined, a practical path opens up for smaller organizations to also meet SLAs on shared clusters. This benefits the overall energy and cost efficiency of inference infrastructure. Scientifically, the contribution is that it explicitly formalizes the control loop in which an LLM agent observes real-time system telemetry and re-tunes serving hyperparameters online, and quantitatively diagnoses its failure modes through a reproducible simulation. This paper is also part of the research lineage ThakiCloud has continued on the same Kueue-and-GPU foundation. Compared with ABJ-Gate, which schedules the judge model’s sampling budget, Attested Confidential Sovereign Inference, which handles TEE remote attestation at admission time, and “Escalate or Act?”, which handles binary escalate-or-act decisions for rare discrete accidents, this paper is distinguished from them in that it tackles a qualitatively different problem: real-time, multi-objective control that is continuously re-tuned on a second-by-second cadence.

Limitations

The limitations this paper itself discloses are clear. The most important is that there is no real-hardware validation of any kind: the empirical protocol was never executed because access to the target cluster context was unavailable, and not a single number in the paper was measured on a GPU. The simulation simplifies tokens and the KV cache down to a single scalar slot count and does not model prefill/decode or memory-pressure dynamics, and even the memory-utilization signal is merely a proxy for slot occupancy rather than an actual device-memory measurement. The tenant types tested are also limited to two synthetic traffic patterns, so the failure modes discovered may not generalize to other traffic mixes. The agent policy inside the simulation is likewise a hand-written surrogate model rather than an actual LLM, and because both the coupling and the signal type were changed at once, there is as yet no verified evidence that an actual LLM agent would satisfy the design requirements derived in Section 4. The authors also state plainly that they reported only the mean and standard deviation across 20 seeds and did not run any test of statistical significance. For these reasons, this paper positions itself not as a validated result but as a formalization, a protocol, and a cautionary simulation.

You can find the paper’s detail page here: https://huggingface.co/datasets/thaki-AI/daily-paper-2026-07-21-agent-dynamic-batch-tuning-vllm