, not concurrency

max_num_batched_tokens is the per-iteration token budget: the most tokens the scheduler may place in one forward pass across every request in the batch. It is not a per-request sequence limit. Dividing it by concurrency gives the tokens available per in-flight request, which is the quantity the error tracks.

, only correct order

Microseconds

Tensor parallelism has to divide both the attention head count and the embedding width, pipeline depth has to divide the layer count, the configuration has to fit inside one node, and GPUs per replica times replica count has to stay within the GPU budget you are willing to provision. Integer arithmetic on the model shape, so effectively free. Removes 3,600 of 11,520, leaving 7,920. One consequence surfaces later: when tensor parallelism exceeds the number of key/value heads, those heads are replicated rather than split, so past that point the KV cache stops shrinking as you add GPUs.

, cheapest first

Figure 4. The seven sieves on the 11,520-configuration grid from the table above. A solid bar means the set shrank at that stage; a dashed outline means the same set with more work done, which is why stages 3 and 5 repeat a count. The cost hierarchy is the point: stages 1 and 2 are arithmetic and run on all 11,520; stage 4’s analytical estimate is cheap enough to run on all 7,840 survivors;

Instead we filter in stages, cheapest first. The early stages are nearly free and run on every configuration, dropping the ones that can’t work. Only the 20 selected candidates reach the detailed simulation. The early stages are set algebra over model and device facts, so they are orders of magnitude cheaper per candidate than a scheduler replay; we report the candidate counts each stage admits and rejects, not per-stage timings.

The most detailed simulator replays the serving engine’s scheduler step by step, and even at a few seconds per configuration that would still take hours across the full grid.

than record

Figure 3. The same five configurations measured on one workload, so the levers read as numbers rather than directions. Scaling replicas dp=1 to dp=8 - 8 GPUs to 64 - multiplies decode throughput by 5.7, not 8. On a fixed 8 GPUs, changing only the parallelism to tp=4/pp=2 gives 1.85×. TTFT here is dominated by queueing at 5,000 arrivals, so read the fall from ~956 s to ~117 s as backlog draining, not as model latency.

The knobs that decide it Each knob trades one outcome against another. The direction, the mechanism, and where the direction stops holding: Tensor parallelism, tensor_parallel_size. Each GPU holds a shard of every layer, so weight bytes, KV-cache bytes and arithmetic per GPU per token all fall roughly as 1/TP - and because decode is bandwidth-bound, less to stream means a shorter step. That benefit has a ceiling. Two all-reduces run per layer, one after attention and one after the MLP; the bytes each GPU sends level off as TP grows, but the number of synchronisation steps does not, and the cost jumps whenever the group stops fitting inside one fast interconnect domain. KV sharding also stops at the model’s KV-head count. Meanwhile total GPUs = tensor parallelism × pipeline parallelism × replicas, so at a fixed GPU count raising TP costs you replicas. Figure 3 shows TP=8 sitting past the useful point for this model: against TP=4 with PP=2 on the same 8 GPUs it delivers 46% less throughput, at 48 ms p99 TPOT against 47 ms - no latency gain at all. That comparison moves PP as well, so it bounds the benefit of higher TP rather than isolating it - but it is enough to say there is no single direction here. TP has a useful range, and 8 is above it. Replicas, data_parallel_size. Each replica is a full copy of the model with its own GPUs, so aggregate throughput rises with replica count - sublinearly, and only while arrivals keep every replica batched. Nothing pools across replicas: weights are duplicated per replica, the prefix cache is per-replica so a 70% hit rate inside one replica does not become 70% across eight, and a request routed to a busy replica cannot borrow a quiet one’s free KV blocks. In Figure 3, dp=1 to dp=8 gives 5.7×, not 8×. Every configuration in that sweep is backlogged - average TTFT is 117 seconds even at dp=8 - so it is a saturated-throughput comparison rather than a latency one, and the gap against linear scaling is not attributable to any single one of those causes from this data alone. Batch size, capped by the token budget, by max_num_seqs, and by free KV blocks - whichever binds first. Larger batches raise decode throughput because the weight bytes streamed per iteration are amortised across more tokens. While weight bytes dominate, step time is nearly flat as the batch grows, so that throughput is close to free. It stops being free in two ways. Once the live KV cache outweighs the weights - the reasoning case above, at 1.5 GB per request - every added request adds bytes to stream and step time grows with batch size directly. And once kernels turn compute-bound, step time grows with arithmetic instead. Either way TPOT is what pays. Net: throughput up, TPOT up, cheap only in the weight-dominated regime. Precision and KV-cache format. FP8 weights instead of BF16 halve weight bytes, which frees HBM for KV cache and cuts what decode must stream, relieving capacity and bandwidth together. An FP8 KV cache does the same to the cache itself. What it costs is output quality, and not equally: quantising the KV cache degrades more than quantising weights at the same width, because the cache is read back as attention state for the rest of the request. Kernel support also varies by GPU, so the gain does not carry across the hardware axis of the grid.

very request runs prefill once, then decode in a loop. Each panel shows what those phases do to one resource, drawn to scale. Coding fills the token budget: prefill is 96% of the work, and a prompt too large for one pass is split into chunks that then compete with decode. Chat fills the latency budget: 48 ms of a 50 ms target, so the batch stops growing while HBM sits largely unused. Reasoning fills HBM, and the two mirrored bars are the mechanism - cache per request grows 6× as the request runs, so requests that fit fall 6×, from 295 to 49, while they are still running.

A coding assistant sends huge prompts - the whole file, the repo context - and gets short edits back. At ISL ~8,000 and OSL ~300, prefill is 96% of the token work. What binds is the token budget: a prompt larger than one forward pass can hold is admitted in chunks, one chunk per iteration - 2 chunks at a 6,144-token budget, 12 at 682. Each of those iterations spends most of its budget on prefill and has little left for decode, so decode tokens per iteration drop and aggregate decode throughput falls for the whole batch while that prompt is being read. Every request already decoding waits longer for its next token, so TPOT rises for all of them. The chunked request needs 12 iterations before its first output token instead of 2, so its TTFT rises with the chunk count. A chatbot sends short prompts and needs a steady stream of words back. Prefill is over quickly, so the request is almost entirely decode, and one iteration advances every active request by exactly one token - so TPOT is iteration time. Admitting one more request lengthens that iteration, because the GPU must compute one more token and stream one more request’s KV cache through HBM within the same forward pass. With a 50 ms per-token SLO against the ~48 ms p99 TPOT measured for this model on an H100 node, 2 ms of margin remain: the next admission consumes it and the one after breaches the SLO. So the SLO caps batch size, and because throughput scales with batch size it caps tokens per second per GPU. Serving the same traffic then takes more replicas - you buy GPUs to hold a latency line rather than to do work. Memory is not the constraint here: each request holds ~125 MB of KV cache, so 72 GB would hold roughly 590 concurrent requests, far more than the latency target allows. A reasoning or agent workload writes far more than it reads - long chains of thought, thousands of output tokens. Every one appends to the KV cache, so a request that begins with 2,000 tokens of cache ends holding 12,000: 250 MB becomes 1.5 GB. Capacity: the same 72 GB holds 295 of these requests at the start but 49 once grown, so as the live requests age fewer new ones fit, the batch drains faster than it refills, and aggregate throughput decays over the run. Bandwidth: decode re-reads the entire live cache every iteration, so bytes streamed per iteration grow as the caches grow and TPOT rises as requests age.

dp=1 to dp=8 buys 5.7× the throughput for 8× the GPUs

same expensive question

Figure 1. Too few GPUs and you miss your target; too many and you pay for idle cards. The right panel is the surprising part: the same eight GPUs, arranged two different ways, differ by nearly 2× in throughput - and which way wins depends on your traffic.

nobody expects

halving it doesn’t make the launch a bit slower - it can double the time to get there.

and you feel it

before you’ve served a single extra user

The prompts are long, the answers longer.

This series explains how - and how close the predictions land to real hardware.

optimal layout

search the whole space in minutes

on CPU

for traffic that doesn’t exist yet?

The solution is a cost-inverted funnel: spend almost nothing on millions of candidates, and increase computation only as the candidate set shrinks.

ace - without the cluster exis

And the answer has an expiration date. Context lengths change. Prefix reuse changes. Traffic grows.

You cannot benchmark your way out

There are three ways to get this wrong. Buy too many. Provision 64 GPUs when 24 would do, and 40 expensive cards sit idle. Buy too few. Miss the throughput target and the RL loop slows with it. Half the required throughput can mean roughly twice the rollout time. Buy the right number, but arrange them wrong. Eight GPUs could be one 8-way tensor-parallel replica, eight single-GPU replicas, or something in between. Those layouts are far from equivalent: on the same simulated workload, two 8-GPU configurations produced 1,255 vs. 2,325 tokens/s - a 1.85× difference from the same hardware. Worse, there is no universally optimal layout. Change the traffic, and the winner can change with it.

The question usually arrives in the same form: a team is building an RL pipeline around a 27B model. Rollouts need ~5,000 tokens/s, with long prompts and even longer completions. How many H100s? It sounds like a division problem. It isn’t. Hidden inside are three questions: how many GPUs, arranged how, and for what traffic? The first determines the purchase. The second can determine whether the answer is 8 or 64. The third determines whether today’s best configuration still wins six months from now.

Today, teams benchmark a handful of configurations and extrapolate from a search space of millions. We take a different approach: simulate the serving stack on CPU, search millions of configurations in minutes, and find the optimal layout before running a single GPU experiment.

Finding the Optimal Inference Serving Configuration Among Millions - Without a Single GPU Run