Where inference time and money actually go

Most teams discover their inference bill before they understand it. The model is the same everywhere; what differs is how well the serving stack keeps the GPU busy. Here is where the time and money go, and the handful of optimizations with real evidence behind them.

Two phases, two bottlenecks

Every request to a language model runs in two phases that behave very differently.

• Prefill reads the whole prompt in one parallel pass. It is mostly large matrix multiplications, so it is usually compute-bound: the GPU's arithmetic throughput sets the pace. It decides your time to first token.
• Decode produces the answer one token at a time. Each step does little arithmetic but has to stream the model's weights and the growing key-value cache from memory, so at small batch sizes it is memory-bandwidth-bound. It decides how fast the rest of the answer arrives.

That split explains most of what follows. A GPU decoding one request at a time is mostly waiting on memory. The cheapest way to cut cost per token is to give each weight read more useful work, by serving more sequences in the same step, and to stop wasting the memory that limits how many you can fit.

Batch per step, not per request

Classic batching waits to fill a batch, runs every request in it to completion, then starts the next batch. Short answers sit idle while long ones finish, and new requests queue behind the whole batch.

The Orca paper (OSDI 2022) changed the unit of scheduling from the request to the single decoding step: after every iteration, finished sequences leave and waiting ones join. On GPT-3 175B it reported a 36.9× throughput improvement over NVIDIA FasterTransformer at the same latency. That is a comparison against one baseline in one setup, not a universal multiplier, but the idea, now usually called continuous or in-flight batching, is the default in modern servers such as vLLM and TensorRT-LLM.

Stop wasting the KV cache

To avoid recomputing attention over the whole context on every step, the server keeps a key-value cache per sequence, and it grows with every token. Allocating it as one contiguous block per request, sized for the worst case, leaves large parts reserved but unused. That waste caps the batch size, and the batch size caps throughput.

PagedAttention, the core of vLLM (SOSP 2023), stores the cache in fixed-size blocks that do not need to be contiguous, the way an operating system pages memory, and lets requests share blocks, for example a common system prompt. The vLLM team reports 2 to 4× higher throughput than FasterTransformer and Orca at the same latency, with larger gains for longer sequences, larger models and more complex decoding.

Decode fewer times

Since decode is bound by memory reads rather than arithmetic, checking several tokens costs about the same as producing one. Speculative decoding exploits that: a small, fast draft model proposes a few tokens, and the large model verifies them all in one pass, keeping the ones it agrees with.

Leviathan, Kalman and Matias (ICML 2023) showed it can be done with a sampling rule that leaves the output distribution exactly unchanged, and reported a 2 to 3× speed-up on T5-XXL with identical outputs, without retraining. The gain depends on how often the draft model is right, so it varies by model pair and by workload: code and templated text tend to speculate well, open-ended creative text less so.

Split prefill from decode

Running both phases on the same GPUs makes them interfere: a long prompt's prefill stalls the decode steps of everyone else, and the two phases want different parallelism. DistServe (OSDI 2024) puts prefill and decode on separate GPUs, sizes each pool for its own latency target, and places them according to the cluster's bandwidth. Its point is that the right metric is not raw throughput but goodput: requests served per second within your latency targets for both the first token and every token after it.

How we measure, before we claim

None of these techniques is free, and none wins everywhere. Quantization trades accuracy for memory. Speculation can slow down workloads where the draft guesses badly. Disaggregation adds data movement. Bigger batches raise throughput but stretch tail latency. That is why we are building Saaf Infer as a measurement-first project. For a given model, hardware and real traffic sample, it reports:

• time to first token and inter-token latency at p50 and p99, not averages;
• goodput against a stated latency target;
• cost per million input and output tokens on the same hardware;
• output quality before and after each optimization, on the same evaluation set.

Saaf Infer is in research. We will publish its first numbers with the method and the scripts to reproduce them, the same way we publish our guardrail benchmarks, and we will not quote a speed-up we cannot reproduce.

Sources

1. Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022.
2. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023; UC Berkeley Sky Lab, vLLM project page.
3. Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding, ICML 2023.
4. Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving, OSDI 2024.
5. Prefill vs. Decode Bottlenecks: SRAM–Frequency Tradeoffs and the Memory-Bandwidth Ceiling (arXiv 2512.22066).

Speed-ups are quoted with the setup they were measured in. Checked against the linked sources on 27 October 2026.