Deep dive
Scale-to-zero for self-hosted LLMs
Sleeping an idle model deployment is easy. Answering the first request after it wakes, without dropping it, is where the design actually lives. This page documents how Devproof does it — and the four invariants that each broke it in a running cluster before they were invariants.
The expensive idle
A served open-weight model is a pod holding its weights in memory. It costs the same whether it is answering a thousand requests a minute or none. For a shared cluster running a catalog of models — a big one for hard turns, a small one for classification, a reasoning model somebody tried once in March — most of them are idle most of the time, and the idle ones are indistinguishable from the busy ones on your bill.
The obvious response is to scale idle deployments to zero replicas. The part that is not obvious is what happens to the next request. Weights have to come back off disk, the engine has to start, and the pod has to become Ready. On our dev cluster that takes roughly 37 seconds from a warm cache PVC. Somebody's request arrives during those 37 seconds. What you do with it determines whether scale-to-zero is a feature or an outage generator.
Why the obvious design fails
The intuitive approach is to treat a sleeping model as an absent one: when it scales to zero, drop it from the gateway's routing table, and add it back when it returns. This is wrong in a specific and instructive way.
Removing a model from the gateway config means rewriting that config and reloading the gateway. A reload is a rolling restart. Rolling restarts drop in-flight connections, and for a period measured in minutes you have a mix of old and new replicas, some of which no longer know the route. So the design that is supposed to make a sleeping model reachable again is the same design that guarantees the request arriving during the transition has nowhere to land.
It also inverts the problem. You want the request to be the thing that causes the wake. If the route is gone, the request cannot reach anything that knows how to wake it.
Sleeping deployments stay routed
The core decision is that sleep and wake never touch gateway configuration. A deployment at zero replicas keeps its route, keeps its entry in the model list, and keeps its ClusterIP Service. In the operator, the routed set is defined by phase:
routed := func(s v1alpha1.ModelDeploymentStatus) bool {
return s.Phase == "Ready" || s.Phase == "Idle"
}
Idle is a healthy, routed, steady state — not a degraded one. That single
choice is what makes the rest possible: because no config changes, no reload happens, and
a connection held open across the sleep-to-wake transition survives it.
The deployment reaches Idle when the scaler has seen a full idle window.
The scaler ticks every 15 seconds and the default window is 15 minutes, so sleep is
sixty consecutive idle ticks. Scale-down between non-zero replica counts uses a
separate, shorter hysteresis — twelve ticks, three minutes — because shedding a replica is
cheap and shedding the last one is not.
The gateway holds the request
With the route intact, the first request after sleep reaches the gateway normally. A pre-call hook checks the model's routing state and, if it is asleep, does two things: signals the control plane, then holds the connection open until the model can serve it.
if send_signal:
await pool.execute(
"INSERT INTO wake_requests (model) VALUES ($1) ON CONFLICT (model) DO NOTHING", model)
await pool.execute("SELECT pg_notify('devproof_wake', $1)", model)
deadline = time.monotonic() + WAKE_HOLD_MAX
while time.monotonic() < deadline:
await asyncio.sleep(WAKE_POLL)
state = await _routing_state(model)
if state == "ready":
return
if state is None:
break # deployment deleted while waking
raise HTTPException(status_code=503,
detail=f"model {model} is waking from scale-to-zero - retry shortly",
headers={"Retry-After": "60"})
The signal is a Postgres row plus a NOTIFY, not an HTTP call. The row is the
durable part and the notification is the fast part: the control plane holds a
LISTEN devproof_wake connection, so the wake normally starts within
milliseconds, but if the control plane is restarting at that exact moment the request
survives in wake_requests and gets picked up by a sweep. Belt and braces, with
the braces costing one row.
The hold is a 2-second poll against a projection table, capped at 300 seconds. Past the
cap the caller gets a 503 with Retry-After rather than a hang.
One detail makes this viable at all: LiteLLM awaits pre-call hooks with a bare
await and no timeout wrapper, so the real bound on the hold is the client's
own timeout, not the gateway's.
Waking is an annotation, not a scale call
The control plane does not scale the deployment directly. It patches one annotation:
await deps.kube.patch("modeldeployments", model, {
metadata: { annotations: { "serving.devproof.ai/target-replicas": "1" } },
});
await deps.repo.setModelRouting(model, "waking");
await deps.repo.clearWakeRequest(model);
There is exactly one writer of the InferenceService replica count — the ModelDeployment reconciler — and it sources that number from this annotation. Everything that wants to influence replicas, including the autoscaler, writes the annotation and lets the reconciler apply it. Without that rule, a wake and a scaler tick race each other and the loser silently wins half the time.
The operation is idempotent by design, because three independent triggers can fire for the same wake: the gateway hook, the session launch gate (a session whose model is asleep parks its payload and wakes the model), and the reconciler sweep.
A projection, plus a sweep that makes it true
The model_routing table the gateway polls is not authoritative state. It is a
projection of two facts — the deployment's phase, and whether the control plane has warmed
it since it became routable:
export function routingStateFor(phase: string, warmed: boolean): RoutingState {
if (phase === "Ready" && warmed) return "ready";
if (phase === "Idle") return "idle";
return "waking";
}
Event hooks update the projection immediately, which is what makes holds release
promptly. A sweep on the reconciler's cadence recomputes the whole table from scratch,
which is what makes it correct after a control-plane crash, a lost notification, or a
wake that failed with no session parked behind it. The same sweep drains
wake_requests and re-fires any wake whose model is somehow still asleep.
Fast path and correct path are different mechanisms. The fast one is allowed to miss; the slow one is not allowed to be wrong.
Four invariants, each learned the hard way
Everything above is the design as it exists after the following four things broke it in a running cluster. They are listed because the failures are more informative than the design.
1. Never infer a cache bounce from zero replicas
When a deployment's placement changes, its model-cache PVC has to be deleted so the weights
re-provision on the target nodes. The reconciler drains to zero, waits for pods to vanish,
then deletes the PVC. The original implementation detected "pods are gone" by checking
liveReplicas == 0.
That check was unambiguous right up until Idle became a steady state. Then every
sleeping deployment started looking exactly like a drained one, and the reconciler deleted
the weights out from under models that were merely asleep. The fix is an explicit sentinel —
a serving.devproof.ai/cache-bounce annotation that is only ever set by the bounce
path:
func bounceAction(annotated, placementMoved bool, liveReplicas int64) string {
if placementMoved { return "drain" }
if annotated { return "finish" }
_ = liveReplicas // steady zero (Idle) is deliberately NOT a bounce state
return "none"
}
The discarded parameter is kept deliberately, with the comment attached, so the next person to think "we could just check replicas here" finds the answer before the bug.
2. Internal traffic must bypass the hold
After a model becomes routable the control plane sends a small warmup completion through the gateway, so the first real user request is not also the first prompt the engine has ever seen. Warmup traffic uses the internal key.
If internal traffic is subject to the wake hold, the deadlock is immediate and total: the
warmup request is held waiting for the model to be ready, but the model only
becomes ready once it is warmed, and the thing that warms it is the request
currently being held. The system waits 300 seconds and then gives up, forever.
3. Warmup bookkeeping must be Ready-phase-only
The control plane tracks which models it has warmed in memory. After a restart that set is
empty, so every routed model briefly projects as waking rather than
ready. If the newly-routed detection that triggers warmup does not filter on
phase, a restart treats every sleeping deployment as newly routable and wakes the entire
catalog — which is both the most expensive possible outcome and the exact opposite of the
feature's purpose. Restarting the control plane and confirming the sleepers are still
Idle three minutes later is now a thing we check.
The same restart window produces a subtler hazard. With the warmed set empty, a normal
request to a busy deployment can briefly see waking and signal a wake.
If the wake handler patched target-replicas: "1" unconditionally, it would stomp
the autoscaler's current target — taking a deployment running four replicas under load down
to one. Hence the guard: read the custom resource and only wake if it genuinely says
Idle.
const d = await deps.kube.get("modeldeployments", model);
if (d?.status?.phase !== "Idle") return;
await wakeModel(deps, model);
Correspondingly, the gateway only sends the notification for a model it observed as
idle. A model already waking just holds and polls.
4. A phase the API server rejects is not a phase
The least interesting bug and the most annoying one. Idle was added to the
controller before it was added to the CRD's status.phase enum. The API server
rejected every status update carrying it, which meant the phase never changed — deployments
scaled to zero while still reporting Ready, so the gateway cheerfully routed
traffic to zero pods and nothing ever woke. Validation failures on a status subresource are
easy to miss precisely because the reconcile loop keeps running and nothing crashes.
The numbers
scaler tick 15s
idle window before sleep 15 min (default, per deployment)
scale-down hysteresis 12 ticks = 3 min
wake signal Postgres row + NOTIFY
gateway hold poll 2s
gateway hold cap 300s -> 503 + Retry-After: 60
measured wake (warm PVC) ~37s
Retry-After rather than pretending the request can always be saved.What generalises
Very little of this is specific to language models. The shape — an expensive resource that should sleep, a request that must survive the wake, and a router that must not forget the sleeper exists — turns up wherever cold starts meet a proxy. Three things were worth the bruises:
- Keep the route; change the state. Routing changes are expensive and disruptive exactly when you can least afford it. A sleeping thing should still be addressable.
- Make the resting state explicit. Most of these bugs are one piece of code inferring "asleep" from a condition that something else also produces. Zero replicas meant three different things; it needed to mean one.
- Separate the fast path from the correct path. Notifications make it snappy, the sweep makes it true, and conflating them gets you a system that is usually right.
The implementation is Apache-2.0 and lives in
the repository — the gateway hook in
helm-charts/files/custom_callbacks.py, the wake path in
control-plane/src/wake.ts and routing-state.ts, and the phase and
scaler logic under operator/internal/.