DEVPROOF.AI Own your scalable AI

Deep dive

Scale-to-zero for self-hosted LLMs

Sleeping an idle model deployment is easy. Answering the first request after it wakes, without dropping it, is where the design actually lives. This page documents how Devproof does it — and the four invariants that each broke it in a running cluster before they were invariants.

The expensive idle

A served open-weight model is a pod holding its weights in memory. It costs the same whether it is answering a thousand requests a minute or none. For a shared cluster running a catalog of models — a big one for hard turns, a small one for classification, a reasoning model somebody tried once in March — most of them are idle most of the time, and the idle ones are indistinguishable from the busy ones on your bill.

The obvious response is to scale idle deployments to zero replicas. The part that is not obvious is what happens to the next request. Weights have to come back off disk, the engine has to start, and the pod has to become Ready. On our dev cluster that takes roughly 37 seconds from a warm cache PVC. Somebody's request arrives during those 37 seconds. What you do with it determines whether scale-to-zero is a feature or an outage generator.

Why the obvious design fails

The intuitive approach is to treat a sleeping model as an absent one: when it scales to zero, drop it from the gateway's routing table, and add it back when it returns. This is wrong in a specific and instructive way.

Removing a model from the gateway config means rewriting that config and reloading the gateway. A reload is a rolling restart. Rolling restarts drop in-flight connections, and for a period measured in minutes you have a mix of old and new replicas, some of which no longer know the route. So the design that is supposed to make a sleeping model reachable again is the same design that guarantees the request arriving during the transition has nowhere to land.

It also inverts the problem. You want the request to be the thing that causes the wake. If the route is gone, the request cannot reach anything that knows how to wake it.

Sleeping deployments stay routed

The core decision is that sleep and wake never touch gateway configuration. A deployment at zero replicas keeps its route, keeps its entry in the model list, and keeps its ClusterIP Service. In the operator, the routed set is defined by phase:

routed := func(s v1alpha1.ModelDeploymentStatus) bool {
    return s.Phase == "Ready" || s.Phase == "Idle"
}

Idle is a healthy, routed, steady state — not a degraded one. That single choice is what makes the rest possible: because no config changes, no reload happens, and a connection held open across the sleep-to-wake transition survives it.

The deployment reaches Idle when the scaler has seen a full idle window. The scaler ticks every 15 seconds and the default window is 15 minutes, so sleep is sixty consecutive idle ticks. Scale-down between non-zero replica counts uses a separate, shorter hysteresis — twelve ticks, three minutes — because shedding a replica is cheap and shedding the last one is not.

The gateway holds the request

With the route intact, the first request after sleep reaches the gateway normally. A pre-call hook checks the model's routing state and, if it is asleep, does two things: signals the control plane, then holds the connection open until the model can serve it.

if send_signal:
    await pool.execute(
        "INSERT INTO wake_requests (model) VALUES ($1) ON CONFLICT (model) DO NOTHING", model)
    await pool.execute("SELECT pg_notify('devproof_wake', $1)", model)

deadline = time.monotonic() + WAKE_HOLD_MAX
while time.monotonic() < deadline:
    await asyncio.sleep(WAKE_POLL)
    state = await _routing_state(model)
    if state == "ready":
        return
    if state is None:
        break  # deployment deleted while waking

raise HTTPException(status_code=503,
                    detail=f"model {model} is waking from scale-to-zero - retry shortly",
                    headers={"Retry-After": "60"})

The signal is a Postgres row plus a NOTIFY, not an HTTP call. The row is the durable part and the notification is the fast part: the control plane holds a LISTEN devproof_wake connection, so the wake normally starts within milliseconds, but if the control plane is restarting at that exact moment the request survives in wake_requests and gets picked up by a sweep. Belt and braces, with the braces costing one row.

The hold is a 2-second poll against a projection table, capped at 300 seconds. Past the cap the caller gets a 503 with Retry-After rather than a hang. One detail makes this viable at all: LiteLLM awaits pre-call hooks with a bare await and no timeout wrapper, so the real bound on the hold is the client's own timeout, not the gateway's.

Waking is an annotation, not a scale call

The control plane does not scale the deployment directly. It patches one annotation:

await deps.kube.patch("modeldeployments", model, {
  metadata: { annotations: { "serving.devproof.ai/target-replicas": "1" } },
});
await deps.repo.setModelRouting(model, "waking");
await deps.repo.clearWakeRequest(model);

There is exactly one writer of the InferenceService replica count — the ModelDeployment reconciler — and it sources that number from this annotation. Everything that wants to influence replicas, including the autoscaler, writes the annotation and lets the reconciler apply it. Without that rule, a wake and a scaler tick race each other and the loser silently wins half the time.

The operation is idempotent by design, because three independent triggers can fire for the same wake: the gateway hook, the session launch gate (a session whose model is asleep parks its payload and wakes the model), and the reconciler sweep.

A projection, plus a sweep that makes it true

The model_routing table the gateway polls is not authoritative state. It is a projection of two facts — the deployment's phase, and whether the control plane has warmed it since it became routable:

export function routingStateFor(phase: string, warmed: boolean): RoutingState {
  if (phase === "Ready" && warmed) return "ready";
  if (phase === "Idle") return "idle";
  return "waking";
}

Event hooks update the projection immediately, which is what makes holds release promptly. A sweep on the reconciler's cadence recomputes the whole table from scratch, which is what makes it correct after a control-plane crash, a lost notification, or a wake that failed with no session parked behind it. The same sweep drains wake_requests and re-fires any wake whose model is somehow still asleep.

Fast path and correct path are different mechanisms. The fast one is allowed to miss; the slow one is not allowed to be wrong.

Four invariants, each learned the hard way

Everything above is the design as it exists after the following four things broke it in a running cluster. They are listed because the failures are more informative than the design.

1. Never infer a cache bounce from zero replicas

When a deployment's placement changes, its model-cache PVC has to be deleted so the weights re-provision on the target nodes. The reconciler drains to zero, waits for pods to vanish, then deletes the PVC. The original implementation detected "pods are gone" by checking liveReplicas == 0.

That check was unambiguous right up until Idle became a steady state. Then every sleeping deployment started looking exactly like a drained one, and the reconciler deleted the weights out from under models that were merely asleep. The fix is an explicit sentinel — a serving.devproof.ai/cache-bounce annotation that is only ever set by the bounce path:

func bounceAction(annotated, placementMoved bool, liveReplicas int64) string {
    if placementMoved { return "drain" }
    if annotated      { return "finish" }
    _ = liveReplicas  // steady zero (Idle) is deliberately NOT a bounce state
    return "none"
}

The discarded parameter is kept deliberately, with the comment attached, so the next person to think "we could just check replicas here" finds the answer before the bug.

2. Internal traffic must bypass the hold

After a model becomes routable the control plane sends a small warmup completion through the gateway, so the first real user request is not also the first prompt the engine has ever seen. Warmup traffic uses the internal key.

If internal traffic is subject to the wake hold, the deadlock is immediate and total: the warmup request is held waiting for the model to be ready, but the model only becomes ready once it is warmed, and the thing that warms it is the request currently being held. The system waits 300 seconds and then gives up, forever.

3. Warmup bookkeeping must be Ready-phase-only

The control plane tracks which models it has warmed in memory. After a restart that set is empty, so every routed model briefly projects as waking rather than ready. If the newly-routed detection that triggers warmup does not filter on phase, a restart treats every sleeping deployment as newly routable and wakes the entire catalog — which is both the most expensive possible outcome and the exact opposite of the feature's purpose. Restarting the control plane and confirming the sleepers are still Idle three minutes later is now a thing we check.

The same restart window produces a subtler hazard. With the warmed set empty, a normal request to a busy deployment can briefly see waking and signal a wake. If the wake handler patched target-replicas: "1" unconditionally, it would stomp the autoscaler's current target — taking a deployment running four replicas under load down to one. Hence the guard: read the custom resource and only wake if it genuinely says Idle.

const d = await deps.kube.get("modeldeployments", model);
if (d?.status?.phase !== "Idle") return;
await wakeModel(deps, model);

Correspondingly, the gateway only sends the notification for a model it observed as idle. A model already waking just holds and polls.

4. A phase the API server rejects is not a phase

The least interesting bug and the most annoying one. Idle was added to the controller before it was added to the CRD's status.phase enum. The API server rejected every status update carrying it, which meant the phase never changed — deployments scaled to zero while still reporting Ready, so the gateway cheerfully routed traffic to zero pods and nothing ever woke. Validation failures on a status subresource are easy to miss precisely because the reconcile loop keeps running and nothing crashes.

The numbers

scaler tick                15s
idle window before sleep   15 min (default, per deployment)
scale-down hysteresis      12 ticks = 3 min
wake signal                Postgres row + NOTIFY
gateway hold poll          2s
gateway hold cap           300s  -> 503 + Retry-After: 60
measured wake (warm PVC)   ~37s
The honest caveat: ~37 seconds is a measurement from a dev cluster with a warm cache PVC and a small model. A large model on a cold cache is a weight download, and that is minutes, not seconds — which is why the hold cap is 300 seconds and why the 503 carries a Retry-After rather than pretending the request can always be saved.

What generalises

Very little of this is specific to language models. The shape — an expensive resource that should sleep, a request that must survive the wake, and a router that must not forget the sleeper exists — turns up wherever cold starts meet a proxy. Three things were worth the bruises:

The implementation is Apache-2.0 and lives in the repository — the gateway hook in helm-charts/files/custom_callbacks.py, the wake path in control-plane/src/wake.ts and routing-state.ts, and the phase and scaler logic under operator/internal/.