For a while, my model router was a process on my laptop. A LiteLLM proxy on localhost:14000 with a few hundred lines of Nix config, and every agent on the LAN pointed at it. It worked until the laptop slept — and every time the laptop slept, the router slept with it. Agents in the cluster couldn't depend on it at all.
In August I moved it onto the cluster: a single-replica StatefulSet in the llm-router namespace with a postgres 17 sidecar sharing one PVC. ClusterIP only, reached through the LAN-only Traefik ingress, with a NetworkPolicy that lets in-cluster agents reach it on port 14000 from exactly one namespace: the goose namespace, where my agents run. Both containers are locked down to UID 1000 on a read-only root filesystem.
The one structural rule: don't scale it. The postgres sidecar owns the router's persisted state: the adaptive routing bandit and the per-deployment cache-hit counters. A second replica would split that state across independent stores. One router, one brain.
The migration was the easy part. The lesson was about what the router routes on: each tier has a set of interchangeable providers, and choosing which one gets each request is the router's whole job.
The first adaptive routing iteration compared providers on raw list prices. Cheap was cheap, so the router steered cheap-tier traffic to the cheapest listed provider — an OpenRouter fallback with 2–15 second time-to-first-token. My routing policy's notes put it plainly: the earlier iteration "measurably steered cheap traffic to a 2-15s-TTFT openrouter fallback."
The traffic found the right price and the wrong experience.
The fix was to price routing the way the bill actually works, which means not trusting the list price. Providers advertise the input rate, but once prompt caching kicks in they bill most input at the cache-read rate instead: DeepSeek lists $0.14 per million for input and $0.0028 for cache-read, a 50x gap. The router measures each provider's hit rate per response from usage.prompt_tokens_details.cached_tokens, and a metered provider's cost term is now (1−h)·input + h·cache_read, where h is that measured rate: the list input price blended toward the cache-read price. On my agent workloads, DeepSeek direct measures around 97% cache hits, so its effective cost sits near the cache-read price, far below the OpenRouter list price that had been steering traffic wrong. Hit rates persist in the postgres sidecar, so the blend survives restarts. Subscription providers route at ~zero cost, since prepaid credits are already spent, and absorb traffic until exhausted; metered providers are compared only after.
The measured result: $0.008 per million tokens averaged across 962 million tokens in five days, versus roughly $0.04 per million via OpenRouter. The month's ledger is the same story at scale: two billion tokens for $16.64. The router is what keeps the cheap tier cheap and reliable: it routes to the cheapest working provider, not the cheapest listed one.
The router is a Nix-built image, a SOPS-encrypted secret, and roughly four hundred lines of manifests. The laptop can sleep now.