Everything ran on one physical host — a single-node k3s cluster with Forgejo, the monitoring stack, the goose ACP servers, cert-manager, step-ca, and two VMs. Backups were restic, daily: a local repository at 03:30 and an offsite one on Backblaze B2 at 05:00. The backup set was documented. What was missing was proof that restoration actually worked.
An earlier drill had restored app data — a 20 GiB rentfree disk, e2fsck-clean — but the platform itself had never come back from a snapshot. The k3s state, the internal CA, the cert-manager chain: none of it had been restored. "We have backups" and "we can rebuild this host" are different claims, and I had only proven the first.
So I ran a platform DR drill. The acceptance criteria were deliberately modest: a measured end-to-end rebuild time and a list of surprises. I started with the PKI, because it's the load-bearing piece. The step-ca BadgerDB holds the intermediate signing key. If that doesn't come back, every internal certificate becomes unverifiable, and every client needs a new root installed by hand.
The PKI phase went almost too well. I restored the step-ca PVC from snapshot 19f38829 — 117.5 GiB in the local repo, but the PVC itself was 529.9 KiB across 11 files: BadgerDB's complete store, vlog and sst tables, MANIFEST, KEYREGISTRY, LOCK. I booted it in a throwaway container (step-ca:0.28.4) on an isolated Docker network. Badger replayed the WAL from the crash-consistent copy with no corruption. It served the identical Homelab root — the same SHA-256 fingerprint as the live CA. I issued a drill certificate through the ACME home provisioner, and openssl verify checked the chain to the root, through the same intermediate that signs the live grafana and registry certificates. PKI: proven, with evidence.
The drill had a second phase: verifying the root CA key's recovery path. The README said it lives at /opt/ca/private/ca.key on the dev laptop, "never in k8s" — a posture that sounds safe until you look. /opt/ca doesn't exist on the laptop. Nothing in the password store. Git history shows the root was created on a long-decommissioned LXC host, backed up to a local rest server and to B2 under an old bucket prefix.
I walked that path end to end. I recovered the old ansible-vault password from the password store's git history — the entry was deleted, but the blob was still decryptable. I decrypted the decommissioned vaults from infra git history. Then I hit the wall: the old B2 application key is dead. Access Denied on both the old and the current bucket prefixes. The endpoint isn't the problem — the current application key works fine on its own prefix. The old keys were revoked after the decommission, which closed every path to that era's backups. The local rest server is unreachable, and there was never a restic password entry for the ca repository. The root private key is declared lost.
The drill also caught a gap that would have bitten on the first boot of a rebuilt host: /etc/ssh wasn't in the backup set. The host SSH key is the sops age identity that decrypts host secrets, so a rebuilt host would generate a fresh key and fail to decrypt its own secrets on first boot. That's fixed and proven — /etc/ssh is in the backup unit paths, and the rebuilt VM's sshd fingerprint matched the live host exactly, sops age identity intact.
The loss is survivable. Clients trust the intermediate, and the intermediate is recoverable. The root key only matters when a new intermediate has to be signed — at expiry, or if the BadgerDB is lost before a fresh snapshot. The current intermediate is valid until May 2031, which gives me until then to run a root-CA replacement drill or accept the intermediate-only posture as permanent. Either way, no mass client reinstall, and the root stays declared lost.
Then the full host rebuild. From the same flake that deploys services-01 I built a scratch VM — 6 GiB, 8 cores, a single disk — and restored the platform paths (/etc/rancher/k3s, /etc/ssh, /var/lib/rancher/k3s/server, /srv/k3s-local-path) from both repositories. Local: 27.66 GiB, 121,367 files in 9:56. Offsite, a full download from B2: 27.64 GiB, 121,356 files in 14:29. The two restores came out byte-identical on the platform trees — the only deltas were files written between the two snapshot times. k3s went from unit start to node Ready in about 35 seconds, and the restored SQLite replayed cleanly: the same crash-consistency question the BadgerDB answered, answered again for k3s's own state.
The verification pass is what I'd previously only claimed. The home-step-ca ClusterIssuer came up Ready with no ACME reset, so cert-manager issued a fresh drill.test certificate through the same intermediate that signs the live certs, verified to the root. Baikal's SQLite passed integrity_check and matched live row counts. A rebuilt host comes back as the same cluster — same CA, same issuer, same node, same data — not a reinstall project.
The run also surfaced its own surprises, none of them restore failures. A stale restic lock, sitting there since 2026-08-15, had been silently blocking forget --prune while nightly backups kept succeeding. restic unlock cleared three stale locks and the prune finally ran. And k3s's node name turned out to be load-bearing: the restored local-path PVs carry hostname affinity, so a scratch VM with a different name registers a second node and PVC-bound pods never schedule. A real rebuild keeps the hostname; the drill overlay now pins node-name: services-01 regardless.
The measured legs: about five minutes to build the scratch host (cached), 9:56 for the local restore, 14:29 for the offsite restore, ~35 seconds to a Ready node. The restore is the long pole, and it's minutes — not a weekend. From nix build to a node that was issuing certificates again, my honest estimate is well under an hour, dominated by B2 egress.
The lesson has a cost. The root key was deliberately kept out of the cluster. "Never in k8s" turned out to mean "on a laptop" — and then not even there. The old B2 keys were rotated after the decommission, silently invalidating the only surviving path to the old CA backups. The exact snapshot IDs, the issued drill certificate, and the dead key references are in my private runbook; the numbers above are kept exact so the record is reproducible.
If you run a DR drill, run the PKI phase first. It's the piece that turns we have backups into we can rebuild this host, and the piece that tells you which key you can no longer restore.
The drill is closed, with both acceptance criteria met: a measured rebuild time and a list of surprises. The platform comes back in minutes. The one backup I couldn't reach — the root key — was already gone, and now I know what that costs.