Seven days away, then a full read-only audit of the lab: every service, dashboard, and log checked against what was actually happening, not what its own health check claimed. The audit turned up four systems that had been reporting green the whole time while doing none of the work they were supposed to. Nothing got fixed during the pass; the goal was to read the failures off the running systems intact, before touching anything.
The clearest case: df, Uptime Kuma, Prometheus, Grafana, and both applications' own free-space APIs all agreed the seedbox had 4.6 TB free, 82% full. quota -s told a different story, the account's quota sat at 99.9% full with about 20 GiB left, because the provider enforces a per-user quota that statfs can't see. The download client's log had the honest number: Disk quota exceeded at 06:08, and the library server crashed seven minutes later. It recovered on its own after roughly 46.5 hours. Working back from the burn rate, about 61 GB/day over the prior 13 days, the fill date was arithmetic, not a surprise; nothing in the lab was watching the number that would have predicted it.
The retention tool had passed its health check for 13 days straight, container up, RestartCount 0, scheduled runs firing, while a query against its own rules API showed handledMediaAmount: 0 across all six collections since the day it was deployed. Five collections were disarmed by a ten-year grace period nobody had released, holding back roughly 1.01 TB. The sixth had a stale API key returning 401 while the live key in the app's config returned 200, failing silently for four days because the tool only notifies on success.
The automation platform reported 4 of 4 workflows active and a container up for 7 days, but docker logs --since across the whole week returned nothing, and the executions table showed zero runs since 2026-08-28, a week with no OS, container, or hypervisor updates running anywhere in the lab. One candidate cause: a five-minute read-only remount of the container's filesystem that lines up with the last successful run, plausible for a scheduler that keeps its state in SQLite, unconfirmed.
Two more issues surfaced without a settled cause: a nightly backup job failing on all three cluster nodes on filesystem and garbage-collection errors, and a home-automation VM reporting running with a responsive guest agent while the service inside it had refused connections for days.
Not everything was broken. A planned reboot came back clean, cluster quorum held 3 of 3, and one alert turned out to be a stale monitor rather than a real outage. The pattern across all four failures was the same: a status that means "alive" was standing in for a status that means "correct," and none of them showed up until someone queried the underlying data by hand.