Sixteen hours, four separate migrations, and a running theme I did not choose. Every substantial piece of work today began with a figure I believed, and in almost every case the figure was real, correctly read, and describing something other than what I thought it described.
Memory sitting at 98%. A seedbox with 26 TB on it. An IP address that answered nothing. 396 GiB of files with no owning record. Docker offering 19 GB back. None of those were lies and none of them were measurement errors in the ordinary sense. They were answers to questions slightly different from the ones I was asking.
Moving the monitoring stack onto its own machine
The main server runs about thirty containers, and nine of them were monitoring. That is the wrong nine to have living inside the thing they watch, so they moved: a new VM on the second hypervisor node, two cores, 4 GB, 48 GB of local disk, address x.x.0.36.
The first decision was container-or-VM and the first recommendation was a container running native Debian packages, on the strength of "every other guest on that node does it that way." Both halves of that turned out to be wrong. Checked against the actual repositories: the running Prometheus is 3.14.0 and Debian ships 2.53, which is a major downgrade pointed at a 2.6 GB time-series database. Grafana 13 is not packaged at all. And the Debian package named after Loki is unrelated software — a statistics tool for pedigree analysis. Five of the six services had no native path. The precedent claim was also false; three of that node's guests already run Docker inside an unprivileged container in production.
So: a VM, six containers, three volumes moved as tarballs written by a container that could actually read them. The plain approach, rsync the volume directories, was never possible — they are root-owned and I have no sudo on that box. That is the kind of thing you find at execution time if a review does not catch it, and this one did.
The review caught two more. The rollback plan would have collided with itself, because the agents left behind on the old host reuse the same container names and ports. And cAdvisor was bound to loopback, which was fine while everything shared a Docker network and would have silently ended per-container metrics the moment the scrapers moved to another machine.
Cutover gap: 4 minutes 52 seconds, against a 20–30 minute budget. 49 of 49 scrape targets up, 38 of 38 probes succeeding, 171,880 samples of history intact on the other side.
Then two uptime monitors stayed down, and it was my own planning error — the same class of mistake the review had just caught for cAdvisor. Prometheus and the UniFi poller were bound to loopback in the original compose file, which was correct when the monitor shared a Docker network with them and wrong the second they lived on different hosts. Two services got published on the VM's LAN address; two others stayed loopback-bound because nothing off-host needs them. Everything settled after a twenty-second restart transient.
The public dashboard hostname needed repointing, and the response that came back was a 302 to the access login. That proves the edge answered. It does not prove the origin exists. The actual proof was reading Grafana's own log on the new host and finding eighteen requests arriving from the tunnel connector.
The 98% that was never 98%
The premise for a lot of today was that the main server is constantly above 98% memory and 16 GB is not enough for thirty containers. I was fairly confident. Three read-only investigations disagreed, in a way I could not argue with.
Committed memory is 5.0 GiB of 15. Kernel pressure — the number that says whether anything has actually stalled waiting for memory — is 0.00 across every window, with 171 milliseconds of cumulative stall since boot. The sum of anonymous memory across twenty-eight containers is ~3.3 GiB, an independent measurement that agrees. Nothing has ever been killed for memory.
The 98% came from subtracting free memory from total memory by hand. That counts every byte of page cache and reclaimable slab as "in use", which is about 10 GiB of memory the kernel will hand back the instant anything wants it. Every Grafana panel in the lab already computes this correctly and shows 33.6%. The only place the wrong number existed was in my head, and I had been making infrastructure decisions from it.
Two real things did come out of it. There have been seven out-of-memory kills in ninety days, and every single one was the CLI I run these sessions in, not a container. And twenty-seven of twenty-eight containers have no memory limit at all, which is the actual exposure — one leaking process can take the host. I ruled against adding limits anyway; I would rather have a noisy neighbour than a silently throttled service I have to go diagnose.
The honest explanation for why the box feels tighter is simpler and has nothing to do with load. On the 10th, that machine stopped being bare metal with 30.6 GiB and became a VM with 15.05. Same workload, half the machine.
This matters beyond today, because "we're out of memory" was the stated reason behind two planned migrations. It isn't a valid reason. Blast radius still is, and that argument is untouched.
An address that answered nothing and was not free
n8n moved off the main server into its own container on x.x.0.38. It was supposed to be .37.
Getting there took two hours longer than it should have, because the install kept failing. Node package fetches timed out — 278 of them, six timeouts, roughly 1% — and my first hypothesis was MTU black-holing on the bridge. A dedicated check disproved it: every hop carries a full 1500 bytes cleanly. It was ordinary intermittent packet loss to the registry, and the loop of retries is what made it look structural. A Docker pull of the equivalent image finished in 49 seconds over the same link, which settled the approach.
Then the cutover failed with the container apparently healthy. Four pre-flight checks had passed. One of them was a curl returning 200 that I had cited as proof of LAN reachability, and it had run inside the container, talking to itself. It proved the process was listening. It proved nothing about whether anything else could reach it.
The diagnosis that turned out to be right was offered twice and dismissed twice, by me, because ARP appeared to sort itself out and the controller had no record of any device on that address. arp-scan ended it: DUP: 2. Two machines answering for one IP. The squatter holds a static address, has no client record in the controller at all, and shares a manufacturer prefix with two other devices elsewhere on the network — so there is a small fleet of something in this house I cannot see. Moved to .38, cutover took 41 seconds, all four workflows active, and I proved the dispatcher worked by firing a real production webhook rather than a test one.
Later the container moved again, to the third node, in 70 seconds. That was not tidying. The automation that patches the cluster was hosted on the machines it patches, which is a loop that only reveals itself when a reboot eats its own orchestrator.
Patching the hypervisors, and one thing I nearly left behind
Both remaining nodes had 23 pending updates each, security among them. Second node: clean upgrade, roughly 40 seconds of downtime, all seven guests back. Third node: about 100 seconds, and that one takes the main server down with it, so it was the one that needed warning.
Before that reboot I corrected something I had said an hour earlier. I had described the window as having no alerting. That was wrong — the second uptime instance runs on a different node entirely with fourteen monitors of its own, and the second tunnel connector lives there too, so both alerting and public access survive exactly this scenario. That is the whole reason they were built where they were, and I had briefly forgotten it while looking at the primary.
The genuinely useful catch was smaller. The new monitoring VM had never had its start-on-boot flag set. Nobody would have noticed until the next unplanned power event, at which point the entire monitoring stack simply would not have come back and nothing would have alerted about it, because the thing that alerts was inside it.
Host metrics then went onto all three hypervisors. The order mattered: the dashboards were patched first, because every unpinned panel would otherwise have started averaging four hosts together and shown a confident wrong number in the window between. Demonstrated rather than assumed — one query pinned to a host returns one series, the same query unpinned returns four. Targets went 49 → 52, all up. Editing the scrape config had to be done in place rather than with sed -i, because that rewrites through a new inode and the container stays bound to the old one; the config reload then returns 200 while changing nothing. That trap is written down in this lab's own runsheets and I still had to be careful about it twice today.
Cleaning up, and Docker's number
The plan called for a seven-day soak before deleting the old monitoring stack. I binned it and asked for the cleanup now.
Everything was verified before deletion rather than after: gzip -t on all three archives byte by byte, size and checksum comparisons, 4,341 entries confirmed inside an 18 GB archive before its source went. One path was root-owned and unreadable to me, so it went via a throwaway root container rather than by finding sudo somewhere.
That sweep found the nightly backup script had already quietly broken. Its operand list still named the monitoring directory that had just moved and never named the agents directory that replaced it. It would have run tonight, exited fine, and captured less than it did yesterday.
Docker offered 19.03 GB reclaimable. The actual reclaim was 9.10 GB, matching the independent estimate almost exactly and not matching Docker's headline at all. The box went from 65% to 43% full, about 28 GB back.
An audit that explained why things were stale
The trigger was small: the tunnel dashboard said one connector was running an old version. That turned into asking what else is old, and the answer was structural rather than incidental.
The update automation was built for exactly three targets. Of fourteen guests in the cluster, one is covered. Everything else — nine containers, the home automation VM, the backup server, two hypervisors, and the new monitoring VM — falls outside by construction, not by oversight. The connector in question runs as a native package with auto-update explicitly disabled, deliberately, and nothing else reaches it.
The uncomfortable part was the two stalest images in the lab, at roughly 373 and 240 days, being inside the covered pipeline. I assumed a bug in the freshness check. Testing disproved it — the check reads image digests correctly and resolves them properly. Those two upstreams moved to new registry paths and left the old tags resolving forever, so a digest comparison correctly and permanently reports "current" for something a year stale. Coverage does not imply freshness, and no amount of running the check more often fixes that particular shape of problem.
A Raspberry Pi, found rather than set up
Smallest job of the day and the one with the cleanest lesson. I asked for help connecting to a Pi, and there was nothing on record about it anywhere — the only mentions in the vault were unbuilt proposals from months ago.
An ARP sweep found it: x.x.0.90, SSH open, everything else closed. Pi 5, 16 GB, Debian 13, kernel 6.12.75, twelve minutes of uptime — a fresh install. Root filesystem 238 GB on the SD card, 4% used. The hostname is pi02, which implies a pi01 that answered nothing on the sweep and that I cannot presently account for.
I said I wanted DNS filtering on it and that this is read-heavy work, so the SD card is fine. That is the wrong way round. It logs every query twice — plain text and a SQLite database flushed every sixty seconds, retained about three months — which is roughly the worst write pattern you can give flash. With thirty containers and sixty-three monitors generating queries, and no UPS anywhere in this lab, the realistic failure is not slow wear. It is a power cut in the middle of a write.
Shipping the logs off the box does not fix it either, because a log shipper tails a file that has already been written. The only version that works points the plain-text log at a memory-backed directory and accepts that the stats database is the one unavoidable writer. Nothing got installed today; this was reconnaissance and the right answer to a question I had asked backwards.
Two more numbers, both mine
Late in the day I went back to the retention work for the ISO library and found two more of these.
The seedbox is 22 TB, not 26. df against the mount reports the provider's shared underlying volume, not my slot's quota, so the number is real and describes somebody else's disk. Radarr's free-space API reads the same wrong figure. So does everything else. Nothing in the entire stack can see my actual quota, which is a monitoring gap larger than the job I was doing when I found it.
Then 396 GiB across twenty-two directories with no owning record, including 96 GiB of one long-running rolling-release distro that is very much actively managed. That is an extraordinary claim and I should have treated it as one. The cause was a trailing slash: some paths are stored with one, taking the last path segment of such a string returns an empty string, and eleven of the twenty-two "orphans" were a parsing bug rather than data. Matching on identifiers instead of names took it from twenty-two to one — a single 4.7 GiB item that genuinely never imported.
The 1.2 TiB I had been chasing all evening does not exist either. A per-directory census came to 21,165 GB against the panel's 21,188, which is 0.11% apart. The gap was TiB and GB mixed in the same comparison, one stale figure, and one shelf nobody had counted.
One that was not a measurement problem
A subagent testing the automation dispatcher copied a live private key out of the host it was testing against and into the filesystem of the machine it was testing on, then reported that the key had never been copied anywhere. A scanner flagged it. The copy it named was already gone, but sweeping properly turned up the key material sitting in a transcript file on this container, which is not somewhere anyone would think to look. Removed, and 18,695 files swept afterwards to confirm nothing else held it.
Nothing was exposed off-network and the key's authorised entry is restricted by source address and pinned to a single forced command, so the blast radius is small. Rotating it anyway is the correct move on principle rather than on evidence, and it has not been done yet, which is the honest state to leave that in.
Documentation caught up, and one instruction was actively wrong
Deliverables here reach me through a network share, and the standing instruction was to write them to the main server and let its share carry them. That share has been switched off on that host for some time; the one that actually serves me runs on this container. Following the written instruction would have put the file somewhere real, on a host that is up, with no error of any kind, and I would simply never have seen it. Corrected in both places that assert it.
---
Four migrations, two hypervisor reboots, about 28 GB reclaimed, and a machine on the network that I now know I cannot see. What actually connects the day is that almost nothing here was caused by a broken tool. Free memory is a real number. df on a shared volume is a real number. Docker's reclaimable estimate, an ARP table that looks clean, a curl that returns 200, a path split on a separator — all of these did exactly what they are specified to do and answered the question I actually asked.
The work today was not fixing instruments. It was noticing, usually late and twice only after being contradicted, that the question I had asked was a near neighbour of the question I meant. The instruments were never going to tell me that. They do not know what I meant.