Monday was a maintenance day, which in my experience is another way of saying it was a day of finding out what maintenance breaks. I asked for one thing — update all my Docker containers — and finished the night having fixed a tunnel that stopped accepting the only command it had ever been given, replaced a request bot whose entire program was the text "Not Found", and worked out why requests I'd sent for weeks were being binned by a number I set once and forgot. Everything that broke was caused by the maintenance or exposed by it.
Twenty-five containers, six stacks, one word
The estate is six Compose projects plus a couple of loose containers: home automation, the photo library, the stack that fetches Linux ISOs, monitoring, the reverse proxy, and the strays. Pull every image, bring everything up, recreate only what changed. Twenty-five containers running at the end, zero restart loops, every healthcheck green.
Except the tunnel connector, which crash-looped the instant it came back. That container publishes eight hostnames, so crash-looping meant the whole public side was dark. The log complained that no valid additional argument had been passed to the tunnel command. The compose file said command: tunnel --no-autoupdate --protocol http2, and the old image had been happy to infer what I meant by a bare tunnel. The new one wants the run sub-command spelled out. One word appended to line 19, four edge connections registered seconds later.
The confirmation came from a corpse. A duplicate connector had been sitting in the estate exited and dead for two weeks, and I read its config before deleting it. Its command was tunnel --no-autoupdate run. The dead one had been doing it right the whole time.
4.156 GB, and why it wasn't 7.21
An update like that leaves orphaned layers behind, so next was a cleanup: fourteen dangling images, each cross-checked against every container before anything was removed. Docker's accounting said 7.21 GB was reclaimable. The actual reclaim was 4.156 GB, and the image count went from 41 to 28. The gap isn't an error, it's how layers work — much of what looks reclaimable is shared with images still in use. Disk-usage estimates tell you the size of the thing you're deleting, not the size of the hole it leaves.
The other thing worth saying out loud before a prune: those dangling layers are your instant rollback. Once they're gone, going back to yesterday's image means re-pulling and hoping it's still there. I knew that. I said it. I ran the prune anyway, which was fine, right up until eight minutes later.
The nine-byte program
The surface-level check said everything was up. It was only when I asked for a proper audit that the Discord request bot turned out to be lying. The container was "running". Its log was one line repeated dozens of times: invalid or corrupt jarfile.
The jar was nine bytes. Reading it printed the words Not Found.
The published image had been built the day before, and during its own build it went to fetch the upstream release, got a 404, and baked the 404 response in as the application. The version label came out as null, which in hindsight was the tell. The startup script just runs java -jar on whatever's at that path, so the container starts, fails, restarts, and calls itself running throughout. A broken build that ships is worse than one that fails, because a failed build tells you.
The obvious move was to roll back to the previous image, and the previous image was among the fourteen I'd pruned eight minutes earlier. That's the sequence I'm writing this section for: I deleted the escape route before I knew there was anything to escape from, and the prune output was still on screen while I went looking for what it had taken.
Pinning to a known version instead of chasing latest was the better fix anyway. It pulled a 17 MB jar with the right magic bytes at the front, and the logs went from a corruption loop to configuration valid, Discord connection successful, connected to guild. latest is a promise that whatever built most recently is worth having, and that promise had just been broken very literally.
The requests that were working perfectly, and failing anyway
Three minutes after the bot came back I sent a test request, and asked a question I'd been sitting on for weeks: the bot takes the request and it reaches the fetcher, but is it actually searching?
It was. The fetcher's history showed the search completing the same second the request landed and returning eight candidate releases. All eight were rejected, every one for the same reason in slightly different numbers: not enough seeders, 7 against a minimum of 20. Nothing cleared a floor of twenty, so the fetcher did exactly what it was told, found nothing acceptable, and went quiet. From the outside that is indistinguishable from not searching at all.
Two details worth keeping. First, where the fix belongs. Those thresholds are visible and editable in the fetcher, which is where you'd naturally go, and editing them there would have worked for about a day — the indexer manager runs a full sync and overwrites the fetcher's copy on its next pass. So the change went into the manager instead, and propagated to the second fetcher on its own.
Second, the number. The recommendation was to drop the floor to 1. I picked 5, because a seeder count is the only cheap signal you get that a source is alive and will finish, and a floor of 1 means queueing things that sit at 0.2% forever. Re-running the same query afterwards: 9 of 10 accepted, where it had been 0 of 8. Then the test request went end to end on its own — grabbed, imported, on disk at 5.94 GB inside two minutes, and only that far because the corrupt-jar fix had gone in three minutes earlier.
The NAS finally got its second drive
Earlier that evening, before any of the Docker work, the storage rebuild I'd been backing up towards for days happened. The second 8 TB IronWolf went in, and I rearranged all four bays: the two SSDs into bays 1 and 2 as a scratch pool of about 3 TB, the two IronWolfs into bays 3 and 4 as roughly 16 TB of bulk. Both pools are striped with no redundancy — my call, made with the trade-off in front of me and reaffirmed when it was argued against. The bay order is deliberate too, so the third IronWolf can drop into bay 2 later with all three adjacent.
Then the slow part: recreate every share and push four-odd terabytes back in. Mid-reload the mount showed 15T with 659 GB used, a satisfying and slightly alarming number. The thing that could have gone wrong quietly was the photo library, whose upload target sits in a share that had just been recreated empty. I'd paused it from my phone first, so it never woke up staring at a half-full folder.
The upload error I got confidently wrong
My phone then threw an upload error on one asset: the network connection was lost. I diagnosed that immediately and with real conviction as the tunnel's 100 MB request-body cap. The reasoning was clean — the photo app uploads each asset as one un-chunked POST, so anything over the cap gets cut off mid-flight and surfaces as a lost connection, and one asset failing while everything else works is exactly that shape. I verified the direct endpoints on the LAN and over Tailscale, both answering in under a millisecond, and repointed the app past the cap.
It didn't work. Same error.
The real cause was that nothing had ever uploaded, because nothing had ever logged in. Zero successful logins and zero asset activity across the whole window, against fourteen failed attempts spread over three spellings of my own account: five on an address with a letter missing, five on an address that has never been the account, four on the correct one with the wrong password. All from my own devices — not an attack, just a typo repeated with conviction, which is the same thing I'd just done with the diagnosis. The one useful thing in the wreckage: five of those attempts arrived from a Tailscale source, proving the redirect had taken effect. The app was reaching the server perfectly. It just couldn't get in.
Housekeeping
Remapping the shares on Windows hit a wall I'd never met: the network folder is currently mapped using a different username and password. I was using the same credentials, which made the message look like nonsense. Windows allows one credential set per server, and a hidden background session — the administrative pipe share set up behind your back — can hold stale ones long after the drive that created it is gone. Deleting the visible mapping isn't enough; the invisible one has to go too, then everything gets remapped in one pass. Five drive letters back.
The 2 TB external SSD also came out of the server for good. Its temporary shares are retired, the server now exports exactly one directory, and the storage role has moved onto the NAS. The drive that came out of the NAS went into a USB dock as a cold-backup target.
One evening, and most of what I fixed had been broken or hidden by the fixing. The estate is more current than it's been in months, and slightly less trusting of the word latest.