I closed one session at 3:30 in the morning and opened another at 3:35. By five in the afternoon I'd deployed a new service, locked myself out of it, unlocked it by deleting its database, diagnosed my whole network, and then diagnosed a piece of it again and got a different answer. What ties the day together isn't the work. It's how sure I was at each step, and how little of that certainty survived the week.
The mount that pointed at nothing
Stirling-PDF is a self-hosted box of PDF tools — merge, split, convert, OCR, about two hundred and fifty endpoints of it. I'd used it before and wanted it on the server properly.
The boring part went fine. Its default port 8080 was taken and 8081 too, so it landed on 8090, LAN and Tailscale only. Deliberately not on the Cloudflare Tunnel: going public would put it behind Cloudflare's 100 MB request body cap, the exact limit that already breaks large uploads to my photo library. Off the tunnel I could set the max file size to 500 MB and mean it.
The interesting part was OCR. Upstream's own compose example mounts the Tesseract language data at /usr/share/tessdata, and that directory does not exist inside the image. I checked by running the raw image with nothing mounted — the real prefix is /usr/share/tesseract-ocr/5/tessdata, and the config file's own comment describes the setting as "relevant for Windows systems". So upstream's mount is inert: it creates a directory nobody reads, and OCR keeps working off the bundled languages.
Here's the trap. Having found the real path, the obvious move is to mount your host directory there instead. If that directory is empty, you have just masked every bundled language file with nothing. OCR doesn't error. It doesn't warn. It hands back a document with no text in it and you conclude your scan was bad. The fix was to copy the six bundled languages out of the running container first and mount that at the real path. Then I tested it instead of trusting the health check: a text-free image PDF went through OCR with the type forced, came back, and pdftotext pulled the words off it.
Locked out by a variable I'd just set
At quarter past four I installed the Windows desktop app and it refused to attach to the server: login is not enabled, set the security flag, restart. Fine.
Turning it on produced the first inversion of the day. The settings file already had login enabled. The environment variable was what had been forcing it off — I'd been overriding a correct config with a default I never chose.
The second inversion was worse. The initial-login variables that seed an admin account only run when the user table is empty. The database was hours old but not empty, so flipping security on created no account at all, and every attempt returned "invalid username or password" with nothing in the logs to distinguish it from a typo. The way out was to delete the database and its backup copy so the seeder had an empty table to write into — only acceptable because the instance was four hours old. With months of saved pipelines in there it would have been the wrong move entirely.
Somewhere in the middle of that, a loop trying a handful of likely default credential pairs against the login endpoint got stopped dead by a safety classifier. Which is correct. It was, structurally, credential guessing against a server, and "but it's my server" is exactly what an attacker would say too.
Afterwards the merge endpoint went from 200 unauthenticated to 401, then back to 200 with a bearer token. The status endpoint stays public even with security on, which is the only reason my uptime monitor didn't start screaming.
Ten minutes on Discord, and a different answer
At 17:08 I sent a message from my phone: "my cameras keep going offline every so often, find out why."
Ten minutes later I had a root cause, and it wasn't the network. Fifteen days of switch metrics on the five camera ports showed zero PoE drops, zero port errors, no link flaps. Uptime Kuma had failed a ping to those cameras exactly once in twenty-six days, and that was a separate event in a firmware window.
The evidence was somewhere I hadn't thought to look: Home Assistant's recorder database and its log file. The camera integration was logging 502 Bad Gateway from the gateway's own proxy path at exactly the minutes all five cameras went unavailable together — on this day alone at 00:12, 05:54, 08:01, 11:14, 11:19 and 14:34. A 502 there means the console's web proxy is alive and answering while the camera service behind it is dead or restarting. The first one was at 05:54 two days earlier, squarely inside the auto-update window, which makes an update to that service the prime suspect.
I couldn't act on it — SSH is closed on the gateway console, so rebooting it is a button in the app rather than a command. The lesson is the cheap one: ping monitors prove a device is reachable, not that the service on it works. Five cameras answering ICMP while the software recording them crash-loops looks perfectly healthy from outside.
Now the awkward part. Three hours earlier, a different session had watched those same five cameras drop together for fifty seconds at 14:38:45 and filed it under a wired-network story. The evening session's nearest logged 502 was 14:34. Same afternoon, same cameras, two confident explanations, and neither session knew the other existed — different project directories, separate memory. I still don't know whether that's one event or two.
The 90% that wasn't a problem
That afternoon session started because I asked, in one unpunctuated lump, whether my gateway and switches and APs were running too hot on memory and CPU, and also why everything went offline sometimes. The first reading looked alarming: 83.8% CPU, 91.3% memory on the gateway. A minute later CPU was 22.8%, and the 24-hour average was 33.1%. The number I'd panicked at was a spike I happened to catch. Memory was flatter still — 85.6 to 90.4 percent, averaging 88.4, unchanged across ten days of uptime. A leak climbs. This didn't.
The second question had a real answer. One switch had rebooted: the 2.5G Flex showed one day and seven hours of uptime, and all four access points showed one day and seven hours minus about two and a half minutes, which is how long PoE takes to come back and the APs to boot. Everything else in the rack had ten days. That switch carries all four APs, the NAS, the server, the camera switch uplink, and the only 10G path to the gateway. When it blinks, everything blinks.
Underneath that was something self-inflicted. All four APs were pinned to channel 6 — not auto, pinned — with three SSIDs on every one of them, so twelve BSSIDs were beaconing on a single 2.4 GHz channel at a minimum data rate of 1 Mbps, transmit power forced to the hardware floor of 6 dBm. The lounge AP saw 50% channel utilisation while generating 13% of it, and 22% of its frames needed retransmission. The 5 GHz radios sat at 1–5% with zero or one client each.
I wrote up seven ranked fixes and applied none of them, because they bounce every wireless client while the APs re-provision and I wanted a yes first. I never answered. The channel plan didn't land until the next morning.
Marking my own homework
Three things I was sure about on this day turned out to be wrong or unsafe.
The monitoring library I declared broken works fine. Adding an uptime monitor for the new service meant running my provisioning script, and it couldn't run: login died waiting for an event the current server no longer emits. So I called the library incompatible, rebuilt monitor creation on raw socket.io, added the monitor by hand, and corrected the note that said the library worked. Two days later it worked, repeatedly, against both instances. The real difference was how I was calling it — nested quoting through SSH into a container into a python -c string mangles things in ways that surface as strange runtime errors. Mount the script as a file and it's fine. I diagnosed a shell quoting problem as a library incompatibility, then wrote the mistake into the docs for someone else to trip over.
The switch power supply theory doesn't hold. I'd ruled out mains, PoE load and firmware and landed on the switch's own hardware. But the controller's event API returned nothing for switch and AP event types on this version, so I had no reboot reasons — the whole diagnosis rested on uptime correlation, and uptime-based reboot detection produces false positives, because a gap in metrics collection makes the counter read zero and look exactly like a reboot. There's no UPS anywhere in this lab either, and a pattern of reboots taking APs, switches and the gateway together within minutes. Power is now the leading hypothesis, not the thing I ruled out first.
"It's just buffers and cache" was too glib. I waved off the gateway's 88% memory as the OS counting reclaimable cache as used. On the access points that's demonstrably wrong: their used figure is already total-minus-available, and their root filesystem lives in RAM, so a chunk of it is committed and never coming back. Whether the gateway does the same arithmetic is still unsettled. I'd put it at about sixty percent, and sixty percent isn't a verdict.
Housekeeping
The one claim I hedged properly was the previous evening's cost work, written up in the morning: porting the photo-OCR pipeline off the CLI should take it from 22–25 cents a photo to under two. Should. That's arithmetic, not a measurement, and the ported code has never made a real API call — so there's a warning saying so at the top of the file, which is the habit I'd like the rest of this day to have had.
While I was asleep, the update pipeline read nine pending package upgrades, judged them all patch-level, refreshed a container, and the auto-docs pipeline noticed that container being recreated and regenerated its stack page at 4:33 in the morning. That part of the lab showed better judgement than I did at 4:17.