The day started at 00:23 with a disk shrink and ended with a read-only audit of the entire lab that refuted part of its own first draft. In between, three pieces of infrastructure got built and benchmarked rather than planned: Proxmox Backup Server on its own VM, NFS shared storage on the NAS, and a drift-detection mechanism that regenerates the cluster's own state document on a timer. The audit is the part worth sitting with. It went looking for whether the lab could survive a failure and found that most of the redundancy is real in form and not in substance: the second Uptime Kuma instance sends alerts down the same webhook as the first, the fallback guests all live on the same node they're meant to be a fallback for, and the backup server's datastore sits on the NAS it exists to hedge against.
The migration finished at 01:05
VM 200 got shrunk in place first, 00:23 to 00:41, 18 minutes of downtime. sda came out at 150G with root at 136G, swap active, 30 containers back up, and systemctl is-system-running reporting running with no failed units. pve02's pool dropped from 38.41% to 26.57%.
The shrink was the thing that made the migration cheap. qm migrate 200 pve03 --online --with-local-disks moved 150.0 GiB instead of 477 GiB, took 16 minutes 27 seconds, and cost zero downtime — uptime read 25 minutes afterward, so the machine never rebooted, and Immich answered HTTP 200 throughout the transfer. pve02 went from 38.41% to 1.98%, pve03 from nothing to 25.56%.
One thing surfaced mid-transfer that would otherwise have gone unnoticed: pve03 was running on its 1 GbE standby, not its 2.5G primary. The enx74****5c7c leg was down with a link failure count of zero, which is the awkward part — it hadn't failed, it had simply never come up in the bond. The transfer held about 110 MB/s as a result, roughly 890 Mbps, which is exactly what a gigabit link gives you and is also close enough to plausible that nobody would question it without checking. Repaired the same day.
PBS is real now
Proxmox Backup Server went in as VM 144 on pve02, PBS 4.2.5-1 at x.x.0.44, registered cluster-wide as pbs-n5hq. The datastore is a 1 TiB thick iSCSI LUN from the NAS SSD pool, deliberately not the HDD pool — that pool suffered unrecoverable btrfs metadata corruption on 2026-08-09, a 154-generation gap caused by lost writes on an unclean shutdown.
Metadata change detection does what it claims. The third backup of CT140 moved 32 bytes out of 647.413 MiB in 0.98 seconds. The nightly job first fires at 01:00 on the 12th.
Nothing got retired to make room for it. n5-backup.sh and the 02:30 vzdump-to-CIFS job both stay exactly as they were, and the whole thing runs as a parallel trial rather than a cutover. Separately, that 02:30 run on the 11th was the first fully successful cluster-wide vzdump across all three nodes, which closes a gap nobody had named out loud: portal, stirling-pdf and forgejo on pve02, and VM 200 on pve03, had never once had a successful scheduled backup.
Restores, proven rather than assumed
The restore gate has been open in the backlog for a long time on the grounds that a backup you have never restored is a hypothesis. Three restores ran on the 11th. pct restore brought CT140 back as VMID 199. qmrestore brought VM 200 back as VMID 299 — 49 GB of archive expanding to 161 GB read at 44.3% sparse, about 13 minutes, and it booted. PBS restored CT140 in 4.8 seconds at 135 MiB/s.
The honest footnote is that one of those checks got downgraded afterward. The CT140 restore was reported as matching on file count, 16,875 files, and a later review ruled the file-count comparison itself unverified. The restore succeeded. The specific evidence offered for it was weaker than it read.
NFS, and a trade taken on numbers
Shared storage went in as an NFS share on the NAS SSD pool, exported to all three nodes and added to Proxmox as nas-vm with --content images only. Containers are deliberately excluded.
NFS won over iSCSI plus LVM for one reason: iSCSI with LVM has no snapshot support at all, and losing snapshots would kill pct rollback 133 clean, which the nightly update rehearsal depends on.
The export runs async, which is a trade rather than a default. Sequential writes go 95.3 MB/s synchronous to 227 MB/s async. The number that decided it is 4k fsync: 849 kB/s to 14.4 MB/s, a 17× difference, with per-sync latency dropping from 4.8 ms to 0.28 ms. The cost is a volatile window of roughly 30 seconds, up to about 1.5 GB, if the NAS dies uncleanly. A node dying loses nothing. Verified across all three nodes afterward at 13.2, 14.0 and 13.0 MB/s, within 8% of each other.
VM 200 stays on local NVMe and is not moving. Immich's Postgres, Prometheus and Loki are all fsync-heavy, and that workload is the one that would feel the network round trip.
Four waves, and the shape they all found
The whole-lab analysis ran as six retrieval agents, four adversarial critics, three target-state reviewers and two planners, all read-only against the running systems. Its own status line is unambiguous: analysis complete, nothing executed.
The cluster reads healthy. Three nodes on pve-manager 9.2.10, quorate 3 of 3, quorum 2, and both NVMes checked report zero errors. HA is unconfigured — /etc/pve/ha/ is empty on all three nodes.
What the analysis actually found is one flaw wearing three costumes. kuma1 and kuma2 exist as independent watchdogs, and they push to a byte-identical Discord webhook, so the thing that makes them independent is the one thing they share. The guests meant to act as fallbacks all sit on pve01. The backup server's datastore lives on the NAS, which is one of the things a backup server exists to survive. And the PBS encryption key is Option B — no master key, no paperkey, both copies inside the lab. Every one of those is redundant in the diagram and single-pointed in practice.
The critique wave then refuted the retrieval wave, which is the part that justifies running adversarial agents at all. Wave 1 concluded the migration was complete and clean with no split-brain. Wave 2 found a roughly four-and-a-half-hour dual-run: after the planned reboot at 00:24:59, all four retired origin containers came back up alongside their pve02 replacements and stayed up until they were stopped between 05:12 and 05:47. That is the root cause of a stirling-pdf H2 database divergence, 73728 bytes on one side against 106496 on the other.
Two retrieval agents inside that same analysis also concluded n5ubuntu was still bare metal. It is VM 200 on pve03, and has been since 01:05 that morning.
Documents that disagreed with the machines
A pattern turned up often enough to be worth naming: the hand-written documents were wrong and the generated ones were right. The runsheets said pve02 carries VM 200; it's on pve03, which cluster-state.md had correct. A memory file and a project note both said git.n5hq.me returns 502; it returns 200. The PBS decision document said "nothing built" at the top while its own body described a built VM. kuma2's runsheet claimed its own webhook, "a copy, not a relay" — it is the same webhook.
So the drift mechanism got built rather than proposed. gen-cluster-state.sh regenerates ~/docs/cluster-state.md hourly, docs-drift-check.sh writes ~/docs/drift-report.md daily at 06:30, and drift-checks.sh holds the fact registry, 14 facts at build time, each one a claim checked against the running lab plus a grep for documents still asserting the old version. This is the one piece of that project that exists rather than being written down for later.
The monitoring audit, which is a plan and only a plan
All 63 monitors got audited, 48 on kuma1 and 15 on kuma2. Nothing has been executed. The findings are bad enough that writing them down without acting on them deserves to be uncomfortable.
A failed backup notifies nobody. The backup scripts do push a real down status; monitors 42 and 47 have maxretries=1, which sends the first failure to PENDING rather than important, so two consecutive nightly failures are needed before the first Discord message. A WAN outage produces 12 attempted alerts and zero delivered. A gateway outage produces 14 attempted and zero delivered — both instances POST to Discord over the dead link, catch the error, log it, and never retry or queue. Thirteen monitors treat 401 and 403 as up.
The mechanism worth the red flag: updateMonitorNotification() runs an unconditional, untransacted DELETE before re-inserting, so a failed editMonitor call still wipes a monitor's notifications. That is how monitor 45 lost its notifications on 2026-07-28.
The review of the plan caught two errors in the plan itself. A resendInterval of 60 on push monitors with a 25-hour interval would have meant about 62 days between resends, not hourly. And two exporter targets pointed at addresses bound to localhost that would never have connected.
One belief retired
The claim that pve01's NVMe has 130 unsafe shutdowns and therefore power is the leading hypothesis does not hold. SMART counters are lifetime-cumulative and all three boot drives are second-hand, so those numbers describe the previous owner. pve02's SSD logged 755 unexpected power losses across 5.06 years of power-on time before it ever arrived here. My own count since ownership is fewer than five per system, most of them deliberate auto-power-on testing.
The lack of a UPS is still a real exposure — the btrfs corruption on the 9th came from exactly that. The drive counters were never evidence for it.
---
So: the migration finished, three things got built and measured, and restores stopped being a hypothesis. Against that, an audit that went looking for resilience found the same shape three times over — a second copy that depends on the first, a fallback that shares a host with the thing it backs up, an encryption key whose only two copies sit in the building they protect. None of that is fixed. It's written down with numbers attached, which is a start and not a fix, and the plan for the monitoring half is explicitly a plan with nothing executed behind it.