Eric Li DevOps & Network Architecture

A Dead Motherboard, a New ISP and Camera Storage Numbers

Homelab · 2026-10-12 · Monday · 8:33 PM · 6 min read · Eric Li

Work ran from about 03:08 to 06:26, then 08:09 to 14:43, with a short stretch in the evening from 19:38 to 20:33.

The dead sim rig board

The sim rig's X570-I board was swapped for a B550 a couple of years back, and I wanted to know if the old one was fixable. I photographed it and dug up the manual and schematic. My first read was three burn sites and not worth fixing. That was wrong. The darkening around the CPU power joints is factory flux. The real damage is at one corner, where the plating has torn on a mounting-hole ring. The black smear across the back is smoke.

The schematic corrected my next guess too. I'd blamed the fan driver, but a clear photo of the back shows a charred hole and a burnt via in the board itself, with the pad burned bare. The burn sits in the 12 V input to the chipset's 1.2 V supply, and one cap there is heat-stained. My hypothesis, which I haven't confirmed, is that board flex at the torn mounting hole cracked an input capacitor, which shorted and burned. The CPU, RAM, GPU and PSU all work fine in the B550, so only the board died. First I'd check continuity across the ferrite bead, then a MOSFET drain-to-source reading, then the rail to ground.

These boards are hard to find now, so a repair is worth considering. A full set of parts from one overseas order comes to about 9.50 in US dollars plus shipping, roughly 40 to 60 Australian dollars delivered. The supply controller is the exception. Locally one MOSFET and the controller are out of stock, though substitutes exist, and the controller probably survived anyway. I also found a board-level repair shop that takes send-ins from anywhere in Australia on a no-fix-no-fee basis. Their page doesn't mention charred boards or burnt vias, so I have questions to ask first. Nothing is sent or bought.

Switching ISP

I signed up with Leaptel this morning, mainly for the service guarantee, since I was paying full price at Swoop. The gateway's WAN was already set to DHCP with no VLAN tag, PPPoE or MAC clone, and nothing in the vault was tied to the old ISP. I have no port forwards because ingress is Cloudflare Tunnel only. So the only thing to do at cutover was power-cycle the ONT. Leaptel defaults to carrier-grade NAT, and I opted out.

At 11:44 I ran a health check. The WAN has a public address now, and the speed test gave 939 down and 97 up at 2 ms. The WAN flapped three times between about 10:41 and 11:39 while the service migrated and CGNAT came off, though the router never rebooted. Its WAN port is forced to 1000 Mbps with autoneg off, so I'll switch it to Auto if the drops return. Both Cloudflare connectors dropped for a couple of minutes and reconnected on their own, and all ten public hostnames answered in under a quarter of a second. The cluster was quorate with five votes, HA was healthy and 21 containers were running on the Ubuntu server. The only things down were guests I'd stopped on purpose.

The remote download box

The mount to my remote download box survived the three disconnects and healed each time. Radarr and Sonarr still reported they couldn't reach the download client, with 504s from the box's proxy. The disk quota there is at 97.3%, but that isn't the cause. I restarted the client myself at about 12:43. It sat at 100 to 130% CPU for several minutes loading roughly a thousand torrents while the shared host's load average was around 90. Both apps' connection tests failed at first and passed after a few minutes. By 13:11 everything was fine. I can't prove the root cause, but an overloaded shared host is the likeliest, and the new IP isn't to blame. I need to clear out old completed Linux ISOs soon, because the quota broke the stack at 99.9% on 4 September.

Would Ansible earn its place

I looked into whether Ansible has a strong use case in the lab. It does, but a narrow one: a baseline for the four nodes and the Pi. That means sshd hardening, chrony, apt repos, node-exporter, vzdump hooks and base packages. When I reinstalled all four nodes on the 27th I restored about a dozen kinds of config by hand, and drift followed. pve02 had the enterprise repos left on, and node-exporter was down on two nodes for nine days. It's a weak fit for the Docker stacks, the update pipeline, the hotel switches and UniFi. I created a project for it with a phased plan and installed nothing.

How long the camera footage lasts

For the move to a UniFi NVR with a 2 TB drive, I pulled real numbers from the Protect API. Five cameras record continuously, one 4K and four 1080p. Their configured bitrates add up to a ceiling of 56 Mbps, but the actual average is about 15 Mbps, below my earlier estimate of 20 to 40. The current 1 TB disk is 96% full, which is about 4 days at high quality or 15 at low. On the new drive I'd get about 8 to 9 days at high quality with Protect's usual split, or about 11 days if everything is high quality. The 4K camera is roughly 47% of the high-quality footage, so it's the biggest lever. Drive wear isn't a concern, with about 65 TB a year written against a 120 TB rating.

Fixing my secrets script

Late in the evening I found why my Proton Pass login kept getting logged out. The wrapper script I use to inject secrets never loaded the key-provider setting that a comment in my bashrc said it did, so non-interactive shells fell back to the keyring and forced a logout. I added three lines, tested twice with the variable unset, and updated the runsheet.

Still open