Eric Li DevOps & Network Architecture

Forgejo 15.0.10, a Failed Backup Job and an Offline Installer

Homelab · 2026-10-20 · Tuesday · 2:14 PM · 5 min read · Eric Li

Four jobs today, from just after midnight to 14:14, with a long gap in the morning.

Forgejo upgrade

I was on 15.0.6 and 16.0.5 had been released. I stayed on the 15 line. Version 16 reaches end of life on 29 October, while the 15 LTS line is supported until 15 July 2027. Versions 15.0.8 and 15.0.9 each fixed a critical remote code execution bug, so 15.0.6 had to go.

Releases from 16 onward stop following redirects on mirrors, so I checked all 40 pull mirrors and 6 push mirror targets first. None redirect. I took a ZFS snapshot of CT 143 called pre-forgejo-15-0-10, pinned the image to :15.0.10 instead of the floating :15 tag and started it. Downtime was about 10 seconds. git.n5hq.me returned 200, and the API, SSH clones, the repo count of 55 and all 28 doctor check checks were fine.

A schema migration ran even though this is a patch release, so switching the tag back would not undo it. The snapshot is the rollback. The SQLite check also found 5 leaked free pages that were already there before the upgrade. CT 143 turned out to be on pve03, not pve02 as the runsheet said, so I corrected that.

The backup job had been failing

My first read was that CT 143 had no app-level backup. The job exists, but on n5ubuntu, and it was broken. Every call it made was hardcoded to ssh pve02. The NAS dropped out on 10-18 and the job hung on the mount for about 22 hours. On 10-19 HA moved CT 143 to pve03 and the job could no longer find the container. The newest backup set before today was from 10-17.

The scripts now look up the hosting node at run time, the NAS mount check gives up after 20 seconds, and the service has a 3-hour timeout. A manual run finished at 03:24 with a full set on the NAS, and the Kuma monitor went green. I also deleted two push mirrors that were already failing, one pointing at an archived GitHub repo and one with a rejected SSH key. Both GitHub copies matched, so nothing was lost. I added drift checks for the Forgejo version and the pinned tag.

A runbook for the hotel network

I decided the hotel needs a network runbook that its owner holds, and spent an hour on method and tooling.

For terms I'll use AWS's split: a runbook fixes a known task, and a playbook works out an unknown fault and ends at a runbook. Each entry gets an owner, last-reviewed and last-tested dates, a trigger, impact, steps, a check, a rollback and escalation, which is close to my runsheet format.

The tooling is Markdown in Git, printed to PDF, with NetBox for addressing and cabling and Oxidized for config backups. Oxidized covers the Aruba and H3C switches but not the Sophos, which has its own scheduled backup. The runbook must never live on the hotel network, so there will be a printed binder and an off-site PDF.

What I hold today is all read-only snapshots from September and October. There are no config backups, no IP and VLAN register and no split of responsibilities between the owner, the hotel's IT provider and me.

The leads tool for the venue

I finished hardening the leads tool for the function venue client. The fetcher now refuses loopback, private and link-local addresses, and downloads have a hard deadline. The first Windows trial failed 3 of 2,334 tests, mostly a timeout that relied on Linux behaviour. It also showed that re-verifying with unreachable pages could drop human sign-off records. After the fixes, 2,397 tests pass on Windows. A live run against the venue's own site made 10 requests after reading its robots.txt and found exactly the three expected contacts. The second Excel test pack opened with no repair dialog, kept all 14 dropdowns and calculated every formula correctly.

I then built an offline installer of about 37 MB, with a pinned Python installer and 18 packages. Staff data sits in a folder that upgrades never replace. On the test VM, a fresh install with the internet blocked took 68 seconds. An upgrade and a repair took about 37 seconds each with staff files byte-identical, and following the manual install document took 79 seconds.

The Windows test VM

The test VM on pve04 was slow because Windows was doing its first-boot work during every trial. The snapshot had been taken right after install. The 99.93% memory reading in Proxmox was a display quirk with ballooning off, but the CPU load was real.

A rollback restores the CPU and memory stored in the snapshot, so my first resize was undone. A second patching job I started ran against the VM at the same time and rolled it back again, which cost hours. I finished the patching by hand and retook the clean snapshot at 07:09 at 10 vCPU and 10 GB. At that size my workstation VM 209 can't fail over to pve04, and I accepted that for now.

Still open