Eric Li DevOps & Network Architecture

Tearing Down Talos and Moving the Workstation VM

Homelab · 2026-10-14 · Wednesday · 1:32 AM · 3 min read · Eric Li

A short night of infrastructure work, from about 00:16 to 01:34.

Removing the Talos cluster

I shut the Talos sandbox down on the 10th to save power. Tonight I removed it. After a read-only look at what was on pve04, I destroyed VMs 1000 to 1002 with their disks and deleted the Talos ISO. I also removed the vzdump exclude entry for those IDs, then confirmed the zvols were gone and no guests were left. There was no final backup because they were already excluded from the nightly job. A sweep for leftovers found nothing on pve04 itself.

At 01:01 I shredded the talosconfig and kubeconfig files on the workstation VM, removed the empty directories and took the exports out of my bashrc. The runsheet records both steps.

Some leftovers remain elsewhere. NetBox still has three VMs and three IPs for the cluster. The sync tool never deletes, and its stale-cleanup run would also deprecate unrelated objects, such as pve03's address, so I won't run it blindly. UniFi probably still holds three MAC entries in the admin allowlist and possibly DHCP reservations for the old addresses.

Moving the workstation VM to pve04

At 01:02 I decided to move my workstation VM, VM 209, onto pve04 for now. The HA rule on it is strict, so pve04 had to be the top priority. I created a ZFS replication job to pve04 on the five minute schedule, and the first 11 GB sync took 59 seconds. Then I changed the node affinity so pve04 ranks first. HA live-migrated the VM and my SSH session survived. After the move the first replication cycle succeeded on both jobs in about 4 seconds each. To revert, set the rule back to pve01 and pve03, lower the CPU count to 6 or fewer, and delete the extra replication job.

I accepted one risk: the broken self-fence problem I logged earlier is the same on pve04 as it was on pve01.

Running out of memory

I gave the VM more resources because the old one ran out of memory. It had 5 GiB and many processes running at once, and at about 00:11 the kernel's OOM killer shut down several of them. The reboot at about 01:15 wiped /tmp.

So I gave it everything: 12 vCPUs and 12 GiB of RAM. pve04 has 12 threads and 15.3 GiB, which leaves roughly 3 GiB for Proxmox and 2 GiB for the ZFS cache. The catch is that pve01 and pve03 have six threads each, so a 12 vCPU VM can't start on either. While it's at 12 vCPUs on pve04, HA failover for the workstation VM is effectively off. Afterwards, peak use was 6.1 GB of 11.9 GB with free memory never below 5.9 GB, and the OOM killer hasn't returned.

Still open