RSS/AtomHomelab — Agentic AI vs routine maintenance + Rook/Ceph upgrade
Every now and then, one of the homelab VMs crashes. An ideal problem for agentic AI with a human-in-the-loop?
I did some research already on why this happens and it’s due to OOMKILL at the hypervisor level. Rebooting fixes it - until next time. A finite choice human-gated solution might render a screen that looks something like this one, from “King-Size Homer”:
Which could then be approved by the operator (or not):
Building this would have been a complete waste of time, effort and tokens. The problems and solutions are known, so let’s just fix this once and for all, and save our agentic loop for a real problem.
What keeps breaking?
Looking through the kernel errors, I could see a VM being OOMKILL‘ed about once every couple of months. The reason for this is very simple: The hypervisor is overcommitted and has swap disabled. This was originally done for performance reasons but it means a temporary increase in memory usage will result in the kernel reclaiming memory the only way it can - by killing processes.
The lab runs mostly on mini PCs with limited memory - 16-32GB. The 32GB boxes lose 4GB to onboard graphics and are always the PCs with problems since most of the work is allocated to them.
TODO.exe
This little cluster has been running for a couple of years without much maintenance being done, so it’s due for some updates. I came up with a five-step plan to fix the memory issues once and for all, as well as some routine upgrades:
- Enable swap at the host (hypervisor) level
- Upgrade Rook/Ceph
- Upgrade Kubernetes (K3s)
- Reduce memory allocations for VMs to fix excessive overcommit
- Use cgroups to reserve memory for the host (hypervisor)
Enabling swap
This was the most straight-forward fix since the hypervisors are managed with ansible. I added some code to my SOE, activated for physical servers only (hypervisors, not Kubernetes nodes) and moved on quickly:
Vars:
swapfile_path: "/swapfile"
swapfile_size: 8G
# prefer dropping page cache over swapping out guest RAM under reclaim.
# swap here is an emergency valve to buy the kernel time, not memory
# we intend to run in
swappiness_value: 10
Tasks:
- name: Create swapfile
community.general.filesize:
path: "{{ swapfile_path }}"
size: "{{ swapfile_size }}"
- name: Set swapfile permissions
ansible.builtin.file:
path: "{{ swapfile_path }}"
owner: root
group: root
mode: '0600'
- name: Check whether swapfile is already active
ansible.builtin.command:
cmd: swapon --show=NAME --noheadings
register: swapfile_active
changed_when: false
- name: Format swapfile
ansible.builtin.command:
cmd: "mkswap {{ swapfile_path }}"
when: swapfile_path not in swapfile_active.stdout_lines
- name: Enable swapfile
ansible.builtin.command:
cmd: "swapon {{ swapfile_path }}"
when: swapfile_path not in swapfile_active.stdout_lines
- name: Persist swapfile
ansible.posix.mount:
path: none
src: "{{ swapfile_path }}"
fstype: swap
opts: sw
state: present
- name: Configure swappiness
ansible.posix.sysctl:
name: vm.swappiness
value: "{{ swappiness_value }}"
state: present
reload: true
Upgrading Rook/Ceph
Rook/Ceph is used in the homelab as replicated, Cloud-Native Storage for Kubernetes.
Upgrading Rook/Ceph is a serious undertaking. It’s actually not dissimilar in scale to upgrading Confluent Platform (my day job), with a lot less hand-holding. Fortunately, there’s no valuable data on my Rook/Ceph cluster since I just use it to host a few crypto nodes.
The notes below supplement the docs:
- Cluster must be healthy before starting
- Upgrade one minor release at a time, eg 1.16.x to 1.17.x
- RTFM!
- No really. RTFM. For each minor release you will upgrade through
- Upgrade
rook-cephfirst, thenrook-ceph-cluster - You must install/upgrade
ceph-csi-driversif you want to be able to access PVs (you do) - Wait for each upgrade step to finish before moving on
Helm tips:
- Helm charts do most of the work
- Explore available versions:
helm search repo rook-release/rook-ceph --versionshelm search repo rook-release/rook-ceph-cluster --versions
- List what’s installed:
helm list -A
Example (upgrade v1.16.5 to v1.17.9):
# Upgrade the operator
helm upgrade --version v1.17.9 --namespace rook-ceph rook-ceph rook-release/rook-ceph
# WAIT for all pods to show correct version
watch "kubectl -n rook-ceph get deploy \
-o custom-columns=NAME:.metadata.name,ROOK:.metadata.labels.rook-version"
# Install the CSI driver now, (omitted in my lab environment)
# Upgrade the cluster itself, using saved values from the original deploy
helm upgrade --version v1.17.9 --namespace rook-ceph rook-ceph-cluster rook-release/rook-ceph-cluster -f helm_values/rook_cluster_values.yaml
# WAIT - watching the logs
kubectl logs -n rook-ceph rook-ceph-operator-5cc9ff9d9b-9zp7k -f
Cluster topology
I access the Rook/Ceph storage as an external cluster at the point of use. The Rook/Ceph storage cluster itself lives in a separate, dedicated Kubernetes cluster for ease of management and sharing.
Ceph-CSI driver problems
The Kubernetes cluster accessing the Rook/Ceph storage is upgraded with a similar process to the storage cluster. Since this is where I want to use the storage, it’s now mandatory to install the Ceph-CSI driver helm chart.
The Ceph-CSI driver actually failed for me and crashed my workloads:
- First a couple of pods died with
OOMKILLED(from Kubernetes) - Thinking the pods must have got stuck for some reason, I rebooted the entire Kubernetes cluster
- At this point, all workloads that attempted to mount storage were marked as
unknownorContainerCreatingand were blocked from starting or progressing.
To troubleshoot this, I did some investigating on the stuck pods and found:
kubectl describe pod <pod-name> -n <namespace>
...
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 18m (x119 over 4h4m) kubelet MountVolume.MountDevice failed for volume "pvc-fe440598-6203-4a69-8aad-4032795274e1" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rook-ceph.rbd.csi.ceph.com not found in the list of registered CSI drivers
Warning FailedMount 9s (x14 over 12m) kubelet MountVolume.MountDevice failed for volume "pvc-fe440598-6203-4a69-8aad-4032795274e1" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rook-ceph.rbd.csi.ceph.com not found in the list of registered CSI drivers
Accompanied by no drivers running:
kubectl -n rook-ceph get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
rook-ceph.cephfs.csi.ceph.com-nodeplugin 3 0 0 0 0 <none> 34h
rook-ceph.rbd.csi.ceph.com-nodeplugin 3 0 0 0 0 <none> 34h
Eventually I found the problem - a mismatch between service account names in ceph-csi-drivers chart vs the deployed cluster:
kubectl describe ds -n rook-ceph rook-ceph.rbd.csi.ceph.com-nodeplugin
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 21m (x37 over 4h9m) daemonset-controller Error creating: pods "rook-ceph.rbd.csi.ceph.com-nodeplugin-" is forbidden: error looking up service account rook-ceph/rbd-nodeplugin-sa: serviceaccount "rbd-nodeplugin-sa" not found
Warning FailedCreate 5m46s (x18 over 16m) daemonset-controller Error creating: pods "rook-ceph.rbd.csi.ceph.com-nodeplugin-" is forbidden: error looking up service account rook-ceph/rbd-nodeplugin-sa: serviceaccount "rbd-nodeplugin-sa" not found
Available service accounts:
kubectl get serviceaccounts -n rook-ceph
NAME AGE
ceph-csi 34h
default 550d
objectstorage-provisioner 548d
rook-ceph-cephfs-csi-ceph-com-ctrlplugin-sa 90m
rook-ceph-cephfs-csi-ceph-com-nodeplugin-sa 90m
rook-ceph-cmd-reporter 548d
rook-ceph-default 548d
rook-ceph-mgr 548d
rook-ceph-nvmeof 34h
rook-ceph-osd 548d
rook-ceph-purge-osd 548d
rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa 90m
rook-ceph-rbd-csi-ceph-com-nodeplugin-sa 90m
rook-ceph-rgw 548d
rook-ceph-system 548d
Note that rbd-nodeplugin-sa has a completely different name on the system: rook-ceph-rbd-csi-ceph-com-nodeplugin-sa. The same applies to three other service accounts as well.
To fix this I had to add configuration for the service account to the default helm values from the docs by adding serviceAccountName to fix the four broken accounts. I also had to helm uninstall the ceph-csi-drivers chart and reinstall it for this to take effect. It was not enough to just helm upgrade:
# Rook-compatible default values for the ceph-csi-drivers Helm chart.
#
# Use these values when installing or upgrading the ceph-csi-drivers chart
# alongside the rook-ceph operator chart. The driver names must match the
# provisioner names used by Rook (e.g. in StorageClasses and VolumeSnapshotClasses).
#
# If the Rook operator is installed in a namespace other than rook-ceph, replace
# "rook-ceph" in the driver names below with your operator namespace.
operatorConfig:
namespace: rook-ceph # namespace:operator
driverSpecDefaults:
imageSet:
name: rook-csi-operator-image-set-configmap
nodePlugin:
priorityClassName: system-node-critical
controllerPlugin:
priorityClassName: system-cluster-critical
drivers:
rbd:
enabled: true
name: rook-ceph.rbd.csi.ceph.com # csi-provisioner-name
nodePlugin:
serviceAccountName: rook-ceph-rbd-csi-ceph-com-nodeplugin-sa
controllerPlugin:
serviceAccountName: rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa
cephfs:
enabled: true
name: rook-ceph.cephfs.csi.ceph.com # csi-provisioner-name
nodePlugin:
serviceAccountName: rook-ceph-cephfs-csi-ceph-com-nodeplugin-sa
controllerPlugin:
serviceAccountName: rook-ceph-cephfs-csi-ceph-com-ctrlplugin-sa
nfs:
enabled: false
name: rook-ceph.nfs.csi.ceph.com # csi-provisioner-name
nvmeof:
enabled: false
name: rook-ceph.nvmeof.csi.ceph.com # csi-provisioner-name
No data was lost during the upgrade, although I did have a brief unplanned outage while I worked out the service account fix.
Upgrading Kubernetes (K3s)
I’m using K3s as a light-weight Kubernetes distribution and manage it with k3s-ansible.
Like Rook/Ceph, Kubernetes should also be upgraded through each minor version, in my case there were three minor version updates to apply.
Hats off to the K3s developers, this was incredibly easy having just finished a Rook/Ceph upgrade:
- Update the
k3s-ansiblecodes - Change
k3s_versionto desired version in inventory file - Run the upgrade playbook (eg
ansible-playbook playbooks/upgrade.yml -i ../inventories/k3s_apps.yml) - Wait for stability, then rinse and repeat
It really was that easy once I figured out I now needed to install the ansible.utils galaxy module.
Reducing memory allocations
With the upgrades done and systems stable, it was time to redistribute and reduce overall memory usage.
On PCs with 28GB usable RAM, there was an overcommit of approximately 6GB which is far too high. 3GB is pushing it and 0GB of overcommit would be better.
When the cluster was first setup, I remember Rook/Ceph needing a lot more memory but it seems this hasn’t been the case for a long time and much of that memory was sitting idle. Since I control my VMs with libvirt/KVM + ansible, changing memory allocation is a very simple process:
- Update
host_varsfor hypervisor with new memory amount - Shut down affected VM(s)
- Run ansible
- VM will be restarted with the updated memory allocation
The main reduction was for the Rook/Ceph VMs: from 20GB to 12GB. Some of the future-use clusters were also downsized, from 4GB to 2GB.
An hour or so later the job was done and the VMs worked fine with their new memory allocations.
Using cgroups to limit memory usage in machine.slice
Everything we have done so far has been a mitigation for the crashes - this step limits the blast radius of OOMKILL: When machine.slice is full, it will kill a VM, not a host process like sshd.
Linux cgroups are a kernel feature that we can use to restrict the memory available to programs. Limiting the total memory available to VMs in machine.slice lets us support a modest overcommit more safely without the risk of the hypervisor running out of memory.
We can deploy the change very easily with ansible.
Vars:
# total held back from machine.slice
host_reserve_mb: 2048
# unreclaimable floor, measured
host_min_mb: 512
# best-effort protection
host_low_mb: 1024
# headroom for fwupd/libvirtd spikes
user_low_mb: 256
# unreclaimable floor for ssh/login sessions
user_min_mb: 64
# throttle band: gap between MemoryHigh and MemoryMax
vm_high_headroom_mb: 1024
Tasks:
- name: Ensure slice drop-in directories exist
ansible.builtin.file:
path: "/etc/systemd/system/{{ item }}.d"
state: directory
owner: root
group: root
mode: '0755'
loop:
- machine.slice
- system.slice
- user.slice
- name: Configure machine.slice memory limit
ansible.builtin.copy:
dest: /etc/systemd/system/machine.slice.d/50-memory.conf
content: |
[Slice]
MemoryAccounting=yes
MemoryHigh={{ (ansible_memtotal_mb - host_reserve_mb - vm_high_headroom_mb) }}M
MemoryMax={{ (ansible_memtotal_mb - host_reserve_mb) }}M
owner: root
group: root
mode: '0644'
notify: systemd daemon reload
- name: Protect host services from VM memory pressure
ansible.builtin.copy:
dest: /etc/systemd/system/system.slice.d/50-memory.conf
content: |
[Slice]
MemoryAccounting=yes
MemoryMin={{ host_min_mb }}M
MemoryLow={{ host_low_mb }}M
owner: root
group: root
mode: '0644'
notify: systemd daemon reload
- name: Protect user slice services from VM memory pressure
ansible.builtin.copy:
dest: /etc/systemd/system/user.slice.d/50-memory.conf
content: |
[Slice]
MemoryAccounting=yes
MemoryMin={{ user_min_mb }}M
MemoryLow={{ user_low_mb }}M
owner: root
group: root
mode: '0644'
notify: systemd daemon reload
Checking the machine.slice counters a while after applying the fix: max, oom and oom_kill are all zero, so nothing has been killed. The 62k high events are the throttle doing its job rather than a problem, though they suggest my 1GB of headroom below MemoryMax is on the tight side. These are cumulative counts since boot, not bytes. Long term, its worth collecting these figures directly, in prometheus.
# 32GB system RAM server
$ cat /sys/fs/cgroup/machine.slice/memory.events
low 0
high 62467
max 0
oom 0
oom_kill 0
oom_group_kill 0
Retrospective
I completed the five-step TODO list over the weekend and systems have been stable all week with no data loss so I’m calling this a success. It remains to be seen if Kubernetes nodes will still crash every couple of months, but I’m pretty sure this particular problem is fixed, without the need to build a glorified AI-enabled cron job to do the occasional reboot.
The Rook/Ceph upgrade was quite an eye opener. It took longer than I would have liked to figure out what was going on, and although there was no data loss, there was an unplanned outage.
Since this is a lab environment, it’s no big deal but if this had been an enterprise system we would now be proceeding with caution. This is why we have dev environments - and support contracts.
Farewell, Rook/Ceph
I’ve run this five node Rook/Ceph cluster for over two and a half years now, and it’s worked really well on my 1GbE network.
I’ve been able to reboot or power off nodes without fear and watch Rook/Ceph recover or rebalance itself when OSDs are back online. It’s really fun to watch it do this and I’ve learned a lot from running the cluster all this time.
With that said, the upgrade process makes me a little nervous about using Rook/Ceph on a system that needs to be running 24/7 without having additional lab environments to model the upgrade.
Considering I don’t actually have any data I care about in the Rook/Ceph cluster, there’s no justification for building out more environments and doing additional testing for upgrades.
My original plan was to just to try out Rook/Ceph replicated storage in lab environment with my sample crypto servers to see how it works in the real wold and maybe add more workloads if useful.
Now that I’ve completed this research task and hard drive costs have quadrupled, it’s time to shut down Rook/Ceph before it burns out the SSDs for the simple reason that I don’t have a workload to justify using it at the moment:
- Crypto node data is replicated at the node level and freely (if slowly) available from the peer-to-peer network. In an enterprise context such as an exchange, the solution needed is multiple independent nodes with their own storage, not “floating” storage
- Messaging systems such as Kafka are best served by local NVMe disks - in my case, the outage I had took all PVs offline at the same time. This would have taken down Kafka if I was running it as its pods would simply not have been able to start. The choke-point of the CSI driver defeats the decentralization of both Ceph and Kafka
- Databases such as postgres are also best served by local NVMe for speed. This removes the network path from all read/write activity
- S3-like storage is a genuine loss, although I don’t currently use this capability and alternatives such as Garage exist
Rook/Ceph really shines in environments where the data being stored is actually valuable and there is some tolerance in terms of latency: Think git, documents, backups, etc.
So what do we do instead?
- topolvm for capacity limit aware local node storage
- Node Affinity to ensure nodes run adjacent to their storage
- A simple NAS somewhere on the network
- Backups/snapshotting for quick recovery (eg for crypto nodes, same with more safeguards for postgres)
- Assess the storage needs of new workloads on a case-by-case basis
If replicated storage requirements come up again in the future, then it’s time to upgrade the network backbone to at least 10GbE, add more PCs for additional environments and re-test Longhorn for good measure. I can’t see this happening any time soon though - I already spent a fortune on mini PCs and there are no power sockets left either.
Somewhat ironically, turning off Rook/Ceph will also free up approximately 12GB system RAM on each hypervisor, down from 20GB before this exercise.
The very problem we were trying to fix.