geoffwilliams@home:~$
RSS/Atom feedRSS/Atom

Homelab - Migrating off Rook/Ceph

In my previous post, I decided my Rook/Ceph experiment with replicated storage is finished, and it’s time for a much simpler homelab with LVM localpv.

With five storage nodes using dedicated devices, I want to migrate off Ceph with zero data loss and some small outages. This will free up a bunch of CPU, RAM and storage hardware all over the cluster which gives me a lot more room for more fun systems. Notably, it will free up an entire N100 mini PC and some SSDs to use as a ZFS-enabled NAS.

Setting up LVM localpv

In the Kubernetes cluster where I wanted to use the storage, I just needed to install the CSI driver with helm, eg:

helm repo add openebs https://openebs.github.io/openebs
helm repo update
helm install openebs --namespace openebs openebs/openebs --create-namespace

Ironically, this helm chart installs a whole bunch of other storage services, including another replicated storage engine: Mayastor. I am not planning to use this. If you like, you can choose not to install these other services - the OpenEBS installation instructions tell you how to do this.

After this I needed a storage class, which was kubectl apply‘ed to the cluster:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-lvmpv
parameters:
  storage: "lvm"
  volgroup: "openebs-lvmpv" # must match the VG name you created
  fsType: ext4
provisioner: local.csi.openebs.io
allowVolumeExpansion: true   # optional but recommended
# don't create the LV until the container is allocated (so it is made in the right VM)
volumeBindingMode: WaitForFirstConsumer
# after releasing a PVC, keep the LV. To free up storage for reallocation it needs to 
# be manually deleted on the host with `lvremove`.
reclaimPolicy: Retain

OSD Draining and removal

Actually allocating a PV to this StorageClass means creating a VG called openebs-lvmpv on nodes that are allowed to use it. In my cluster all storage is fully allocated. Draining and removing a single OSD will give me enough space to migrate everything (eg a spare 4TB drive) - a larger cluster would mean more planning and more work.

This is “The Safe-ish Way”(TM). Using “The Dangerous Way” (delete a VM and format its OSD) should be a perfectly valid way of reclaiming the space also, but would put the cluster into a degraded mode straight away and fire off a ton of recovery network activity for hours.

Of course, I skipped a few steps to do with OSD node allocation, OSD pod deletion and operator re-enabling since this cluster is getting deleted. These are the rough steps I followed, and will result in a half-baked clean up and semi-broken cluster. Fine in my case since I’m deleting the whole thing once I’m finished, so take these notes as inspiration rather than instruction:

Before starting, make sure there is enough free space and that the cluster is healthy with the UI or toolbox pod, eg:

ceph df
ceph osd df
ceph osd pool ls detail   # check size and failure domain

# make sure cluster is healthy before proceeding
ceph status

Then do the operation itself:

# 1. Stop rook from trying to reclaim devices and nodes, set 
#       useAllDevices: false
#       useAllNodes: false

kubectl -n rook-ceph edit cephcluster rook-ceph

# 2. Stop the operator so it doesn't fight us
kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas=0

# 3. In the toolbox: mark it out (the OSD stays up and drains gracefully)
ceph osd out osd.<ID>

# 4. I had to wait about a day for all PGs to be active+clean
ceph osd safe-to-destroy osd.<ID>    # confirm it's safe to delete now

# 5. Stop the OSD pod
kubectl -n rook-ceph scale deploy rook-ceph-osd-<ID> --replicas=0

# 6. In the toolbox: remove it from CRUSH, auth, and the osdmap
ceph osd purge <ID>  # only works if safe to delete
ceph osd crush rm <hostname>

Kubernetes node shutdown

With the data drained from the OSD, I had no use for the Kubernetes node it was attached to any more. I deleted the node from Kubernetes with kubectl delete node ..., then powered off the VM and virsh undefine‘d it.

This took out a mon for me, and because the operator was scaled to zero, this did not get fixed. Since I had plenty of redundancy it wasn’t a problem but this is a good gotcha to look for in your own migrations.

The old OSD was a pass-through disk so was now sitting unused.

Getting the disk to the Kubernetes node

To actually get the freed up disk available at the point of use in the crypto node, I formatted it as regular ext4 on the hypervisor, mounted it to /data/nvme and then created a .qcow2 file under /data/nvme which was mapped as storage in the crypto Kubernetes VM. There are some pros and cons to using qcow2 files but on balance, I think it’s fine for what I want to do:

  • One physical device can only be shared with one VM
  • In the Rook/Ceph world, it makes sense to just pass the whole drive through to the VM
  • qcow2 add some operational overhead and additional management but I’m not planning to change too many things, and I already wrote Ansible scripts to create and map the files to libvirt XML
  • I can create additional .qcow2 files and share with different VMs for different Kubernetes nodes if I want to, eg for my apps cluster
  • Beware of overcommitting - qcow2 files are sparse by default, so nothing will stop you allocating a 10TB file on a 500GB disk

It’s possible to do some cool things with LVM storage pools and KVM/Libvirt to make allocation a slicker process. I decided against this in my lab since I’m already using .qcow2 files for the OS images and I didn’t want two different ways of managing disks in the lab.

LVM VG creation

After rebooting the VM (Ansible script requirement), I had a free disk to use for whatever I wanted so I just did what I normally do with a new disk:

  1. gparted /dev/somedisk - Create GPT partition table, allocate LVM to the whole drive
  2. pvcreate /dev/somedisk1 - the partition you created
  3. vgcreate openebs-lvmpv /dev/somedisk1 - must match above

No need to create any LVs - that’s what the localpv driver does for us.

Copy out of existing pods

Copying out the data from Rook/Ceph to LVM was straightforward, if a little slow:

Create a PVC

First create a PVC, nothing gets allocated yet:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lvm-meowcoin-pvc
  namespace: crypto
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: openebs-lvmpv
  resources:
    requests:
      storage: 10Gi

Assign to a pod, with a selector

Now we want to start a pod with both the Rook/Ceph and LVM storage mounted at the same time. A dedicated pod could be used for this, but since I need to alter the deployment anyway, I just did it all in one place.

Note that Recreate strategy is used to ensure pods are shut down cleanly because the storage PVC can only be mounted to one node at a time since it’s marked ReadWriteOnce. This doesn’t prevent two pods on the same node from doing so though! Eg during a rolling restart. There are a few ways to ensure only one pod at a time has a PVC mounted but this is what works for me and it was the answer to multiple blockchain corruption incidents over the years:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: crypto
  name: meowcoin
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: meowcoin
  template:
    metadata:
      name: meowcoin
      labels:
        app: meowcoin
    spec:
      hostname: meowcoin
      nodeSelector:
        # make sure we run only on this one server:
        kubernetes.io/hostname: kps-crypto-1 
      containers:
      - name: meowcoind
        image: quay.io/declarativesystems/cryptodaemons_meowcoin:30.2.7

        # Make a pod that just runs sleep so we can `exec` into it...
        command: [/bin/sleep]
        args: [infinity]

        # ...and disable the normal pod actions...
        # args:
        # - "-server"
        # - "-printtoconsole"
        volumeMounts:
        - mountPath: /root/.meowcoin
          name: meowcoin
        - mountPath: /new
          name: meowcoin-new
        resources:
          limits:
            memory: "2Gi"
            cpu: 1

      restartPolicy: Always
      volumes:
      - name: meowcoin
        persistentVolumeClaim:
          claimName: rook-ceph-meowcoin-pvc
      - name: meowcoin-new
        persistentVolumeClaim:
          claimName: lvm-meowcoin-pvc

Applying this takes down the meowcoin server and brings up a pod we can use to copy the data between filesystems. After checking the allocations are correct with df -h, the data can easily be copied, eg:

cp -a /root/.meowcoin/. /new/

After the command completes, I checked with df -h that about the same amount of storage was being used on the LVM mount as the Rook/Ceph one, which is good enough for me.

A better check would have been du -s on both directories or even running rsync might be a useful secondary verification. Not a big deal in this case since the crypto server would just sync any missing blockchain data.

Return to service

Once this finishes, we reconfigure the deployment for production use again:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: crypto
  name: meowcoin
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: meowcoin
  template:
    metadata:
      name: meowcoin
      labels:
        app: meowcoin
    spec:
      hostname: meowcoin
      nodeSelector:
        # make sure we run only on this one server:
        kubernetes.io/hostname: kps-crypto-1 
      containers:
      - name: meowcoind
        image: quay.io/declarativesystems/cryptodaemons_meowcoin:30.2.7

        # Keep for emergencies...
        # command: [/bin/sleep]
        # args: [infinity]

        # ...and restore the normal pod actions...
        args:
        - "-server"
        - "-printtoconsole"
        volumeMounts:
        - mountPath: /root/.meowcoin
          name: meowcoin
        resources:
          limits:
            memory: "2Gi"
            cpu: 1

      restartPolicy: Always
      volumes:
      - name: meowcoin
        persistentVolumeClaim:
          claimName: lvm-meowcoin-pvc

A few seconds later, the pod was up and running with all required data.

Clean up

Once the new pod is up and running the Rook/Ceph PVC can be deleted, which should free up its storage unless you changed the default reclaimPolicy.

After repeating the process above for all PVCs in the Rook/Ceph cluster and double-checking my work, I was left with a Rook/Ceph cluster that was still happily running OSDs and other pods. If I was keeping the cluster then additional clean up steps would be needed to restore normal operations. If you need to do this, I suggest you read the instructions carefully.

In my case, this is the end of this Rook/Ceph cluster. Since it runs entirely within KVM via libvirt, I didn’t bother to properly clean up anything, I just virsh destroyed and virsh undefineed until the entire Kubernetes cluster was deleted.

At this point, the pass-through OSD disks could be repartitioned with cfdisk.

The final step of clean up was to remove the Rook/Ceph deployment from the consumer cluster, by uninstalling the helm chart and deleting the namespace.

Tips and tricks

Use screen if available

In case your session gets interrupted, you can use tmux or screen inside the pod if it’s available, so your command doesn’t stop if your kubectl exec disconnects for some reason.

If it’s missing and you’re lucky, you might be able to apt install it.

local-storage

Don’t forget you can still use K3s built-in local-storage as an alternative path to copy out data from Rook/Ceph if you’re having trouble finding space elsewhere. Similar technique to above.

Grow an LV

If you need more space, you can just increase the value of the storage field and apply your yaml. A couple of minutes later, the additional space was available in the pod with no restart needed. Eg:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lvm-meowcoin-pvc
  namespace: crypto
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: openebs-lvmpv
  resources:
    requests:
      storage: 50Gi # <~~~ edit here

What did I lose?

The switch has some drawbacks, that I willingly signed up for:

  • Storage is now explicitly pinned to nodes, it does not “float” around the cluster anymore
  • Pods are manually pinned to Kubernetes nodes:
    • Pods are linked to a named PVC, so moving a workload means pinning the deployment to another Kubernetes node and creating a new PVC with a new name, so Kubernetes can create it in the right place
    • If a pod restarts with a fresh PVC it needs to download its entire blockchain from the peer-to-peer network again. This can take days unless I build a NAS and write a backup and restore procedure to get up and running quicker
    • The typical worst case would be a brief outage of minutes/hours while a mini PC is rebooted or has maintenance done on it
    • Additional manual steps to restore operation make the advantage of Kubernetes less compelling in this case but do not eliminate it altogether. It’s still the best way to manage containerized apps for my workload
  • No more replicated storage
  • No UI
  • No S3 compatible storage
  • No “storage as a service” for other Kubernetes clusters

What did I gain?

Crypto nodes don’t really benefit from replicated storage the way I had it set up:

  • The data is already replicated over the peer-to-peer network
  • The “floating” storage was fun to test out and speeds up crypto node restarts, but restarts still take several minutes, during which, the node cannot be used.
  • If some kind of HA crypto node was actually needed (eg for an exchange), then the way to do this is with multiple, independent nodes - with their own independent storage. I don’t have the need or the hardware for this
  • Mitigation for slow node rebuilding (hours/days) if a disk fails is to periodically snapshot the blockchain data to the NAS I’m planning on building, or just replace the drive and suck up the delay in the interim

I knew this walking into my distributed storage adventure of course - running a crypto server is a low-stress, low-effort way to get some constantly updating data to play with on your storage cluster. If you want to start over, you can do so guilt- and worry-free.

In terms of extra hardware, I gain:

  • Entire 16GB N100 mini PC
  • Valuable SSDs and NVMes (at least 2x 4TB drives)
  • CPU and RAM used by 5 big VMs on 5 PCs
  • Not storing 3 copies of all data means a 66% reduction in lab storage needed
  • I’m wondering if this will even bring down the power bill (or room temperature)

And in terms of admin:

  • 5 big VMs no longer need monitoring, etc
  • 2 VLANs can be retired
  • Network traffic rules can be simplified
  • Less traffic on the LAN in general - important on a 1GbE network
  • No more scary Rook/Ceph upgrades

Conclusion

I hope reading this you don’t think I’m picking faults with Rook/Ceph - I’m not, it’s excellent. I’ve learned a lot using it, but the more useful workloads I planned to use with it failed to materialize and for my crypto homelab it’s like using a nuclear weapon to swat a fly.

Using Rook/Ceph daily also confirmed to me that if I was going to use this for something important or a customer, a single 1GbE network cluster is not going to cut it. Production needs mean faster networking, multiple clusters and probably a support contract.

Onward!

Post comment

Markdown is allowed, HTML is not. All comments are moderated.