I looked at a few options, with the requirements:

• NVME storage (No eMMC - its slow and wears out)
• Ideally 16GB RAM to support intrusion detection and extra tools
• At least 3 ethernet ports (for CARP)
• Must be easily fixable/replaceable by me, in Australia
• 1GgE ethernet is all my switches support, and I’m not planning on upgrading due to cost

OPNSense Appliance

OPNSense ship their own appliances, designed and manufactured in Europe. These are slick units with a choice of either desktop or rack mount form factors.

As much as I wanted to buy one to support the project, the basic model I selected was still €678 ($1124AUD) + shipping.

There’s a few video reviews on other models on youtube. The hardware seems solid but getting parts under warranty in Australia is not going to be a 24 hour turnaround and its just SO expensive.

OPNSense Appliance x2, for HA with CARP failover

OPNsense lets you do HA for routers with HA-CARP. This is an OPNSense specific HA protocol that lets a backup router take over if the main one fails. This would also let you do scheduled maintenance without an internet outage.

The drawback of this approach is that you now need 2x routers and 2x uplinks to your ISP. This type of HA protects you only against outages caused by router fault/reboot.

Since I can only get one IP address at a time from my ISP, I abandoned this approach.

Protectli Mini PC

The Protectli mini PCs are quite nice, I almost bought one but they are quite expensive for what you get and there are some identical looking boxes on AliExpress as well. Since its a mini-pc, your also stuck with whatever you bought in terms of network ports.

Cost with options came to about $500USD (700AUD) + shipping, high enough that we’re into full PC territory, but we are still getting very basic PC hardware for the price.

Build a small PC!

In the end, I decided to just build myself a mini ITX PC, recycling some parts I already had and going deluxe with the case and cooler. Is this overkill? Absolutely:

This setup means I have a quiet, powerful router platform that will last for years to come and handle anything I can throw at it with my current 1GbE network. I can very easily upgrade the networking hardware if I want to, along with anything else. If I get any hardware failures, I can source replacements locally and get them within 24 hours, and I have a seriously cool looking router I built myself.

Deployment architecture

There is enough hardware here to do things like run OPNSense in a VM, and also host other network services. This could let me do things like implement the HA/CARP setup with VMs. I’m pretty confident I could make this work with libvirt but at this point I took a pause and reached out to my colleagues at Confluent and setup a quick slack poll on our #homelab channel.

The results was unanimous: Run OPNSense on bare metal.

While this means no VM based HA or ability to take VM snapshot backups, the gain is a much simpler overall system that is basically just a simple FreeBSD system. Big problems? Plugin a screen and keyboard. The consensus was that an appliance-style deployment is just so much simpler to troubleshoot, and the nice thing about having this DIY setup is that if I ever did want to go the VM route in the future, its very easy to do so with no additional hardware purchase needed.

OPNSense Install

Installing OPNSense was extremely easy. Just boot the VGA installer from USB and install.

This time, I made sure to fix all the static leases I cared about in the old router before doing a final backup and then restored this onto the new router.

There were only a couple of gotchas in this process:

1. Had to enable CSM boot/disable secure boot
2. Interface devices had changed vs the backup so needed adjustment

After these changes, I was up and running in minutes.

Verdict

It’s been about a month now and this router PC has been rock solid. It just sits there looking pretty and is almost completely silent thanks to the Noctua cooler. I have not thought more about converting to VMs since everything just works.

If your thinking of building your own PC for OPNSense, I highly recommend it if your comfortable building your own Computers.

To cut a long story short, this happened because the DHCP leases database was reset and they all got new IP addresses, which meant individual nodes (K3S) could not startup.

There were 3 steps needed to recover the clusters, and I can prevent this from happening again very easily:

Solution

Step 1 - Switch back to old IP address

I’m pretty sure my etcd was completely broken. The easiest way I found to get the old IP address was to just search the logs:

sudo journalctl | grep -i dhcp

Which worked on all of my Debian kubernetes nodes.

Note down the IP address, MAC address and hostname, then in your router, create a static lease with the old details. Don’t click apply until you are ready for the next step.

Step 2 - Reboot all nodes

eg, with ansible:

ansible all -i inventories/hosts.yml -b -m ansible.builtin.reboot

Now click apply on your router while the systems are rebooting.

Step 3 - Test access and update if needed

On a cluster node, see if access is now working:

sudo kubectl get nodes

If this worked, you will now see your cluster nodes as usual instead of the some error.

Outside the cluster, you may find your previous credentials now give certificate errors. After checking requests are reaching the right cluster endpoint IP address, there is a good chance the cluster has re-issued itself new certificates. In this case, replace your credentials for .kube/config with a fresh copy from the node and this should fix things.

Agent nodes may have similar issues and find themselves permanently evicted. This happened to me and it was easiest to just delete the agent node from the control plane and reinstall the node as a fresh worker.

Fixed

After performing these steps, my cluster was operational again but this was NOT a fun exercise.

Thankfully this was a pretty empty lab setup. In the real world, 100% there would have been big disruption, crashed pods and customer impact.

I note that my rook/ceph managed to recover itself on its own once IP addresses were fixed, which is outstanding.

Prevention

Basically Kubernetes likes to have static IP addresses, but I like the convenience and flexibility of DHCP, especially when I might want to reconfigure hosts from the router.

Most of my hosts are headless and with only one NIC, so any screw up in static address reconfiguration with say ansible means crawling around with HDMI cables and keyboards to fix.

There is however an easy fix I overlooked in OPNSense: On the far right, there is a + button that adds a static lease.

So my new solution is when I’m happy a lab system is up and running nicely, just click the button and add a static lease if Kubernetes is involved.

That way it gets included in backups and if there router need re-flashing, systems should come back online with the right address soon after the router is back online.

If you want to know more about deploying and using ClickStack, The Manual is excellent.

Based on this doc, I was able to get up and running with the docker.io/clickhouse/clickstack-all-in-one container very quickly in docker on my laptop.

The all-in-one image bundles everything you need for a functioning log collector and frontend:

• ClickHouse database
• HyperDX UI
• OTEL collectors
• Glue scripts, etc example Dockerfile

ClickStack into “production”

Disclaimer: this is for a homelab!

I found a permanent home for ClickStack on my “services” mini PC.

Podman Quadlet

I was able to run ClickStack as a first class homelab service quite easily with Podman Quadlet like I do for some of my other lab services, like this:

/etc/containers//systemd/clickstack-pod.kube

[Install]
WantedBy=default.target

[Unit]

[Kube]
Yaml=/etc/containers/systemd/clickstack-pod.yml

# ui/hyperdx
PublishPort=127.0.0.1:8080:8080

# clickhouse/db
PublishPort=127.0.0.1:8123:8123

# otel/grpc
PublishPort=127.0.0.1:4317:4317

# otel/http
PublishPort=127.0.0.1:4318:4318

/etc/containers//systemd/clickstack-pod.yml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    io.kubernetes.cri-o.ContainerType/app: container
    io.kubernetes.cri-o.TTY/app: "false"
    io.podman.annotations.autoremove/app: "FALSE"
    io.podman.annotations.init/app: "FALSE"
    io.podman.annotations.privileged/app: "FALSE"
    io.podman.annotations.publish-all/app: "FALSE"
  labels:
    app: clickstack
  name: clickstack
spec:
  automountServiceAccountToken: false
  dnsPolicy: None
  containers:
  - image: docker.io/clickhouse/clickstack-all-in-one:2.15.1
    name: app
    env:
    - name: FRONTEND_URL
      value: "https://clickstack.infrastructure.asio"
    ports:
    - containerPort: 8123
      hostPort: 8123
    - containerPort: 8080
      hostPort: 8080      
    - containerPort: 4317
      hostPort: 4317
    - containerPort: 4318
      hostPort: 4318 
    resources: {}
    securityContext:
      capabilities:
        drop:
        - NET_ADMIN
        - CAP_MKNOD
        - CAP_AUDIT_WRITE
    volumeMounts:
    - mountPath: /data/db
      name: clickstack-data-vol
      subPath: db
    - mountPath: /var/lib/clickhouse
      name: clickstack-data-vol
      subPath: clickhouse
    - mountPath: /var/log/clickhouse-server
      name: clickstack-log-vol

  enableServiceLinks: false
  hostname: clickstack
  restartPolicy: Never
  volumes:
  - hostPath:
      path: /data/containers/clickstack/data
      type: Directory
    name: clickstack-data-vol
  - hostPath:
      path: /data/containers/clickstack/log
      type: Directory
    name: clickstack-log-vol
status: {}

Traefik frontend

With traefik already setup on the host, I only needed a single drop-in:

/etc/traefik/traefik.d/clickstack.infrastructure.asio.yml

# ansible managed

#
# HTTP
#
http:
  routers:
    clickstack-http:
      # HostSNI only for TCP - we get the real headers for http routes
      # as TLS already terminated and http call parsed
      rule: "Host(`clickstack.infrastructure.asio`)"

      tls: {}

      entryPoints:
      - infrastructure.https
      service: clickstack-8080

  services:
    clickstack-8080:
      loadBalancer:
        servers:
          - url: "http://localhost:8080"

#
# TCP
#
tcp:
  routers:
    otel-grpc:
      # always match - if not tls, there is no such thing as a header
      # so there is nothing to check against so just accept any value
      rule: "HostSNI(`*`)"
      entryPoints:
      - infrastructure.otel-grpc
      service: clickstack-4317
    otel-http:
      # always match - if not tls, there is no such thing as a header
      # so there is nothing to check against so just accept any value
      rule: "HostSNI(`*`)"
      entryPoints:
      - infrastructure.otel-http
      service: clickstack-4318


  services:
    clickstack-4317:
      loadBalancer:
        servers:
          - address: "localhost:4317"
    clickstack-4318:
      loadBalancer:
        servers:
          - address: "localhost:4318"

Setting up ClickStack

With these items setup, hitting the URL I configured in the browser got me straight to a set password dialog in HyperDX. Done.

Client setup

Now its time to send logs to ClickStack. I have a lone Windows machine on the network which is a source of constant pain, so this is the perfect starting point.

To collect logs from Windows:

1. Install OTEL client for windows and install it as a windows service
2. Install the OTEL contrib files for windows which are needed to collect the logs from Event Viewer
3. Finally, configure the OTEL client in C:\Program Files\OpenTelemetry Collector\config.yaml. You will need a token for the ClickStack OTEL collector which you can find in the HyperDX UI by looking for Your Ingestion API Key:

# To limit exposure to denial of service attacks, change the host in endpoints below from 0.0.0.0 to a specific network interface.
# See https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks


receivers:
  windowseventlog/application:
     channel: application
  windowseventlog/system:
     channel: system


processors:
  batch:
  resource:
    attributes:
      - key: host.os
        value: windows
        action: insert


exporters:
  otlp/logs:
    endpoint: clickstack.infrastructure.asio:4317
    tls:
      insecure: true
    headers:
      authorization: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    compression: gzip

service:
  pipelines:
    logs:
      receivers:
        - windowseventlog/application
        - windowseventlog/system
      processors:
        - batch
      exporters:
        - otlp/logs

If everything worked, windows will start sending event logs to the OTEL collector previously exposed, and you will see events landing live in the UI:

Verdict

This is a very slick and easy to use stack. The hardest part of setting this up turned out to be the OTEL client on Windows. I think its safe to say if it works on Windows, I’ll be able to collect the rest of the logs I need no problem, although I’m sure kubernetes logs will be a pain.

For an enterprise grade logging solution, I’d be looking for a more scalable deployment architecture probably running on Kubernetes with a support contract but for my little homelab this is perfect.

Pretty sure both of those asks are available…

This time I was able to fix it without a network redesign since the chaos was isolated to a single box.

Here’s my notes on diagnosis and solution.

Diagnosis

Inexplicable MQTT errors:

• Home Assistant/zigbee2mqtt: z2m: MQTT error: Keepalive timeout
• Mosquitto server: Client XXXX has exceeded timeout, disconnecting
• mosquitto_sub - no errors
• Many Default deny / state violation rule packets on router

Proof

This was easy. I just adjusted the tcp command in debugging asymmetric routing to monitor MQTT traffic. It was very obvious that replies were being sent out on the wrong interface and this was breaking routing

We can see why the OS does this by inspecting the routing table:

root@atlas:/home/geoff# ip route show
default via 10.100.0.1 dev br0 proto dhcp src 10.100.254.246 metric 1004 
default via 172.16.0.1 dev br1 proto dhcp src 172.16.0.244 metric 1006 
default via 10.110.0.1 dev br110 proto dhcp src 10.110.94.63 metric 1010 
10.89.0.0/24 dev podman1 proto kernel scope link src 10.89.0.1 
10.100.0.0/16 dev br0 proto dhcp scope link src 10.100.254.246 metric 1004 
10.110.0.0/16 dev br110 proto dhcp scope link src 10.110.94.63 metric 1010 
172.16.0.0/16 dev br1 proto dhcp scope link src 172.16.0.244 metric 1006 

And these all come from dhcpcd running on several interfaces, as defined in /etc/network/interfaces (debian trixie):

...
# os bridge
auto br0
iface br0
    bridge-ports enp1s0
    use dhcp

# tagged vlan bridges
auto br1
iface br1
    bridge-ports enp1s0.1
    use dhcp

auto br110
iface br110
    bridge-ports enp1s0.110
    use dhcp

Solution

Turns out dhcpcd needed will always add routes for new interfaces so we need to clean them up ourselves to avoid mayhem with some little bash script hooks, like this:

/etc/dhcpcd.exit-hook

#!/bin/sh
# logs: journalctl -u networking

# if DHCP gives us a new address, we could check with $old_ip_addres == $new_ip_address
# but this happens basically 0% of the time so just reboot until run out of IP addresses
# for now/forever if changed address breaks things (traefik)

for script in /etc/dhcpcd.exit-hook.d/*.sh ; do
    echo "dhcpcd.exit-hook run script: ${script}" 
    "$script"

/etc/dhcpcd.exit-hook.d/br1_isolate.sh

#!/bin/bash

# only when we actually have an address
if [ "$interface" = "br1" ] && [ -n "$new_ip_address" ]; then
    echo "delete routes to avoid asymmetric routing - br1"
    ip route del default dev br1
    ip route del 172.16.0.0/16 dev br1
fi

/etc/dhcpcd.exit-hook.d/br110_isolate.sh

#!/bin/bash

# only when we actually have an address
if [ "$interface" = "br110" ] && [ -n "$new_ip_address" ]; then
    echo "delete routes to avoid asymmetric routing - br110"
    ip route del default dev br110
    ip route del 10.110.0.0/16 dev br110
fi

After restarting networking, routing table looks perfect:

default via 10.100.0.1 dev br0 proto dhcp src 10.100.254.246 metric 1025 
10.89.0.0/24 dev podman1 proto kernel scope link src 10.89.0.1 
10.100.0.0/16 dev br0 proto dhcp scope link src 10.100.254.246 metric 1025

And MQTT works in Home Assistant again!

Root of the problem

This whole problem was caused by enabling a bunch of interfaces that grant access to different VLANS, but as mentioned in previous debugging post, VLANS are for traffic separation, and if you have something essentially bridging the VLANS as above, it defeats the whole point of the exercise.

So what was I trying to do here? I was trying to expose a bunch of services like nexus and MQTT to the whole network and run prometheus in the management VLAN.

The problem is the main host is not officially on the management VLAN, it has an interface setup with an IP address and thats it. After deleting the routes so it can’t reach anything, this effectively blocks me from using docker/podman bridge networking to run prometheus boudn to the management VLAN without a lot of extra/brittle work since podman expects the host to participate in networking.

MACVLAN and IPVLAN cause problems with dhcp which I already looked at extensively which leaves… VMs.

In the end, the solution is simple and bulletproof, if a little clunky: Just bind the interface into a VM directly and podman containers can just use the default network. Ironically, this also means the interface no longer needs an IP address on the host and you guessed it, removing the use dhcp from /etc/network/interfaces would have also prevented the asymmetric routing problem.

What a weekend.

Instead of tried and true haproxy, I tried out traefik.

Why Traefik?

• Dynamic config sections
• Up-skill in traefik
• Much more flexible routing
• Choose HTTP support or raw TCP, mix and match as you please
• Perfect choice for flexible ingress in homelab or enterprise environments

How?

Due to needing to operate across different networks, I chose to run the traefik binary directly on the host. Since the entrypoints section must be static, I ended up writing a .j2 template for traefik.yml:

traefik.yml.j2

# ansible managed

#accessLog: 
#  format: json

entryPoints:

{% if br110_ip is defined %}
  infrastructure.http:
    address: "{{ br110_ip }}:80"
  infrastructure.https:
    address: "{{ br110_ip }}:443"
    http:
      tls: {}    
  infrastructure.docker:
    address: "{{ br110_ip }}:5000"
  infrastructure.dockers:
    address: "{{ br110_ip }}:5001"
  infrastructure.mqtts:
    address: "{{ br110_ip }}:8883/tcp"
  infrastructure.otel-grpc:
    address: "{{ br110_ip }}:4317"
  infrastructure.otel-http:
    address: "{{ br110_ip }}:4318"    
{% endif %}

{% if default_ip is defined %}
  default.http:
    address: "{{ default_ip }}:80"
  default.https:
    address: "{{ default_ip }}:443"
    http:
      tls: {}
  default.alertmanager:
    address: "{{ default_ip }}:9093"
  default.blackbox:
    address: "{{ default_ip }}:9115"
{% endif %}

providers:
  file:
    directory: /etc/traefik/traefik.d
    watch: true


log:
  level: INFO

The template configures itself to listen on specific IP addresses to avoid port clashes and expose services selectively.

The template gets pre-processed by a simple script and jinja2:

**/usr/local/bin/configure_traefik.sh

#!/bin/bash

# generate traefik.yml by merging IP addresses with traefik template

for iface in $(ip -o -4 addr show | awk '{print $2}' | grep '^br'); do
    ip=$(ip -o -4 addr show "$iface" | awk '{print $4}' | cut -d/ -f1)
    export "${iface}_ip=$ip"
done

# wait up to 60 seconds for an IP
for i in {1..60}; do
    export default_ip=$(hostname -i)
    if [ -n "$default_ip" ]; then
        break
    fi
    sleep 1
done

if [ -z "$default_ip" ] ; then
    echo "no default IP after waiting!" 
    exit 1
fi

# run j2 with all in-scope environment variables available
echo "rebuild traefik config file"
j2 -e "" -o /etc/traefik/traefik.yml /etc/traefik/traefik.yml.j2

Each time the service is restarted, the script is called due to ExecStartPre and this generates the final traefik.yml file:

traefik.service

[Unit]
Description=Traefik Reverse Proxy

# online target does not work with dhcpcd
After=network.target

[Service]
# Grant the specific capability we need (ambient set preserves it across execs if any)
AmbientCapabilities=CAP_NET_BIND_SERVICE

# Optional but strongly recommended: restrict to ONLY this capability
# (prevents the process from gaining others accidentally)
# allow bind port < 1024 without root
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

# Good security hygiene: prevent privilege escalation tricks
NoNewPrivileges=yes

User=traefik
Group=traefik

Type=simple
# runs as traefik user
ExecStartPre=/usr/local/bin/configure_traefik.sh
ExecStart=/opt/traefik/traefik --configFile=/etc/traefik/traefik.yml
Restart=always

[Install]
WantedBy=multi-user.target

TLS config

TLS can only be configured with a dynamic file placed in scanned directory as configured above (/etc/traefik/traefik.d):

/etc/traefik/traefik.d/tls.yml

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /etc/traefik/tls/wildcard.infrastructure.asio.cert.fullchain.pem
        keyFile: /etc/traefik/tls/wildcard.infrastructure.asio.key.pem

Per-service drop-ins

The final part of the puzzle was to create drop-ins for each required service in the /etc/traefik/traefik.d/ directory. Here’s my most complicated example - Nexus:

/etc/traefik/traefik.d/nexus.infrastructure.asio.yml

# ansible managed

#
# HTTP
#
http:
  routers:
    nexus-http:
      # HostSNI only for TCP - we get the real headers for http routes
      # as TLS already terminated and http call parsed
      rule: "Host(`nexus.infrastructure.asio`)"


      entryPoints:
      - infrastructure.http
      service: nexus-8081
    docker:
      # HostSNI only for TCP - we get the real headers for http routes
      # as TLS already terminated and http call parsed
      rule: "Host(`nexus.infrastructure.asio`)"


      entryPoints:
      - infrastructure.docker
      service: nexus-5000

  services:
    nexus-8081:
      loadBalancer:
        servers:
          - url: "http://localhost:8081"
    nexus-5000:
      loadBalancer:
        servers:
          - url: "http://localhost:5000"

#
# TCP
#
tcp:
  routers:
    nexus-https:
      rule: "HostSNI(`nexus.infrastructure.asio`)"
      tls:
        passthrough: true
      entryPoints:
      - infrastructure.https
      service: nexus-8443
    dockers:
      rule: "HostSNI(`nexus.infrastructure.asio`)"
      tls:
        passthrough: true
      entryPoints:
      - infrastructure.dockers
      service: nexus-5001


  services:
    nexus-8443:
      loadBalancer:
        servers:
          - address: "localhost:8443"
    nexus-5001:
      loadBalancer:
        servers:
          - address: "localhost:5001"

Of course, I had some ansible create all of these codes for me from some basic per-host structured variables :)

Verdict

After automating the above for my podman services, I finally have:

• Services available in the right VLAN
• No more MACVLAN DHCP chaos
• Correct host/VLAN isolation (see gotchas - below)
• TLS everywhere with wildcard certificate
• Easy, repeatable method for adding new podman services

Why not Kubernetes?

These are my vital network services that Kubernetes needs itself - my other clusters use this one extensively, so these need to be amongst the most bulletproof services in my home lab and fool-proof containers/VMs are perfect for this.

Gotchas

There were some unique challenges I had with traefik which are not well documented:

• If you have any entrypoints specified, you cannot override them with arguments or environment variables
• You cannot configure your TLS certificates in the static section. They are only processed in the dynamic config files
• Static config files are fully static - no variables, etc
• Cannot use this pattern to bridge host-isolated VLANS, use a VM instead
• Podman Quadlet services must bind to localhost only, to avoid port collision, like this:

/etc/containers/systemd/nexus-pod.kube

[Install]
WantedBy=default.target

[Unit]

[Kube]
Yaml=/etc/containers/systemd/nexus-pod.yml

# web UI
PublishPort=127.0.0.1:8081:8081

# h2 database (nothing normally listens, not exposed outside host)
PublishPort=127.0.0.1:1234:1234

# web UI + TLS
PublishPort=127.0.0.1:8443:8443

# docker
PublishPort=127.0.0.1:5000:5000

# docker + TLS
PublishPort=127.0.0.1:5001:5001

Don’t do this. bind ports to host and port-forward.

Extended version

This was a dead-end but documenting here for clarity in-case it helps someone.

I have a multi-homed/multi-VLAN host running podman kube services for the network. So far I’ve been essentially plugging connectors directly into specific VLANs on the network.

Networking has always been a bit flakey for me with these containers. Today, I decided to fix things.

After much research, I ended up finding a pattern that worked better:

• Separate podman network with bridge networking and external DHCP server
• In-container dhcp client

The in-container DHCP client turns the container into a full network host on a different VLAN to the host as far as the router is concerned, and this has nice side effects like automatic DNS hostnames if you have configured this.

Separate podman network

I manage my podman networks with ansible. I was able to define a new network bound to my separately setup linux network bridge br110 like this:

- name: podman bridge vlan br110/infrastructure
  containers.podman.podman_network:
    name: podman-vlan-infrastructure
    driver: bridge
    opt:
      bridge_name: br110
    ipam_driver: none
    force: true

Bridge networking is way more reliable then MACVLAN which I was previously using.

The network is referenced it in a .kube file like this:

PodmanArgs=--mac-address de:3b:ee:01:02:03
PublishPort=1883:1883
PublishPort=8883:8883
PublishPort=9001:9001
Network=podman-vlan-infrastructure

The fixed MAC address is so I can issue a static DHCP lease on the router.

In-container DHCP client

There’s a few moving parts to this:

Container image must provide a DHCP client

Alpine images provide udhcpc, if nothing already in image, you will need to build a custom image

Writable /etc/resolve.conf

/etc/resolv.conf inside containers is often a mount point provided by the Podman network stack itself. udhcpc will fail to rename it with Resource busy errors. We can fix this by providing our own mount instead.

Add a volume to the pod:

spec:
  volumes:
  - name: resolv-override
    emptyDir: {}

Mount in the container(s) at the right place:

    volumeMounts:
    - name: resolv-override
      mountPath: /etc/resolv.conf

Security context

Boost privileges as needed in the container(s) to allow DHCP broadcasts and file updates:

    securityContext:
      capabilities:
        # if container runs as non-root, boost UID to root to let it
        # write /etc/resolve.conf
        # runAsUser: 0
        # runAsGroup: 0    
        add: 
        - NET_ADMIN
        - NET_RAW       

Container startup

Container(s) need their command and args tweaked to run DHCP client first and then delegate to the normal docker entrypoint with exec. This needs to be customized for each image you want to deal with, eg this one is for mosquitto:

    command: 
    - /bin/sh
    args:
    - -c
    - |
      udhcpc
      exec sh /docker-entrypoint.sh /usr/sbin/mosquitto -c /mosquitto/config/mosquitto.conf

Complete example

My completed mosquitto definition looks like this:

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.3.1
apiVersion: v1
kind: Pod
metadata:
  annotations:
    io.kubernetes.cri-o.ContainerType/app: container
    io.kubernetes.cri-o.SandboxID/app: e026b5b478c4489f78b85e875af39f4d439467a7b7abdf469629a9daa312cdf
    io.kubernetes.cri-o.TTY/app: "false"
    io.podman.annotations.autoremove/app: "FALSE"
    io.podman.annotations.init/app: "FALSE"
    io.podman.annotations.privileged/app: "FALSE"
    io.podman.annotations.publish-all/app: "FALSE"
  creationTimestamp: "2025-01-27T10:57:38Z"
  labels:
    app: nexus
  name: nexus
spec:
  automountServiceAccountToken: false
  containers:
  - image: docker.io/sonatype/nexus3:3.89.1-alpine
    name: app
    command: 
    - /bin/sh
    args:
    - -c
    - |
      udhcpc
      exec sh /opt/sonatype/nexus/bin/nexus run
    ports:
    - containerPort: 1234
      hostPort: 1234
    - containerPort: 8081
      hostPort: 8081
    - containerPort: 8443
      hostPort: 8443 
    resources: {}
    securityContext:
      capabilities:
        runAsUser: 0
        runAsGroup: 0    
        add: 
        - NET_ADMIN
        - NET_RAW       
        drop:
        - CAP_MKNOD
        - CAP_AUDIT_WRITE
    volumeMounts:
    - name: resolv-override
      mountPath: /etc/resolv.conf
    - mountPath: /nexus-data
      name: nexus-data-vol
    - mountPath: /nexus-data/etc
      name: nexus-config-vol  
    - mountPath: /nexus-data/etc/tls
      name: nexus-tls-vol            
  enableServiceLinks: false
  hostname: nexus
  restartPolicy: Never
  volumes:
  - hostPath:
      path: /data/containers/nexus/data
      type: Directory
    name: nexus-data-vol    
  - hostPath:
      path: /data/containers/nexus/config
      type: Directory
    name: nexus-config-vol   
  - hostPath:
      path: /data/containers/nexus/tls
      type: Directory
    name: nexus-tls-vol
  - name: resolv-override
    emptyDir: {}
status: {}

Reboot and test

The container should appear on the network and DHCP should now be rock solid

Why not do this?

While this works, running DHCP clients in containers reduces security and increases complexity. Specifically, entrypoint needs to be configured differently for each container.

In the end it turns out to be much simpler to just forward ports from the host and register a DNS alias on the router.

In the past, I’ve held off doing this because having multiple VLANs with active IP addresses on my host was creating asymmetric routing chaos.

I’m confident I can solve these problems now and install a simple reverse proxy instead which should be way simpler to manage - stay tuned.

ESP32 with a vfat SD card. Everything working fine. Take SD card out, copy .json config file over top of existing file on PC, SD card back in ESP32, power on and WIFI no longer connecting/soft lockup.

Running fskc on SD card gives no errors.

Luckily my program is configured to print out the config file on startup for debugging purposes. On a serial console, I see most of the config file being printed out, however, the end of the file is replaced by a continuous stream of smiley face (☺/0x01) characters:

"lomq32/devices/makerfabs/malo4mos/121": { "name": "tv pumps", "descript☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺☺

After some digging, it turns out on ESP32, the FAT implementation can sometimes read the old/wrong clusters when you copy files over the top of existing ones - and the strange characters I was seeing were basically an overflow/underflow when reading from the card.

The fix?

On PC:

1. Delete config file from SD card
2. Copy the (identical) file from the PC to the SD card again
3. Unmount SD card, insert into ESP32, power on

The System booted normally and joined WIFI after this.

Strange huh?

Fixing this for the whole network on an OPNsense router while preserving VLAN security is surprisingly simple. On the OPNsense web UI:

Step 0: RTFM

• Multicast DNS Proxy

Step 1: Install

System -> Firmware -> Plugins

Select and install os-mdns-repeater, then reboot

Step 2: Configure

Services -> mDNS Repeater

• Enable
• Listen Interfaces is the networks you want to bridge. Apparently there is a limit of 5 although the UI does not prevent you selecting more. By bridging, we are effectively making a flat networking space for mDNS so that Home Assistant will find devices on other VLANs

Step 3: Firewall

The last thing to do is add firewall rules to allow mDNS traffic where needed. I put the very slack rule:

Which just allows all mDNS traffic to anywhere. You could restrict this further if needed.

Of course, Home Assistant also needs to be able to reach back to the devices it discovers, so this may also require additional rules depending how your network is setup.

Step 4: Save/apply/reboot

After making changes like this, its good to reboot the router so you can be sure settings survive a reboot.

Some devices only send mDNS packets on startup as well, so this is a good time to go around the house rebooting printers etc.

Step 5: Test/enjoy

If everything worked, that’s really all there is to it. A few minutes after rebooting devices, I saw things showing up in Home Assistant -> Settings -> Devices & Services -> Discovered:

On my phone, my TV was detected in Prime Video and I was able to watch content, whereas I normally have to join a different WIFI SSID.

Finally, on my Linux desktop, avahi-browse also finds devices:

$ avahi-browse -a
+ wlp3s0 IPv4 Brother HL-L2460DW                            Web Site             local
+ wlp3s0 IPv4 Brother HL-L2460DW                            Secure Internet Printer local
+ wlp3s0 IPv4 Brother HL-L2460DW                            Internet Printer     local
+ wlp3s0 IPv4 Brother HL-L2460DW                            UNIX Printer         local
+ wlp3s0 IPv4 Home                                          _home-assistant._tcp local
...

In summary this makes using Home Assistant way simpler and restored casting to TV so that a normal human being can use it.

Eventually, any serious IT conversation ends up with a discussion about robot vacuum cleaners

I bought my Roomba 980 10 years ago and it still works fine, although some of the plastic parts have started breaking and its loud as hell. Sadly, Roomba recently went bankrupt - who knows what effect this will have on the Roomba app and privacy, or if the app will even continue to work at all.

With this in mind I searched for a modern replacement with good privacy controls and drew a blank. Most manufacturers have mandated cloud control for their new devices. There’s just no way I’m spending almost $2000 on a new robot that doesn’t respect privacy. I did find two exceptions:

1. Valetudo - An outstanding project from the author and clearly a labour of love. As much as I wanted to give this a go, I’ve just got too much going on to hunt down a compatible model, crack open the case and then hookup a serial programmer while also being careful to prevent any future firmware updates. If I already had a compatible robot I’d totally install this but buying into this ecosystem with activly hostile manufacturers is not something I’m willing to pay for.
2. Matic - Promising. It claims to work locally instead of requiring cloud, but not open source, too big to clean under the bed, expensive and requires vacuum bags only available from Matic.

Fixing up the Roomba

I decided to just fixup and secure the Roomba, and integrate it properly with Home Assistant.

Firewall ports

Roomba documentation reproduced here, for your convenience:

Internal Network Traffic

• UDP port 5353/5678 for discovery.
• TCP/HTTPS 443 for data traffic.
• TCP/MQTT 8080/8883 for data traffic.

Outbound Traffic to the Internet

• UDP/SNTP port 123 for time.
• TCP/HTTPS 443 (/80) for data traffic.
• TCP/MQTT 8080/8883 for data traffic.
• UDP/TCP port 53 for DNS.

Repairs

I made a Roomba 980 collection on Printables.com and 3D printed replacement parts I needed, then ordered a new front wheel for $5 on Ali Express and a bag of spare rollers and filters for $20 on Amazon.

That covers all physical needs and the robot is back to tip-top condition.

Network Security Setup

With the decision made that this robot is function complete and I’m not interested in maps, I decided to to cut the robot off from the Internet completely and just allow it access to NTP and DNS.

I have a complex setup spanning managed devices and routers but essentially what’s needed to do this is:

Omada (wifi and switches)

• hell VLAN to completely isolate traffic
• WIFI access point called hell, mapped to corresponding VLAN
• Modify trunk ports to carry hell VLAN, and allow Home Assistant to reach hell on its LAN port

OPNsense (router)

• Interface for hell
• Device mapping for hell
• DHCP server for hell
• Firewall rules for hell

The disabled rule at the top is to temporarily allow internet access when onboarding the robot with the app.

Robot/App

Temporarily allow internet access in firewall, then either factory reset or change access point to hell in the app.

When the app says things are working, force close the app so that Home Assistant will be able to connect.

Back in your firewall, assign a static IP address to the robot.

Home Assistant

Home assistant provides a Roomba integration that works very nicely.

Even with mDNS forwarding now working, the Roomba is not detectable on a separate VLAN as it the integration uses the other discovery port, so just get the robot IP address from the router.

To connect, just allow the auto-detection to fail, then type the IP ADDRESS (not the hostname! - does not work) of the Roomba and follow the prompts on screen to get access automatically by pressing some buttons. There’s no need to run funny commands or docker images to get MQTT passwords any more.

If connecting fails, make sure the app is closed, reboot the Roomba and some reports even suggest to start vacuuming as well.

After successfully adding the Roomba, there will be a new panel on the dashboard where you can control the robot and you can also use voice control on the Home Assistant android app, like this (my robot is called Lucy):

Noise

Not much I can do about noise except avoid vacuuming at night and either go out or use noise cancelling headphones.

Disable internet access

With all systems working, I disabled the rule that allowed internet access, then connected to the hell SSID and tested it to make sure access was really blocked.

The app will no longer work but the Roomba is now fully secured from future updates and app breakages.

Conclusion

Really the only thing missing from this setup is the cleaning maps. Im not sure if my firmware is supported by rest980 and at this point, I’m happy to let this one feature slide.

I’m confident I can get a few more years out of my Roomba, and perhaps there will be some more consumer friendly vacuums to buy by then. Otherwise, perhaps maintaining a vintage robot may become the modern equivalent of driving a classic car 😂

In my case, I want to 3D print a WIFI QR code, mostly from the commandline on linux.

Step 1 - Generate QR code for WIFI

If you want a scan-to-join WIFI code, there is a special payload to use for your QR code, replce THESSID and THEPASSWORD for your own network:

WIFI:T:WPA2;S:THESSID;P:THEPASSWORD;;

Now generate your QR code as an SVG:

# apt install qrencode
qrencode -t SVG -o guest_wifi.svg "WIFI:T:WPA2;S:THESSID;P:THEPASSWORD;;"

Step 2 - Make it look pretty

Open inkscape, import the QR code SVG, add boxes and text to your hearts content, then export as a new SVG file.

Step 3 - Convert SVG to STL

I found a handy tool svg2stl to convert the pretty SVG from inkscape to an STL file, like this:

./svg2stl-linux --thickness 1.0 --pixel_size 0.025 ../guest_wifi_badge.svg

It worked perfectly for me, first time!

Step 4 - Slice it

The STL file generated above is just the black parts of the SVG. It needs to go on a base to print nicely. There’s probably an easier way to do this but the steps I needed in the Prusaslicer were:

1. Import the STL
2. Right click, add part, box
3. Scale and translate box to cover the QR code badge
4. Position the QR code above the box
5. Add a layer change and assign black colour
6. Make sure to print it big so the lines are clear - mine was about 12cm x 15cm
7. Check the slicing result with the layer slider to check everything looks good

Don’t forget to test-scan your code from the slicer app with your phone before printing.

Update: after printing, removing the QR code from the base caused all the bits of QR code to turn into projectiles and fly across the room, making a horrendous mess - make sure to not mix plastics and/or making the base thicker to reduce flexing 😂