traefik frontend for podman
Following on from Podman in-container DHCP networking, I needed a frontend proxy for my podman services.
Instead of tried and true haproxy, I tried out traefik.
Why Traefik?
- Dynamic config sections
- Up-skill in traefik
- Much more flexible routing
- Choose HTTP support or raw TCP, mix and match as you please
- Perfect choice for flexible ingress in homelab or enterprise environments
How?
Due to needing to operate across different networks, I chose to run the traefik binary directly on the host. Since the entrypoints section must be static, I ended up writing a .j2 template for traefik.yml:
traefik.yml.j2
# ansible managed
#accessLog:
# format: json
entryPoints:
{% if br110_ip is defined %}
infrastructure.http:
address: "{{ br110_ip }}:80"
infrastructure.https:
address: "{{ br110_ip }}:443"
http:
tls: {}
infrastructure.docker:
address: "{{ br110_ip }}:5000"
infrastructure.dockers:
address: "{{ br110_ip }}:5001"
infrastructure.mqtts:
address: "{{ br110_ip }}:8883/tcp"
infrastructure.otel-grpc:
address: "{{ br110_ip }}:4317"
infrastructure.otel-http:
address: "{{ br110_ip }}:4318"
{% endif %}
{% if default_ip is defined %}
default.http:
address: "{{ default_ip }}:80"
default.https:
address: "{{ default_ip }}:443"
http:
tls: {}
default.alertmanager:
address: "{{ default_ip }}:9093"
default.blackbox:
address: "{{ default_ip }}:9115"
{% endif %}
providers:
file:
directory: /etc/traefik/traefik.d
watch: true
log:
level: INFO
The template configures itself to listen on specific IP addresses to avoid port clashes and expose services selectively.
The template gets pre-processed by a simple script and jinja2:
**/usr/local/bin/configure_traefik.sh
#!/bin/bash
# generate traefik.yml by merging IP addresses with traefik template
for iface in $(ip -o -4 addr show | awk '{print $2}' | grep '^br'); do
ip=$(ip -o -4 addr show "$iface" | awk '{print $4}' | cut -d/ -f1)
export "${iface}_ip=$ip"
done
# wait up to 60 seconds for an IP
for i in {1..60}; do
export default_ip=$(hostname -i)
if [ -n "$default_ip" ]; then
break
fi
sleep 1
done
if [ -z "$default_ip" ] ; then
echo "no default IP after waiting!"
exit 1
fi
# run j2 with all in-scope environment variables available
echo "rebuild traefik config file"
j2 -e "" -o /etc/traefik/traefik.yml /etc/traefik/traefik.yml.j2
Each time the service is restarted, the script is called due to ExecStartPre and this generates the final traefik.yml file:
traefik.service
[Unit]
Description=Traefik Reverse Proxy
# online target does not work with dhcpcd
After=network.target
[Service]
# Grant the specific capability we need (ambient set preserves it across execs if any)
AmbientCapabilities=CAP_NET_BIND_SERVICE
# Optional but strongly recommended: restrict to ONLY this capability
# (prevents the process from gaining others accidentally)
# allow bind port < 1024 without root
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
# Good security hygiene: prevent privilege escalation tricks
NoNewPrivileges=yes
User=traefik
Group=traefik
Type=simple
# runs as traefik user
ExecStartPre=/usr/local/bin/configure_traefik.sh
ExecStart=/opt/traefik/traefik --configFile=/etc/traefik/traefik.yml
Restart=always
[Install]
WantedBy=multi-user.target
TLS config
TLS can only be configured with a dynamic file placed in scanned directory as configured above (/etc/traefik/traefik.d):
/etc/traefik/traefik.d/tls.yml
tls:
stores:
default:
defaultCertificate:
certFile: /etc/traefik/tls/wildcard.infrastructure.asio.cert.fullchain.pem
keyFile: /etc/traefik/tls/wildcard.infrastructure.asio.key.pem
Per-service drop-ins
The final part of the puzzle was to create drop-ins for each required service in the /etc/traefik/traefik.d/ directory. Here’s my most complicated example - Nexus:
/etc/traefik/traefik.d/nexus.infrastructure.asio.yml
# ansible managed
#
# HTTP
#
http:
routers:
nexus-http:
# HostSNI only for TCP - we get the real headers for http routes
# as TLS already terminated and http call parsed
rule: "Host(`nexus.infrastructure.asio`)"
entryPoints:
- infrastructure.http
service: nexus-8081
docker:
# HostSNI only for TCP - we get the real headers for http routes
# as TLS already terminated and http call parsed
rule: "Host(`nexus.infrastructure.asio`)"
entryPoints:
- infrastructure.docker
service: nexus-5000
services:
nexus-8081:
loadBalancer:
servers:
- url: "http://localhost:8081"
nexus-5000:
loadBalancer:
servers:
- url: "http://localhost:5000"
#
# TCP
#
tcp:
routers:
nexus-https:
rule: "HostSNI(`nexus.infrastructure.asio`)"
tls:
passthrough: true
entryPoints:
- infrastructure.https
service: nexus-8443
dockers:
rule: "HostSNI(`nexus.infrastructure.asio`)"
tls:
passthrough: true
entryPoints:
- infrastructure.dockers
service: nexus-5001
services:
nexus-8443:
loadBalancer:
servers:
- address: "localhost:8443"
nexus-5001:
loadBalancer:
servers:
- address: "localhost:5001"
Of course, I had some ansible create all of these codes for me from some basic per-host structured variables :)
Verdict
After automating the above for my podman services, I finally have:
- Services available in the right VLAN
- No more MACVLAN DHCP chaos
- Correct host/VLAN isolation (see gotchas - below)
- TLS everywhere with wildcard certificate
- Easy, repeatable method for adding new podman services
Why not Kubernetes?
These are my vital network services that Kubernetes needs itself - my other clusters use this one extensively, so these need to be amongst the most bulletproof services in my home lab and fool-proof containers/VMs are perfect for this.
Gotchas
There were some unique challenges I had with traefik which are not well documented:
- If you have any
entrypointsspecified, you cannot override them with arguments or environment variables - You cannot configure your TLS certificates in the static section. They are only processed in the dynamic config files
- Static config files are fully static - no variables, etc
- Cannot use this pattern to bridge host-isolated VLANS, use a VM instead
- Podman Quadlet services must bind to localhost only, to avoid port collision, like this:
/etc/containers/systemd/nexus-pod.kube
[Install]
WantedBy=default.target
[Unit]
[Kube]
Yaml=/etc/containers/systemd/nexus-pod.yml
# web UI
PublishPort=127.0.0.1:8081:8081
# h2 database (nothing normally listens, not exposed outside host)
PublishPort=127.0.0.1:1234:1234
# web UI + TLS
PublishPort=127.0.0.1:8443:8443
# docker
PublishPort=127.0.0.1:5000:5000
# docker + TLS
PublishPort=127.0.0.1:5001:5001