geoffwilliams@home:~$

traefik frontend for podman

Following on from Podman in-container DHCP networking, I needed a frontend proxy for my podman services.

Instead of tried and true haproxy, I tried out traefik.

Why Traefik?

  • Dynamic config sections
  • Up-skill in traefik
  • Much more flexible routing
  • Choose HTTP support or raw TCP, mix and match as you please
  • Perfect choice for flexible ingress in homelab or enterprise environments

How?

Due to needing to operate across different networks, I chose to run the traefik binary directly on the host. Since the entrypoints section must be static, I ended up writing a .j2 template for traefik.yml:

traefik.yml.j2


# ansible managed

#accessLog: 
#  format: json

entryPoints:

{% if br110_ip is defined %}
  infrastructure.http:
    address: "{{ br110_ip }}:80"
  infrastructure.https:
    address: "{{ br110_ip }}:443"
    http:
      tls: {}    
  infrastructure.docker:
    address: "{{ br110_ip }}:5000"
  infrastructure.dockers:
    address: "{{ br110_ip }}:5001"
  infrastructure.mqtts:
    address: "{{ br110_ip }}:8883/tcp"
  infrastructure.otel-grpc:
    address: "{{ br110_ip }}:4317"
  infrastructure.otel-http:
    address: "{{ br110_ip }}:4318"    
{% endif %}

{% if default_ip is defined %}
  default.http:
    address: "{{ default_ip }}:80"
  default.https:
    address: "{{ default_ip }}:443"
    http:
      tls: {}
  default.alertmanager:
    address: "{{ default_ip }}:9093"
  default.blackbox:
    address: "{{ default_ip }}:9115"
{% endif %}

providers:
  file:
    directory: /etc/traefik/traefik.d
    watch: true


log:
  level: INFO

The template configures itself to listen on specific IP addresses to avoid port clashes and expose services selectively.

The template gets pre-processed by a simple script and jinja2:

**/usr/local/bin/configure_traefik.sh

#!/bin/bash

# generate traefik.yml by merging IP addresses with traefik template

for iface in $(ip -o -4 addr show | awk '{print $2}' | grep '^br'); do
    ip=$(ip -o -4 addr show "$iface" | awk '{print $4}' | cut -d/ -f1)
    export "${iface}_ip=$ip"
done

# wait up to 60 seconds for an IP
for i in {1..60}; do
    export default_ip=$(hostname -i)
    if [ -n "$default_ip" ]; then
        break
    fi
    sleep 1
done

if [ -z "$default_ip" ] ; then
    echo "no default IP after waiting!" 
    exit 1
fi

# run j2 with all in-scope environment variables available
echo "rebuild traefik config file"
j2 -e "" -o /etc/traefik/traefik.yml /etc/traefik/traefik.yml.j2

Each time the service is restarted, the script is called due to ExecStartPre and this generates the final traefik.yml file:

traefik.service

[Unit]
Description=Traefik Reverse Proxy

# online target does not work with dhcpcd
After=network.target

[Service]
# Grant the specific capability we need (ambient set preserves it across execs if any)
AmbientCapabilities=CAP_NET_BIND_SERVICE

# Optional but strongly recommended: restrict to ONLY this capability
# (prevents the process from gaining others accidentally)
# allow bind port < 1024 without root
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

# Good security hygiene: prevent privilege escalation tricks
NoNewPrivileges=yes

User=traefik
Group=traefik

Type=simple
# runs as traefik user
ExecStartPre=/usr/local/bin/configure_traefik.sh
ExecStart=/opt/traefik/traefik --configFile=/etc/traefik/traefik.yml
Restart=always

[Install]
WantedBy=multi-user.target

TLS config

TLS can only be configured with a dynamic file placed in scanned directory as configured above (/etc/traefik/traefik.d):

/etc/traefik/traefik.d/tls.yml

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /etc/traefik/tls/wildcard.infrastructure.asio.cert.fullchain.pem
        keyFile: /etc/traefik/tls/wildcard.infrastructure.asio.key.pem

Per-service drop-ins

The final part of the puzzle was to create drop-ins for each required service in the /etc/traefik/traefik.d/ directory. Here’s my most complicated example - Nexus:

/etc/traefik/traefik.d/nexus.infrastructure.asio.yml

# ansible managed

#
# HTTP
#
http:
  routers:
    nexus-http:
      # HostSNI only for TCP - we get the real headers for http routes
      # as TLS already terminated and http call parsed
      rule: "Host(`nexus.infrastructure.asio`)"


      entryPoints:
      - infrastructure.http
      service: nexus-8081
    docker:
      # HostSNI only for TCP - we get the real headers for http routes
      # as TLS already terminated and http call parsed
      rule: "Host(`nexus.infrastructure.asio`)"


      entryPoints:
      - infrastructure.docker
      service: nexus-5000

  services:
    nexus-8081:
      loadBalancer:
        servers:
          - url: "http://localhost:8081"
    nexus-5000:
      loadBalancer:
        servers:
          - url: "http://localhost:5000"

#
# TCP
#
tcp:
  routers:
    nexus-https:
      rule: "HostSNI(`nexus.infrastructure.asio`)"
      tls:
        passthrough: true
      entryPoints:
      - infrastructure.https
      service: nexus-8443
    dockers:
      rule: "HostSNI(`nexus.infrastructure.asio`)"
      tls:
        passthrough: true
      entryPoints:
      - infrastructure.dockers
      service: nexus-5001


  services:
    nexus-8443:
      loadBalancer:
        servers:
          - address: "localhost:8443"
    nexus-5001:
      loadBalancer:
        servers:
          - address: "localhost:5001"

Of course, I had some ansible create all of these codes for me from some basic per-host structured variables :)

Verdict

After automating the above for my podman services, I finally have:

  • Services available in the right VLAN
  • No more MACVLAN DHCP chaos
  • Correct host/VLAN isolation (see gotchas - below)
  • TLS everywhere with wildcard certificate
  • Easy, repeatable method for adding new podman services

Why not Kubernetes?

These are my vital network services that Kubernetes needs itself - my other clusters use this one extensively, so these need to be amongst the most bulletproof services in my home lab and fool-proof containers/VMs are perfect for this.

Gotchas

There were some unique challenges I had with traefik which are not well documented:

  • If you have any entrypoints specified, you cannot override them with arguments or environment variables
  • You cannot configure your TLS certificates in the static section. They are only processed in the dynamic config files
  • Static config files are fully static - no variables, etc
  • Cannot use this pattern to bridge host-isolated VLANS, use a VM instead
  • Podman Quadlet services must bind to localhost only, to avoid port collision, like this:

/etc/containers/systemd/nexus-pod.kube

[Install]
WantedBy=default.target

[Unit]

[Kube]
Yaml=/etc/containers/systemd/nexus-pod.yml

# web UI
PublishPort=127.0.0.1:8081:8081

# h2 database (nothing normally listens, not exposed outside host)
PublishPort=127.0.0.1:1234:1234

# web UI + TLS
PublishPort=127.0.0.1:8443:8443

# docker
PublishPort=127.0.0.1:5000:5000

# docker + TLS
PublishPort=127.0.0.1:5001:5001

Post comment

Markdown is allowed, HTML is not. All comments are moderated.