RSS/AtomClickStack Homelab
I stumbled on ClickStack a couple of months ago. For a long time I’ve been looking for a logging solution for the homelab. I’ve played with Loki before but never got arround to onboarding all my lab systems.
If you want to know more about deploying and using ClickStack, The Manual is excellent.
Based on this doc, I was able to get up and running with the docker.io/clickhouse/clickstack-all-in-one container very quickly in docker on my laptop.
The all-in-one image bundles everything you need for a functioning log collector and frontend:
- ClickHouse database
- HyperDX UI
- OTEL collectors
- Glue scripts, etc example Dockerfile
ClickStack into “production”
Disclaimer: this is for a homelab!
I found a permanent home for ClickStack on my “services” mini PC.
Podman Quadlet
I was able to run ClickStack as a first class homelab service quite easily with Podman Quadlet like I do for some of my other lab services, like this:
/etc/containers//systemd/clickstack-pod.kube
[Install]
WantedBy=default.target
[Unit]
[Kube]
Yaml=/etc/containers/systemd/clickstack-pod.yml
# ui/hyperdx
PublishPort=127.0.0.1:8080:8080
# clickhouse/db
PublishPort=127.0.0.1:8123:8123
# otel/grpc
PublishPort=127.0.0.1:4317:4317
# otel/http
PublishPort=127.0.0.1:4318:4318
/etc/containers//systemd/clickstack-pod.yml
apiVersion: v1
kind: Pod
metadata:
annotations:
io.kubernetes.cri-o.ContainerType/app: container
io.kubernetes.cri-o.TTY/app: "false"
io.podman.annotations.autoremove/app: "FALSE"
io.podman.annotations.init/app: "FALSE"
io.podman.annotations.privileged/app: "FALSE"
io.podman.annotations.publish-all/app: "FALSE"
labels:
app: clickstack
name: clickstack
spec:
automountServiceAccountToken: false
dnsPolicy: None
containers:
- image: docker.io/clickhouse/clickstack-all-in-one:2.15.1
name: app
env:
- name: FRONTEND_URL
value: "https://clickstack.infrastructure.asio"
ports:
- containerPort: 8123
hostPort: 8123
- containerPort: 8080
hostPort: 8080
- containerPort: 4317
hostPort: 4317
- containerPort: 4318
hostPort: 4318
resources: {}
securityContext:
capabilities:
drop:
- NET_ADMIN
- CAP_MKNOD
- CAP_AUDIT_WRITE
volumeMounts:
- mountPath: /data/db
name: clickstack-data-vol
subPath: db
- mountPath: /var/lib/clickhouse
name: clickstack-data-vol
subPath: clickhouse
- mountPath: /var/log/clickhouse-server
name: clickstack-log-vol
enableServiceLinks: false
hostname: clickstack
restartPolicy: Never
volumes:
- hostPath:
path: /data/containers/clickstack/data
type: Directory
name: clickstack-data-vol
- hostPath:
path: /data/containers/clickstack/log
type: Directory
name: clickstack-log-vol
status: {}
Traefik frontend
With traefik already setup on the host, I only needed a single drop-in:
/etc/traefik/traefik.d/clickstack.infrastructure.asio.yml
# ansible managed
#
# HTTP
#
http:
routers:
clickstack-http:
# HostSNI only for TCP - we get the real headers for http routes
# as TLS already terminated and http call parsed
rule: "Host(`clickstack.infrastructure.asio`)"
tls: {}
entryPoints:
- infrastructure.https
service: clickstack-8080
services:
clickstack-8080:
loadBalancer:
servers:
- url: "http://localhost:8080"
#
# TCP
#
tcp:
routers:
otel-grpc:
# always match - if not tls, there is no such thing as a header
# so there is nothing to check against so just accept any value
rule: "HostSNI(`*`)"
entryPoints:
- infrastructure.otel-grpc
service: clickstack-4317
otel-http:
# always match - if not tls, there is no such thing as a header
# so there is nothing to check against so just accept any value
rule: "HostSNI(`*`)"
entryPoints:
- infrastructure.otel-http
service: clickstack-4318
services:
clickstack-4317:
loadBalancer:
servers:
- address: "localhost:4317"
clickstack-4318:
loadBalancer:
servers:
- address: "localhost:4318"
Setting up ClickStack
With these items setup, hitting the URL I configured in the browser got me straight to a set password dialog in HyperDX. Done.
Client setup
Now its time to send logs to ClickStack. I have a lone Windows machine on the network which is a source of constant pain, so this is the perfect starting point.
To collect logs from Windows:
- Install OTEL client for windows and install it as a windows service
- Install the OTEL contrib files for windows which are needed to collect the logs from Event Viewer
- Finally, configure the OTEL client in
C:\Program Files\OpenTelemetry Collector\config.yaml. You will need a token for the ClickStack OTEL collector which you can find in the HyperDX UI by looking forYour Ingestion API Key:
# To limit exposure to denial of service attacks, change the host in endpoints below from 0.0.0.0 to a specific network interface.
# See https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks
receivers:
windowseventlog/application:
channel: application
windowseventlog/system:
channel: system
processors:
batch:
resource:
attributes:
- key: host.os
value: windows
action: insert
exporters:
otlp/logs:
endpoint: clickstack.infrastructure.asio:4317
tls:
insecure: true
headers:
authorization: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
compression: gzip
service:
pipelines:
logs:
receivers:
- windowseventlog/application
- windowseventlog/system
processors:
- batch
exporters:
- otlp/logs
If everything worked, windows will start sending event logs to the OTEL collector previously exposed, and you will see events landing live in the UI:
Verdict
This is a very slick and easy to use stack. The hardest part of setting this up turned out to be the OTEL client on Windows. I think its safe to say if it works on Windows, I’ll be able to collect the rest of the logs I need no problem, although I’m sure kubernetes logs will be a pain.
For an enterprise grade logging solution, I’d be looking for a more scalable deployment architecture probably running on Kubernetes with a support contract but for my little homelab this is perfect.
Pretty sure both of those asks are available…