Keep ClickHouse Healthy with Docker Health Checks Now

Running ClickHouse inside a container is a great way to get a fast, column‑oriented database up and running in minutes. The trade‑off? Containers are stateless by design, so the surrounding orchestration layer has to watch over them. That’s where Docker health checks step in – they give you a simple, built‑in mechanism to ask the database, “Are you still breathing?” and act on the answer before users even notice a hiccup.

Why a health check matters for ClickHouse

ClickHouse excels at ingesting massive streams of data and serving analytical queries in sub‑second time. Yet, like any service, it can stumble: a corrupted config file, a full disk, or an unexpected shutdown can leave the container alive but unable to serve queries. Docker’s HEALTHCHECK directive runs a command at regular intervals, marking the container as healthy or unhealthy. Orchestrators such as Kubernetes or Docker Compose then know whether to restart the container, route traffic elsewhere, or raise an alert.

Choosing the right probe

ClickHouse offers a few built‑in ways to test its health:

• HTTP endpoint: /ping returns Ok. if the server is alive.
• Native client query: a simple SELECT 1 over the TCP protocol verifies that the query engine is functional.
• System tables: checking system.metrics for Uptime or MemoryUsage can surface deeper issues.

For most deployments the HTTP /ping is enough – it’s lightweight, needs no authentication by default, and works even when the TCP port is firewalled.

Crafting the Dockerfile

Adding a health check is just a single line in your Dockerfile. Here’s a minimal example that assumes you’re using the official ClickHouse image:

FROM clickhouse/clickhouse-server:latest
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD curl -f http://localhost:8123/ping || exit 1

The flags deserve a quick rundown:

• --interval – how often Docker runs the probe (30 seconds is a sensible default).
• --timeout – how long the command may run before Docker kills it.
• --retries – number of consecutive failures before the container is marked unhealthy.

If your ClickHouse instance sits behind a reverse proxy or uses a non‑standard port, adjust the URL accordingly.

Testing locally with Docker Compose

When you spin up a stack with docker-compose.yml, the health status becomes visible via docker compose ps:

services:
clickhouse:
image: clickhouse/clickhouse-server:latest
ports:
- "8123:8123"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8123/ping"]
interval: 30s
timeout: 5s
retries: 3

Run docker compose up -d, then docker compose ps. You’ll see healthy, starting, or unhealthy in the State column. This immediate feedback is priceless during development – you can spot configuration slips before they hit production.

Integrating with Kubernetes

In a Kubernetes world, the Docker health check translates to a livenessProbe and optionally a readinessProbe. The two aren’t identical: liveness tells the kubelet when to restart a pod, while readiness controls whether the service routes traffic to it.

apiVersion: apps/v1
kind: Deployment
metadata:
name: clickhouse
spec:
replicas: 3
selector:
matchLabels:
app: clickhouse
template:
metadata:
labels:
app: clickhouse
spec:
containers:
- name: clickhouse
image: clickhouse/clickhouse-server:latest
ports:
- containerPort: 8123
livenessProbe:
httpGet:
path: /ping
port: 8123
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /ping
port: 8123
initialDelaySeconds: 5
periodSeconds: 10

Notice the initialDelaySeconds – you give ClickHouse a few seconds to boot before the first check. Skipping that delay can cause false‑positive restarts during pod startup.

When to go beyond /ping

Even a successful /ping doesn’t guarantee that queries will succeed. If you’ve noticed intermittent timeouts or data corruption, consider a more thorough probe:

• Run SELECT 1 via the native client inside the health check. It validates the query path.
• Inspect system.errors for recent critical errors.
• Check disk space with df and abort if usage exceeds a threshold.

Here’s a Bash‑based health script that does a quick query and a disk check:

#!/bin/sh
# health.sh – place in /usr/local/bin and reference in Dockerfile
if ! curl -sf http://localhost:8123/ping >/dev/null; then
exit 1
fi
if ! clickhouse-client --query="SELECT 1" >/dev/null 2>&1; then
exit 1
fi
if [ $(df /var/lib/clickhouse | awk 'NR==2 {print $5}' | tr -d '%') -ge 90 ]; then
exit 1
fi
exit 0

Then update the HEALTHCHECK line to CMD /usr/local/bin/health.sh. This adds a small performance cost, but the extra safety net can be worth it for mission‑critical analytics pipelines.

What to do when a container is marked unhealthy

Docker itself won’t automatically restart an unhealthy container unless you’ve set --restart=always or are using an orchestrator. In Kubernetes, the pod is killed and recreated, which clears transient state. However, ClickHouse stores data on a volume, so a restart doesn’t lose data – just be sure the volume is persistent (e.g., hostPath, persistentVolumeClaim, or a cloud block store).

If you prefer manual handling, you can attach a docker events listener that watches for the health_status: unhealthy event and triggers a custom script. This approach works in bare‑metal Docker Swarm setups where you might want to alert a Slack channel before a restart.

Best‑practice checklist

• Start with the lightweight /ping probe; add deeper checks only when necessary.
• Set initialDelaySeconds (or Docker’s --start-period) to accommodate ClickHouse’s cold‑start time.
• Make health checks idempotent – they should never modify data.
• Log enough context inside the health script so that when a failure occurs you can quickly identify the cause.
• Combine liveness and readiness probes in Kubernetes to separate “is it alive?” from “is it ready to serve traffic?”.

By weaving Docker health checks into your ClickHouse deployment, you turn a passive container into a self‑monitoring service. The result? Fewer surprise outages, smoother rolling updates, and a database that stays as responsive as the queries it’s built to serve.