Skip to content

VPS Headscale and Headplane

Updated: 2026-07-01

This page documents the public VPS Headscale control plane deployed on ovps-me. The service mirrors the older Docker Compose pattern from ovps-ben, but runs under rootless Podman Quadlets on Ubuntu 24.04.

Service Roles

Component Role in KH3
Headscale Self-hosted coordination/control plane for the KH3 tailnet at vpn.kh3group.com
Headplane Administrative web UI for Headscale at vpn-ui.kh3group.com
Caddy Public TLS termination and reverse proxy for Headscale and Headplane
Tailscale clients Endpoint software on servers, workstations, phones, or future subnet routers

Headscale coordinates identity, node registration, keys, routes, DNS settings, and policy. It is not the data tunnel itself. Client-to-client traffic is handled by the Tailscale client using WireGuard, directly where possible and through relays when direct connectivity is not possible.

Hosts

Host Role Notes
ovps-me Active Headscale, Headplane, and Caddy VPS Ubuntu 24.04.4 minimal image; rootless Podman as podsvc
ovps-ben Reference source host Ubuntu 22.04 Docker setup used as migration reference
pve First infrastructure client Proxmox VE 9.2 on Debian 13; joined as Headscale node pve

The reference setup on ovps-ben used Docker Compose services named headscale, headplane, and caddy on an external Docker network named scale-net. The active ovps-me deployment preserves those service names and network shape under Podman.

Public Names

Name Service
vpn.kh3group.com Headscale control plane
vpn-ui.kh3group.com Headplane UI
tailnet.kh3group.com Headscale MagicDNS base domain

External DNS records for vpn.kh3group.com and vpn-ui.kh3group.com must point to ovps-me. Caddy uses Cloudflare DNS-01 for certificates, so certificate issuance can succeed even before local resolvers can resolve the public names.

Current Tailnet Nodes

Node Owner Tailnet IPs Notes
pve infra 100.64.0.1, fd7a:115c:a1e0::1 First Proxmox host; former/bootstrap subnet router; currently not advertising routes
archlinux [email protected] 100.64.0.2, fd7a:115c:a1e0::2 Linux workstation client; use as a generic pattern only, not an infrastructure constant
ts-router infra 100.64.0.3, fd7a:115c:a1e0::3 CT 105 dedicated subnet router for 192.168.2.0/24

Infrastructure machines should be owned by a service identity such as infra, not by a human OIDC user. This keeps server lifecycle and audit separate from a person's account. Use a human OIDC user for Headplane administration, and use infra for always-on machines.

Short-lived pre-auth keys are only enrollment credentials. Once a node joins, the node receives its own machine key and remains registered. The pre-auth key can expire without disconnecting the node.

Joining a machine to this Headscale control plane gives it a tailnet identity and KH3 tailnet IP. It does not automatically make the joined machine a tunnel to every local LAN resource behind it. Access to non-Tailscale LAN IPs requires a deliberate subnet-router design and route approval. Access to the internet through a KH3 node requires a deliberate exit-node design.

The first approved subnet route is 192.168.2.0/24, now served by ts-router. pve remains approved for rollback but does not currently advertise the route, so it is not available or serving. Clients must opt in with --accept-routes before using the route.

Current Headscale route state validated on 2026-07-01:

Node Approved Available Serving primary
pve 192.168.2.0/24 none none
ts-router 192.168.2.0/24 192.168.2.0/24 192.168.2.0/24

Runtime Layout

Purpose Path
Headscale config /opt/podman/config/headscale/config.yml
Headscale data /opt/podman/volumes/headscale/lib/
Headplane config /opt/podman/config/headplane/config.yml
Headplane data /opt/podman/volumes/headplane/plane-data/
Caddyfile /opt/podman/config/caddy/Caddyfile
Caddy env file /opt/podman/env/caddy.env
Caddy data and certs /opt/podman/volumes/caddy/data/
Caddy runtime config /opt/podman/volumes/caddy/config/
Rootless Quadlets /home/podsvc/.config/containers/systemd/

Do not commit the contents of the config or env files. They contain OIDC secrets, Cloudflare API material, Headscale API keys, and session secrets.

Services

Rootless Podman runs as:

user: podsvc
uid/gid: 2000:2000
linger: enabled

The active Quadlets are:

scale-net.network
headscale.container
headplane.container
caddy.container

The active images are:

docker.io/headscale/headscale:stable
ghcr.io/tale/headplane:latest
localhost/caddy-cloudflare:latest

The Caddy image was copied from the prebuilt caddy-web:latest Docker image on ovps-ben and retagged on ovps-me. Avoid rebuilding it on the 1 GiB VPS unless there is no alternative.

Caddy

Caddy publishes:

80/tcp
443/tcp
443/udp

The Caddyfile uses the Cloudflare DNS provider and explicit public resolvers:

(tls_cloudflare) {
  tls {
    dns cloudflare {env.CLOUDFLARE_API_TOKEN}
    resolvers 1.1.1.1 1.0.0.1
  }
}

vpn.kh3group.com {
  import tls_cloudflare
  reverse_proxy headscale:8080
}

vpn-ui.kh3group.com {
  import tls_cloudflare
  reverse_proxy headplane:3000
}

The explicit resolver block avoids Podman bridge DNS propagation failures during ACME DNS-01 validation.

Headscale

Headscale uses:

server_url: https://vpn.kh3group.com
listen_addr: 0.0.0.0:8080
database: sqlite at /var/lib/headscale/db.sqlite
MagicDNS base domain: tailnet.kh3group.com
OIDC issuer: https://accounts.google.com
OIDC allowed domain: kh3group.com

Config file source of truth:

/opt/podman/config/headscale/config.yml

The Headscale container listens without TLS because Caddy terminates public TLS and proxies to headscale:8080 on the internal scale-net Podman network.

The container healthcheck from the source Docker image was not retained in the Quadlet. In this deployment it produced a false unhealthy state, while the actual service health endpoint passed.

Validate health from the VPS:

sudo -iu podsvc podman exec caddy wget -qO- http://headscale:8080/health

Expected result:

{"status":"pass"}

Headplane

Headplane uses:

base_url: https://vpn-ui.kh3group.com
headscale.url: http://headscale:8080
headscale.public_url: https://vpn.kh3group.com
config_path: /etc/headscale/config.yml
OIDC issuer: https://accounts.google.com

Config file source of truth:

/opt/podman/config/headplane/config.yml

Headplane requires:

  • server.cookie_secret exactly 32 characters long;
  • a Headscale API key in both Headplane API key fields;
  • Google OIDC client ID and client secret for the vpn-ui.kh3group.com app.

Generate a replacement Headscale API key only after Headscale is running:

sudo -iu podsvc podman exec headscale headscale apikeys create --expiration 90d

Store the generated key only in /opt/podman/config/headplane/config.yml.

Headplane OIDC and Admin Model

Headplane SSO uses Google OIDC for the kh3group.com domain. The first OIDC user who signed in became the Headplane Owner. In the Headplane database this was visible as the first user row with caps=65535; the UI displayed that person's email address and full name from OIDC/session data.

Do not assume every domain user becomes an administrator. With the current expected model:

  • the first OIDC login is Owner;
  • later OIDC users default to Member unless a Headplane role is assigned;
  • Member has no UI access;
  • API-key login bypasses the Headplane role model and effectively has full Headscale administrative access because possession of the API key is already administrative.

Keep disable_api_key_login under review. Leaving API-key login enabled is useful for recovery, but the Headscale API key must be treated as a full administrative secret.

Hardening Baseline

The ovps-me VPS was hardened during deployment:

Control State
UFW Active; default deny incoming, allow outgoing
Allowed inbound ports 22/tcp, 80/tcp, 443/tcp
fail2ban Active with SSH jail
rpcbind Disabled, masked, inactive
SSH root login Disabled
SSH password auth Disabled
SSH X11 forwarding Disabled
Rootless low-port bind net.ipv4.ip_unprivileged_port_start=80
Swap 2 GiB /swapfile

Oracle Image Firewall Pitfall

The Oracle Ubuntu image carried a persisted iptables ruleset in /etc/iptables/rules.v4 that accepted SSH and then rejected all other inbound traffic:

-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

When UFW was enabled during hardening, those legacy rules remained ahead of UFW's chains. UFW showed 80/tcp and 443/tcp as allowed, but external HTTPS still failed with No route to host because packets hit the earlier icmp-host-prohibited reject.

The fix was to back up /etc/iptables/rules.v4, remove only those two catch-all reject lines, delete the same live rules, and reload UFW:

ssh ovps-me 'cp -a /etc/iptables/rules.v4 /etc/iptables/rules.v4.pre-ufw-fix-YYYYMMDDTHHMMSSZ'
ssh ovps-me 'sed -i -e "/^-A INPUT -j REJECT --reject-with icmp-host-prohibited$/d" -e "/^-A FORWARD -j REJECT --reject-with icmp-host-prohibited$/d" /etc/iptables/rules.v4'
ssh ovps-me 'iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited 2>/dev/null || true'
ssh ovps-me 'iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited 2>/dev/null || true'
ssh ovps-me 'ufw reload'

Do not diagnose curl -vk https://127.0.0.1 as a TLS failure for this Caddy deployment. The Caddyfile only defines vpn.kh3group.com and vpn-ui.kh3group.com; a localhost probe does not send matching SNI and can return a TLS internal error. Test with the real hostname and forced address:

curl -vk --resolve vpn-ui.kh3group.com:443:127.0.0.1 https://vpn-ui.kh3group.com/
curl -vk --resolve vpn-ui.kh3group.com:443:141.147.20.184 https://vpn-ui.kh3group.com/

Validation

Use these checks after changing config or rebooting:

ssh ovps-me 'sudo -iu podsvc XDG_RUNTIME_DIR=/run/user/2000 systemctl --user --no-pager --state=running,failed list-units "headscale.service" "headplane.service" "caddy.service" "scale-net-network.service"'
ssh ovps-me 'sudo -iu podsvc podman ps --format "table {{.Names}}\t{{.Status}}\t{{.Networks}}\t{{.Ports}}"'
ssh ovps-me 'sudo -iu podsvc podman exec caddy wget -qO- http://headscale:8080/health'
ssh ovps-me 'sudo -iu podsvc podman logs --tail=80 caddy'
ssh ovps-me 'sudo ufw status verbose'
ssh ovps-me 'sudo iptables -nvL INPUT --line-numbers | sed -n "1,18p"'

Expected service state:

caddy      running, publishes 80/443
headscale  running on scale-net
headplane  running on scale-net

On 2026-06-30 Caddy obtained Let's Encrypt certificates for both public names, and Headplane successfully reached Headscale's OpenAPI endpoint. On 2026-07-01, external HTTPS was verified through Caddy after removing the stale Oracle image catch-all iptables rejects.

Official References

These links point to the official areas relevant to this deployment. KH3 operators should use this page as the local source of truth and use these only when checking upstream behavior or options.