Skip to content

Podman Ecosystem Standards

Updated: 2026-07-02

This page records the operating pattern that replaced the earlier Docker LXC restore attempt. Use it when adding, restoring, or reviewing services in the current KH3 container platform.

Architecture Evolution

Date Change Result
2026-06-09 Runtime inventory captured around pfSense, Docker, Traefik, Pi-hole, and Cloudflared Historical baseline only
2026-06-14 to 2026-06-17 Docker restore exercise attempted from the NAS backup artifacts Useful artifact lessons, but not the active rebuild path
2026-06-22 CT 101 podman-lxc restored as an unprivileged LXC running rootless Podman as podsvc Active application and data host
2026-06-22 VM 100 router verified as OPNsense OPNsense is the active routing, DHCP, NAT, and DNS-enforcement point
2026-06-22 CT 102 technitium-dns created at 192.168.2.2 DNS replacement for the historical Pi-hole/Cloudflared role
2026-06-22 CT 103 caddy-ingress created at 192.168.2.3 HTTP/HTTPS ingress replacement for Traefik
2026-06-26 Live state rechecked from Proxmox CTs 101, 102, 103, and 104 are running; dozzle is running in rootless Podman; Forgejo runner unit needs repair
2026-07-02 Documentation site restored through rootless static service docs-static serves MkDocs artifacts on 30084; Caddy routes docs.kh3group.com to CT 101

The old Docker, Traefik, Pi-hole, Cloudflared, and pfSense pages are retained as recovery history. Do not use them as the source of truth for new changes unless a page explicitly says the old component is still active.

Current Topology

Role Target Address Pattern
Firewall/router VM 100 router 192.168.2.1 on DMZ OPNsense
Application containers CT 101 podman-lxc 192.168.2.20 Unprivileged LXC, rootless Podman as podsvc
DNS CT 102 technitium-dns 192.168.2.2 Dedicated unprivileged LXC
HTTP/HTTPS ingress CT 103 caddy-ingress 192.168.2.3 Dedicated unprivileged LXC, Caddy binary with narrow low-port capability
Static site workload CT 104 khysite 192.168.2.5 Separate unprivileged LXC, role still needs a full service page

CT 101 uses Technitium for DNS. Its Proxmox config has:

nameserver: 192.168.2.2

CT 101 Rootless Podman Baseline

CT 101 podman-lxc is deliberately unprivileged:

unprivileged: 1
features: nesting=1,keyctl=1

Rootless Podman runs as:

user: podsvc
uid/gid: 2000:2000

The subordinate ID range inside the unprivileged LXC must stay inside the container-visible ID range:

/etc/subuid: podsvc:10000:50000
/etc/subgid: podsvc:10000:50000

Do not replace that with the usual 100000:65536; it failed in this LXC with newuidmap: write to uid_map failed: Operation not permitted.

Podman 5 rootless networking needs the LXC TUN device for pasta. The CT creation script adds only this narrow passthrough:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

This does not make application containers rootful.

File Layout

Use this layout for every rootless Podman service on CT 101:

Purpose Path
Human-edited service config /opt/podman/config/<service>/
Secret env file /opt/podman/env/<service>.env
Persistent service data /opt/podman/volumes/<service>/
Rootless Quadlet unit /home/podsvc/.config/containers/systemd/<service>.container
Rootless network unit /home/podsvc/.config/containers/systemd/kh3-backend.network
Staged restore data /opt/podman/restore/proxmox-rebuild-20260614-191305
Temporary extraction/work files /opt/podman/tmp/

Do not store data under the Quadlet directory. Quadlets describe services; data lives under /opt/podman/volumes, and secrets live under /opt/podman/env.

Live Rootless Services

Verified from CT 101 on 2026-06-26:

Service State Published port Notes
postgres Running, healthy Not published Shared PostgreSQL on kh3-backend
forgejo Running 30080->3000, 2222->22 Uses database forgejo and role forgejo
vaultwarden Running 30081->80 SMTP disabled when recovered config lacks SMTP_FROM
adminer Running 30082->8080 Must stay protected by Caddy/admin policy before broad access
dozzle Running 30083->8080 Rootless log viewer using the podsvc Podman socket
docs-static Running 30084->8080 Serves generated MkDocs artifacts from the runner data volume
runner Running Not published Forgejo Actions runner using rootless Podman socket

The current Quadlet directory contains:

adminer.container
dozzle.container
docs-static.container
forgejo.container
kh3-backend.network
postgres.container
runner.container
vaultwarden.container

The repository restore script currently generates the core restored services: PostgreSQL, Forgejo, Vaultwarden, and Adminer. dozzle, docs-static, and runner exist in the live Quadlet directory and should either be added to a future script pass or documented as post-restore manual services before relying on a fresh rebuild.

New Service Pattern

Use rootless Podman in CT 101 when the service:

  • is an application, worker, database, or internal tool;
  • can run on an unprivileged high port or only on the internal Podman network;
  • benefits from shared PostgreSQL or proximity to existing app data;
  • can be restored from /opt/podman/config, /opt/podman/env, and /opt/podman/volumes.

Use a separate unprivileged LXC when the service:

  • is infrastructure needed before app recovery, such as DNS;
  • is public ingress or holds certificate/OIDC material;
  • needs simple, reliable low-port binding such as 53, 80, or 443;
  • should have a smaller blast radius than the app/data host.

This is why Technitium and Caddy are not rootless Podman services in CT 101.

Quadlet Standard

Create services as rootless Quadlets under:

/home/podsvc/.config/containers/systemd/

Use this shape:

[Unit]
Description=KH3 <service> rootless container
After=postgres.service
Requires=postgres.service

[Container]
Image=<registry>/<image>:<tag>
ContainerName=<service>
Network=kh3-backend.network
EnvironmentFile=/opt/podman/env/<service>.env
Volume=/opt/podman/volumes/<service>:/data
PublishPort=<high-host-port>:<container-port>
AutoUpdate=registry

[Install]
WantedBy=default.target

Only include After=postgres.service and Requires=postgres.service for services that actually require PostgreSQL. Use high host ports on CT 101; let Caddy and OPNsense handle normal user-facing ports.

After editing Quadlets:

ssh pve 'pct exec 101 -- runuser -u podsvc -- env HOME=/home/podsvc XDG_RUNTIME_DIR=/run/user/2000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/2000/bus bash -lc "cd \$HOME && systemctl --user daemon-reload"'
ssh pve 'pct exec 101 -- runuser -u podsvc -- env HOME=/home/podsvc XDG_RUNTIME_DIR=/run/user/2000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/2000/bus bash -lc "cd \$HOME && systemctl --user enable --now <service>.service"'

Always run rootless Podman and user systemd commands from podsvc home. Running from /root caused cannot chdir to /root: Permission denied.

Secrets and Env Files

Rules:

  • Do not commit scripts/podman/podman-answer.env.
  • Do not print /opt/podman/env/*.env.
  • Do not paste recovered app.ini, private keys, ACME stores, OPNsense exports, or backup contents into docs.
  • Store only names, paths, ports, and validation results in the repository.
  • Remove stale duplicate env keys before adding overrides. Do not rely on an empty duplicate assignment at the end of a file to override an earlier value.

The restore script already applies this pattern for Forgejo and Vaultwarden:

  • Forgejo imports the historical gitea.dump into database forgejo.
  • Forgejo uses env overrides for postgres:5432, database forgejo, and role forgejo.
  • Vaultwarden SMTP keys are removed when the recovered env has SMTP_HOST but no SMTP_FROM.

Ownership Rules

Rootless volume ownership must be mapped with podman unshare when the image runs as a non-root container user:

podman unshare chown -R 70:70 /opt/podman/volumes/postgres/data
podman unshare chown -R 1000:1000 /opt/podman/volumes/forgejo/data

Do not fix these by broadening filesystem permissions. Use the container UID/GID expected by the image.

Network and Ingress Rules

CT 101 should not expose normal public service ports. Current high-port pattern:

Service User-facing route CT 101 backend
Forgejo https://git.kh3group.com through Caddy 192.168.2.20:30080
Forgejo SSH Direct or firewall-controlled SSH 192.168.2.20:2222
Vaultwarden https://pass.kh3group.com through Caddy 192.168.2.20:30081
Adminer https://dbgui.kh3group.com through Caddy only after admin policy 192.168.2.20:30082
Dozzle Admin-only route still to be designed 192.168.2.20:30083
Documentation site https://docs.kh3group.com through Caddy 192.168.2.20:30084

DNS is handled by Technitium at 192.168.2.2. Ingress is handled by Caddy at 192.168.2.3. OPNsense is responsible for DHCP, NAT, firewall enforcement, and DNS bypass controls.

Validation Standard

Run the rootless service validation after any CT 101 change:

ssh pve 'pct exec 101 -- bash -s -- --answer-file /root/podman-answer.env' < scripts/podman/05-validate-rootless-services.sh

Also check the full user service state:

ssh pve 'pct exec 101 -- runuser -u podsvc -- env HOME=/home/podsvc XDG_RUNTIME_DIR=/run/user/2000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/2000/bus bash -lc "cd \$HOME && systemctl --user --no-pager --type=service --state=running,failed list-units"'

Expected for a healthy app stack:

  • Core containers are running.
  • PostgreSQL is healthy.
  • Databases forgejo and vaultwarden exist.
  • Forgejo env points to postgres:5432, database forgejo, and role forgejo.
  • Direct HTTP checks on 30080, 30081, and 30082 pass.
  • No unexpected user unit is failed.

As of 2026-07-02, runner.service and docs-static.service are running.

When Updating Scripts

Keep script behavior idempotent:

  • Write templates only when missing unless the file is generated state.
  • Replace Quadlet unit files deterministically.
  • Keep env files if they have already been recovered or edited.
  • Use marker files for completed restore steps.
  • Require explicit confirmation before writing application data or importing databases.
  • Use FORCE_RESTORE=1 only when intentionally reapplying restored data.

If a live service exists but the scripts do not recreate it, either add it to the scripts or document it as a manual post-restore step. Do not leave live-only services invisible to the runbook.