Skip to content

Rootless Podman Restore Runbook

This is the active rebuild path for CT 101 podman-lxc. It replaces the abandoned Docker LXC exercise for the services listed here:

  • PostgreSQL
  • Forgejo
  • Vaultwarden
  • Adminer

The design is rootless-first. Proxmox still creates and manages the LXC as root, but the application containers run as the dedicated Linux user podsvc. For the broader operating standard, live service inventory, and future-service rules, use Podman Ecosystem Standards.

Layout

Purpose Path
Editable service configuration /opt/podman/config/<service>/
Secret env files /opt/podman/env/<service>.env
Service data /opt/podman/volumes/<service>/
Staged restore copy inside LXC /opt/podman/restore/proxmox-rebuild-20260614-191305
Rootless Quadlets /home/podsvc/.config/containers/systemd/
Rootless network kh3-backend

Use /opt/podman/config for human-edited config files that should be easy to inspect. Use /opt/podman/volumes for application data directories mounted into containers. Do not put service data in the Quadlet directory.

Why Rootless

Rootless Podman reduces the impact of a container escape or bad application write because the service process maps to podsvc, not root on the LXC. This is appropriate for PostgreSQL, Forgejo, Vaultwarden, and Adminer because the initial restore does not need to bind ports below 1024.

The first restored ports are high ports:

Service Container port LXC host port
Forgejo web 3000 30080
Forgejo SSH 22 2222
Vaultwarden web 80 30081
Adminer web 8080 30082
PostgreSQL 5432 Not published

Route low public ports such as 80 and 443 through the planned Caddy ingress LXC and OPNsense port-forwarding policy. Do not lower net.ipv4.ip_unprivileged_port_start just to make rootless containers bind privileged ports unless the security tradeoff is accepted and documented.

Shared PostgreSQL

A shared PostgreSQL container is acceptable for this environment when each application has its own database and login role. The restore creates:

Application Database Role
Forgejo forgejo forgejo
Vaultwarden vaultwarden vaultwarden

The historical backup uses gitea.dump and likely a gitea database role. The Podman restore intentionally imports gitea.dump into a new forgejo database and renames or creates the role as forgejo. The old database password is preserved from the recovered env file when present. In the verified backup, the Forgejo compose env did not contain a DB password; the password was stored in gitea/data/gitea/conf/app.ini under [database] PASSWD, so the restore script uses that as a fallback. The username and database name change.

Rootless file ownership matters. The restore script uses podman unshare to map ownership for container users:

Service Container owner Path
PostgreSQL 70:70 /opt/podman/volumes/postgres/data
Forgejo 1000:1000 /opt/podman/volumes/forgejo/data

The verified Vaultwarden env had SMTP_HOST but no SMTP_FROM. Current Vaultwarden refuses that configuration, so the restore script disables SMTP by removing the recovered SMTP keys when SMTP_FROM is absent. Configure mail manually after login if email sending is required. Do not add empty duplicate SMTP keys to the end of the env file; Podman may keep using the earlier value.

Scripts

Script Run from Purpose
scripts/podman/00-create-podman-lxc-101.sh Proxmox host as root Create or converge CT 101 podman-lxc
scripts/podman/01-bootstrap-rootless-podman-lxc.sh Inside CT 101 as root Install Podman, create podsvc, and prepare /opt/podman
scripts/podman/02-stage-backup-to-podman-lxc.sh Administration workstation Copy backup from NAS to local /tmp, then into the LXC
scripts/podman/03-generate-rootless-quadlets.sh Inside CT 101 as root Create env templates and rootless Quadlets
scripts/podman/04-restore-rootless-services.sh Inside CT 101 as root Restore app archives, import PostgreSQL dumps, and start services
scripts/podman/05-validate-rootless-services.sh Inside CT 101 as root Validate Podman, systemd user services, and restored databases

Answer File

Create a local answer file:

cp scripts/podman/podman-answer.env.example scripts/podman/podman-answer.env
chmod 600 scripts/podman/podman-answer.env

Review these values first:

CTID=101
HOSTNAME=podman-lxc
NET_IPV4=dhcp
GATEWAY=
NAMESERVER=192.168.2.2
ENABLE_TUN_DEVICE=1
PODMAN_USER=podsvc
PODMAN_ROOT=/opt/podman
NAS_BACKUP_PATH=/volume1/vm_backup/proxmox-rebuild-20260614-191305
LOCAL_STAGE_ROOT=/tmp/proxmox-rebuild-20260614-191305-staged
LXC_STAGE_ROOT=/opt/podman/restore/proxmox-rebuild-20260614-191305

Leave this blank until the destructive restore step:

CONFIRM_PODMAN_RESTORE=

Set it only when ready to restore application data and databases:

CONFIRM_PODMAN_RESTORE=restore-podman-services

Do not commit a filled answer file.

If CT creation succeeds but the container only shows an IPv6 link-local address and networking.service shows repeated DHCPDISCOVER lines, switch to the documented DMZ static address before bootstrap:

NET_IPV4=192.168.2.100/24
GATEWAY=192.168.2.1
NAMESERVER=1.1.1.1

Rerun 00-create-podman-lxc-101.sh. It accepts the existing podman-lxc container and updates net0 safely. The NAMESERVER=1.1.1.1 fallback is for bootstrap only when the internal DNS address 192.168.2.2 is unavailable; move back to Technitium after it is restored.

Keep ENABLE_TUN_DEVICE=1 for rootless Podman. Podman 5 uses pasta for rootless networking, and the restored CT showed this failure without the TUN device:

Failed to open() /dev/net/tun: No such file or directory
Failed to set up tap device in namespace

The creation script adds only these LXC config lines:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

This does not make the application containers rootful; it exposes the narrow kernel device needed by rootless Podman networking inside the unprivileged LXC.

Command Sequence

From the administration workstation, copy the Podman scripts to Proxmox and run the CT creation script there:

ssh pve 'mkdir -p /root/kh3-podman-restore'
scp scripts/podman/common.sh scripts/podman/00-create-podman-lxc-101.sh scripts/podman/podman-answer.env pve:/root/kh3-podman-restore/
ssh pve 'bash /root/kh3-podman-restore/00-create-podman-lxc-101.sh --answer-file /root/kh3-podman-restore/podman-answer.env'

Bootstrap rootless Podman inside CT 101:

ssh pve 'pct push 101 /root/kh3-podman-restore/podman-answer.env /root/podman-answer.env --perms 0600'
ssh pve 'pct exec 101 -- bash -s -- --answer-file /root/podman-answer.env' < scripts/podman/01-bootstrap-rootless-podman-lxc.sh

Generate the rootless Quadlets:

ssh pve 'pct exec 101 -- bash -s -- --answer-file /root/podman-answer.env' < scripts/podman/03-generate-rootless-quadlets.sh

Stage the NAS backup from the workstation:

scripts/podman/02-stage-backup-to-podman-lxc.sh --answer-file scripts/podman/podman-answer.env

During the June 22, 2026 Podman restore, the NAS alias changed to 192.168.0.50 on port 2242. SSH correctly refused the changed host key until the new fingerprint was verified. Do not bypass this check with StrictHostKeyChecking=no. After confirming the NAS identity, add the verified key with:

ssh -o StrictHostKeyChecking=accept-new -p 2242 nas 'hostname'

When ready to restore data, set the confirmation value in both copies of the answer file:

nano scripts/podman/podman-answer.env
scp scripts/podman/podman-answer.env pve:/root/kh3-podman-restore/podman-answer.env
ssh pve 'pct push 101 /root/kh3-podman-restore/podman-answer.env /root/podman-answer.env --perms 0600'

Run the restore and validation:

ssh pve 'pct exec 101 -- bash -s -- --answer-file /root/podman-answer.env' < scripts/podman/04-restore-rootless-services.sh
ssh pve 'pct exec 101 -- bash -s -- --answer-file /root/podman-answer.env' < scripts/podman/05-validate-rootless-services.sh

Expected Results

After bootstrap:

  • podsvc exists.
  • /etc/subuid and /etc/subgid contain podsvc:10000:50000. This is intentional for the unprivileged LXC because the CT itself is mapped to a bounded host ID range. The common 100000:65536 subordinate range failed with newuidmap: write to uid_map failed: Operation not permitted.
  • /opt/podman/config, /opt/podman/env, /opt/podman/volumes, and /opt/podman/restore exist.
  • podman info works as podsvc.
  • loginctl enable-linger podsvc has been run so user services can start at boot.

After Quadlet generation:

  • /home/podsvc/.config/containers/systemd/ contains .container and .network files.
  • Env files exist in /opt/podman/env.
  • Existing env files are not overwritten on normal reruns.

After staging:

  • The local staged backup contains .copy-complete.
  • The LXC staged backup contains .copy-complete.
  • Rerunning staging does not recopy unless FORCE_BACKUP_COPY=1.

After restore:

  • Rootless containers postgres, forgejo, vaultwarden, and adminer exist.
  • PostgreSQL answers pg_isready.
  • Databases forgejo and vaultwarden exist.
  • PostgreSQL reports healthy.
  • Forgejo, Vaultwarden, and Adminer answer on the LXC high ports.
  • Forgejo env points at postgres:5432, database forgejo, and role forgejo. If the recovered app.ini still names an old database host such as db2:5432, keep the env override and document the warning.
  • No required container is in a restart loop.
  • No user systemd unit is failed.

The scripted restore path currently covers the core restored services: PostgreSQL, Forgejo, Vaultwarden, and Adminer. The live CT 101 ecosystem also contains Dozzle and a Forgejo Actions runner Quadlet. Dozzle was running on 30083 during the June 26, 2026 check. The runner unit was failed and must be repaired before Forgejo Actions is treated as restored.

The June 22, 2026 strict validation passed with:

  • postgres healthy.
  • forgejo, vaultwarden, and adminer running.
  • PostgreSQL accepting connections.
  • Databases forgejo and vaultwarden present.
  • Forgejo env pointing at postgres:5432 / forgejo / forgejo.
  • Forgejo, Vaultwarden, and Adminer direct HTTP checks returning 200.
  • No failed user units.

Application-level login is still a separate check. Validate Forgejo at http://192.168.2.100:30080, Vaultwarden at http://192.168.2.100:30081, and Adminer at http://192.168.2.100:30082 from an approved network path.

Remaining Operator Steps

Before treating the restore as production-ready:

  • Restore or confirm Technitium DNS, then move CT 101 nameserver back from temporary public DNS to 192.168.2.2 when the new DNS LXC is available.
  • Route Forgejo, Vaultwarden, and Adminer through the Caddy ingress LXC after each high-port backend validates directly. Keep the high ports available for recovery checks during migration.
  • Log in to Forgejo and Vaultwarden with existing accounts and confirm expected data is present.
  • Leave Vaultwarden SMTP disabled unless a complete current mail configuration is supplied.
  • Create and verify a fresh backup from the restored Podman layout.

Safe Reruns

Normal reruns are expected:

  • CT creation accepts an existing CT only if the hostname is podman-lxc.
  • Bootstrap keeps the existing podsvc user and /opt/podman tree.
  • Staging skips copies when .copy-complete exists.
  • Quadlet generation replaces unit files but keeps edited env files.
  • Restore skips marked application and database restores unless FORCE_RESTORE=1.

FORCE_RESTORE=1 is destructive for restored data. Use it only after preserving the failed state for inspection.

Never Do This

  • Do not run these scripts against the old Docker CT.
  • Do not start services with CHANGE_ME_BEFORE_START in any env file.
  • Do not publish PostgreSQL to the network unless a specific service requires it and firewall rules are reviewed.
  • Do not manually delete restore markers to hide a failed restore. Preserve the state and either set a new stage path or intentionally use FORCE_RESTORE=1.
  • Do not blindly edit SSH known_hosts after a host key mismatch. Confirm the Proxmox or NAS identity first.