Skip to content

Restoration Validation

Historical restore checklist

Addresses, Docker networks, and service scope on this page belong to the June 2026 exercise. Rebuild a current checklist from the service catalog before a production restore.

Use this checklist for every restore exercise. Record the date, operator, answer file path, backup run ID, and any deviations.

Access Validation

Run from the administration workstation:

ssh pvessh 'hostname; command -v pct; command -v qm; pvesm status'
ssh -p 2242 nas 'test -d /volume1/vm_backup/proxmox-rebuild-20260614-191305 && ls -la /volume1/vm_backup/proxmox-rebuild-20260614-191305'

Required results:

  • Proxmox SSH succeeds.
  • pct and qm are present.
  • pvesm status succeeds.
  • NAS backup path exists.
  • Any direct ssh docker check is optional and guarded by a timeout.

Backup Staging Validation

  • scripts/restore/01-stage-backup.sh either copies the backup or reports that an existing staged copy is being used.
  • .copy-complete exists only after successful extraction.
  • archive-list.txt exists in the staged directory.
  • Staged files are outside the repository and treated as sensitive.
  • Broad NAS/CIFS mode bits are compensated for by Synology ACLs.

Artifact Validation

scripts/restore/02-validate-artifacts.sh must pass these checks:

  • tar -tzf succeeds for Docker, Proxmox, Forgejo, Vaultwarden, Traefik, Appsmith, and Homebox archives that are present.
  • gzip -t succeeds for all MariaDB .sql.gz files.
  • PostgreSQL dumps are validated with pg_restore --list when installed, otherwise file reports PostgreSQL custom-format dumps.
  • pfSense XML starts as XML and contains the expected pfSense XML marker.
  • Docker definitions contain compose and .env files for PostgreSQL, Forgejo, Vaultwarden, and Adminer.
  • Forgejo archive contains repositories/app data, app.ini, SSH host keys, runner config, and .runner.
  • Vaultwarden archive contains rsa_key.pem.
  • Traefik archive contains traefik.yml, conf.d, and acme.json.
  • Empty websites is recorded as a known warning, not ignored.

Podman Restore Validation

After 04-restore-rootless-services.sh and 05-validate-rootless-services.sh:

  • CT 101 podman-lxc is running at 192.168.2.100/24.
  • CT 101 remains unprivileged.
  • Rootless Podman commands work as podsvc.
  • /etc/subuid and /etc/subgid contain podsvc:10000:50000.
  • The rootless network kh3-backend exists.
  • PostgreSQL container is running and healthy.
  • PostgreSQL pg_isready succeeds.
  • Databases forgejo and vaultwarden exist.
  • Forgejo container is running and direct HTTP on port 30080 answers.
  • Vaultwarden container is running and direct HTTP on port 30081 answers.
  • Adminer container is running and direct HTTP on port 30082 answers.
  • Dozzle container is running and direct HTTP on port 30083 answers if it is in scope for the restore.
  • Forgejo env points to database host postgres:5432, database forgejo, and role forgejo.
  • If recovered Forgejo app.ini still names an old database host such as db2:5432, the env override is documented and validation still passes.
  • Vaultwarden SMTP is disabled when the recovered env lacks SMTP_FROM.
  • No user systemd unit is failed.
  • Forgejo Actions runner is either running and validated, or its failed / intentionally-disabled state is documented.

Docker Restore Validation

After 03-restore-docker-services.sh:

  • Docker network aproxy exists with subnet 172.18.0.0/16.
  • Docker network backend exists with subnet 172.19.0.0/16.
  • PostgreSQL pg_isready succeeds.
  • gitea database exists.
  • vaultwarden database exists.
  • Forgejo container is running.
  • Vaultwarden container is running.
  • Adminer container is running.
  • Runner is running, or the missing registration/config action is documented.
  • Traefik route checks are run if Traefik is restored or already available.
  • No required container is restarting or unhealthy.

Application Checks

Forgejo:

  • Web UI loads.
  • Existing admin login works.
  • Repositories, LFS objects, attachments, and SSH keys are present.
  • Git over SSH on port 2222 works.
  • Runner appears online and one known workflow runs, or manual registration is documented.

Vaultwarden:

  • Existing user login works.
  • Vault unlock works.
  • Attachments and sends are present.
  • SMTP and SSO/OIDC settings are validated if configured.
  • Admin token came from the approved source, not a generated replacement.

Adminer:

  • Adminer route or direct access loads only from the approved network path.
  • It can connect to PostgreSQL with a least-privilege test user.
  • Exposure is protected by firewall, VPN, or Traefik middleware.

Idempotency Validation

Run the active restore stages a second time with FORCE_BACKUP_COPY=0 and FORCE_RESTORE=0. For the current Podman path, use the Podman scripts. For a historical Docker exercise, use the Docker scripts.

  • Backup is not recopied from NAS.
  • LXC staged backup is not recopied.
  • Edited .env files are not overwritten.
  • PostgreSQL dumps are not re-imported.
  • Container networks are not duplicated.
  • Correct existing networks are accepted.
  • Wrong-subnet existing networks fail safely.
  • Service definitions converge without changing edited env files.
  • Existing correct CT 101 podman-lxc is accepted for the Podman path.
  • Existing correct CT 101 docker-host is accepted only for a historical Docker restore exercise.
  • Existing wrong CT ID or hostname fails before restore.

Completion Gate

The restore is not complete until:

  • This checklist is fully filled.
  • The validation report from 02-validate-artifacts.sh is retained.
  • The service validation output is retained.
  • Any manual reverse-proxy, runner, Traefik, DNS, or pfSense action is documented.
  • Forgejo and Vaultwarden login tests pass from an approved network path.
  • A fresh backup from the restored environment is created and verified.