Skip to content

Restoration Validation

Use this checklist for every restore exercise. Record the date, operator, answer file path, backup run ID, and any deviations.

Access Validation

Run from the administration workstation:

ssh pve 'hostname; command -v pct; command -v qm; pvesm status'
ssh -p 2242 nas 'test -d /volume1/vm_backup/proxmox-rebuild-20260614-191305 && ls -la /volume1/vm_backup/proxmox-rebuild-20260614-191305'

Required results:

  • Proxmox SSH succeeds.
  • pct and qm are present.
  • pvesm status succeeds.
  • NAS backup path exists.
  • Any direct ssh docker check is optional and guarded by a timeout.

Backup Staging Validation

  • scripts/restore/01-stage-backup.sh either copies the backup or reports that an existing staged copy is being used.
  • .copy-complete exists only after successful extraction.
  • archive-list.txt exists in the staged directory.
  • Staged files are outside the repository and treated as sensitive.
  • Broad NAS/CIFS mode bits are compensated for by Synology ACLs.

Artifact Validation

scripts/restore/02-validate-artifacts.sh must pass these checks:

  • tar -tzf succeeds for Docker, Proxmox, Forgejo, Vaultwarden, Traefik, Appsmith, and Homebox archives that are present.
  • gzip -t succeeds for all MariaDB .sql.gz files.
  • PostgreSQL dumps are validated with pg_restore --list when installed, otherwise file reports PostgreSQL custom-format dumps.
  • pfSense XML starts as XML and contains the expected pfSense XML marker.
  • Docker definitions contain compose and .env files for PostgreSQL, Forgejo, Vaultwarden, and Adminer.
  • Forgejo archive contains repositories/app data, app.ini, SSH host keys, runner config, and .runner.
  • Vaultwarden archive contains rsa_key.pem.
  • Traefik archive contains traefik.yml, conf.d, and acme.json.
  • Empty websites is recorded as a known warning, not ignored.

Podman Restore Validation

After 04-restore-rootless-services.sh and 05-validate-rootless-services.sh:

  • CT 101 podman-lxc is running at 192.168.2.100/24.
  • CT 101 remains unprivileged.
  • Rootless Podman commands work as podsvc.
  • /etc/subuid and /etc/subgid contain podsvc:10000:50000.
  • The rootless network kh3-backend exists.
  • PostgreSQL container is running and healthy.
  • PostgreSQL pg_isready succeeds.
  • Databases forgejo and vaultwarden exist.
  • Forgejo container is running and direct HTTP on port 30080 answers.
  • Vaultwarden container is running and direct HTTP on port 30081 answers.
  • Adminer container is running and direct HTTP on port 30082 answers.
  • Dozzle container is running and direct HTTP on port 30083 answers if it is in scope for the restore.
  • Forgejo env points to database host postgres:5432, database forgejo, and role forgejo.
  • If recovered Forgejo app.ini still names an old database host such as db2:5432, the env override is documented and validation still passes.
  • Vaultwarden SMTP is disabled when the recovered env lacks SMTP_FROM.
  • No user systemd unit is failed.
  • Forgejo Actions runner is either running and validated, or its failed / intentionally-disabled state is documented.

Docker Restore Validation

After 03-restore-docker-services.sh:

  • Docker network aproxy exists with subnet 172.18.0.0/16.
  • Docker network backend exists with subnet 172.19.0.0/16.
  • PostgreSQL pg_isready succeeds.
  • gitea database exists.
  • vaultwarden database exists.
  • Forgejo container is running.
  • Vaultwarden container is running.
  • Adminer container is running.
  • Runner is running, or the missing registration/config action is documented.
  • Traefik route checks are run if Traefik is restored or already available.
  • No required container is restarting or unhealthy.

Application Checks

Forgejo:

  • Web UI loads.
  • Existing admin login works.
  • Repositories, LFS objects, attachments, and SSH keys are present.
  • Git over SSH on port 2222 works.
  • Runner appears online and one known workflow runs, or manual registration is documented.

Vaultwarden:

  • Existing user login works.
  • Vault unlock works.
  • Attachments and sends are present.
  • SMTP and SSO/OIDC settings are validated if configured.
  • Admin token came from the approved source, not a generated replacement.

Adminer:

  • Adminer route or direct access loads only from the approved network path.
  • It can connect to PostgreSQL with a least-privilege test user.
  • Exposure is protected by firewall, VPN, or Traefik middleware.

Idempotency Validation

Run the active restore stages a second time with FORCE_BACKUP_COPY=0 and FORCE_RESTORE=0. For the current Podman path, use the Podman scripts. For a historical Docker exercise, use the Docker scripts.

  • Backup is not recopied from NAS.
  • LXC staged backup is not recopied.
  • Edited .env files are not overwritten.
  • PostgreSQL dumps are not re-imported.
  • Container networks are not duplicated.
  • Correct existing networks are accepted.
  • Wrong-subnet existing networks fail safely.
  • Service definitions converge without changing edited env files.
  • Existing correct CT 101 podman-lxc is accepted for the Podman path.
  • Existing correct CT 101 docker-host is accepted only for a historical Docker restore exercise.
  • Existing wrong CT ID or hostname fails before restore.

Completion Gate

The restore is not complete until:

  • This checklist is fully filled.
  • The validation report from 02-validate-artifacts.sh is retained.
  • The service validation output is retained.
  • Any manual reverse-proxy, runner, Traefik, DNS, or pfSense action is documented.
  • Forgejo and Vaultwarden login tests pass from an approved network path.
  • A fresh backup from the restored environment is created and verified.