Skip to content

Pre-Reinstall Migration Checklist

Purpose

Use this run sheet before shutting down pve02 for a clean Proxmox installation. Do not begin the reinstall until every required checkbox is complete and the final go/no-go gate passes.

This procedure preserves rebuild inputs and business data, not guest operating system disks. The one exception is the protected recovery image of damaged LXC 105.

Execution Model

The backup storage is mounted on the Proxmox host, not inside Docker LXC 100. Run every shell block in this document from a root Bash session on pve02 unless the block is explicitly marked administration workstation or the instruction names a web UI.

  • Proxmox host: pve02
  • Docker LXC: CT 100, accessed from Proxmox with pct exec 100 -- ...
  • DNS LXC: CT 107, accessed from Proxmox with pct exec 107 -- ...
  • Backup mount on Proxmox: /mnt/pve/network-backup-syn

Before running any block, inspect the shell prompt:

root@pve02:~#    correct: Proxmox host
root@proxy:~#    wrong: Docker LXC 100

From root@proxy, run exit, then connect to the Proxmox host. If the administration workstation has the documented SSH alias, use:

ssh pve

At the beginning of each new Proxmox session, run this identity check before any redirection to BACKUP_ROOT:

test "$(hostname -s)" = "pve02" || {
  printf '%s\n' 'STOP: this command must run on pve02, not inside an LXC' >&2
  return 1 2>/dev/null || exit 1
}
command -v pct >/dev/null || {
  printf '%s\n' 'STOP: pct is unavailable; this is not the Proxmox host' >&2
  return 1 2>/dev/null || exit 1
}

BACKUP_ROOT is a shell variable. It is not automatically available in a new SSH session, on the administration workstation, or inside an LXC. On every new Proxmox shell, restore it with:

test "$(hostname -s)" = "pve02"
command -v pct >/dev/null
source /root/rebuild-backup.env
printf 'RUN_ID=%s\nBACKUP_ROOT=%s\n' "$RUN_ID" "$BACKUP_ROOT"
test -d "$BACKUP_ROOT"
findmnt -T "$BACKUP_ROOT"

Do not type a BACKUP_ROOT containing $RUN_ID until RUN_ID has been set. Otherwise a command such as the following creates a path ending in proxmox-rebuild-:

export BACKUP_ROOT="/mnt/pve/network-backup-syn/proxmox-rebuild-$RUN_ID"

Rules

  • Record the operator, start date, planned reinstall date, and backup location.
  • Keep two copies of the final encrypted backup on different physical storage.
  • Encrypt any archive containing .env, XML recovery exports, private keys, or tokens.
  • Confirm the NAS share ACL permits only approved administrators during the capture.
  • Do not include logs, query history, caches, Docker images, or generated sites.
  • Do not run repairing fsck against the only copy of LXC 105.
  • Do not shut down Proxmox while any required export is unverified.

1. Prepare Restricted Storage

Open a root Bash session on pve02. Create a new run ID, verify that the path is on the expected CIFS mount, and persist the variables for later SSH sessions:

set -euo pipefail
umask 077

test "$(hostname -s)" = "pve02"
command -v pct >/dev/null

export RUN_ID="$(date +%Y%m%d-%H%M%S)"
export BACKUP_ROOT="/mnt/pve/network-backup-syn/proxmox-rebuild-${RUN_ID}"

test "$(findmnt -n -o TARGET -T /mnt/pve/network-backup-syn)" = \
  "/mnt/pve/network-backup-syn"
test "$(findmnt -n -o FSTYPE -T /mnt/pve/network-backup-syn)" = "cifs"
pvesm status | awk '$1 == "network-backup-syn" && $3 == "active" {ok=1} END {exit !ok}'

install -d -m 0700 \
  "$BACKUP_ROOT" \
  "$BACKUP_ROOT/proxmox" \
  "$BACKUP_ROOT/pfsense" \
  "$BACKUP_ROOT/dns" \
  "$BACKUP_ROOT/docker" \
  "$BACKUP_ROOT/databases/postgresql" \
  "$BACKUP_ROOT/databases/mariadb" \
  "$BACKUP_ROOT/applications" \
  "$BACKUP_ROOT/websites" \
  "$BACKUP_ROOT/checksums"

{
  printf '%s\n' \
    'test "$(hostname -s)" = "pve02" || {' \
    '  printf "%s\n" "STOP: rebuild-backup.env is for pve02 only" >&2' \
    '  return 1 2>/dev/null || exit 1' \
    '}'
  printf 'export RUN_ID=%q\nexport BACKUP_ROOT=%q\n' \
    "$RUN_ID" "$BACKUP_ROOT"
} > /root/rebuild-backup.env
printf '%s\n' "$RUN_ID" > "$BACKUP_ROOT/RUN_ID"
touch "$BACKUP_ROOT/.write-test"
rm "$BACKUP_ROOT/.write-test"
df -h "$BACKUP_ROOT"

The live check on June 14, 2026 showed about 2.1 TiB free on this NAS mount. Recheck immediately before copying the approximately 571 GiB allocated LXC 105 image.

The same check showed that the CIFS client forces file_mode=0755 and dir_mode=0755. Therefore, install -m 0700 documents the intended access but does not enforce it on this mount. Before writing secrets, verify the Synology share and folder ACLs from another client and confirm that only approved administrators can read this run directory. Keep unencrypted captures only for the duration of the migration, create the encrypted archive promptly, and remove the unencrypted tree only after both encrypted copies and restore tests pass.

Preflight required host commands:

for command in bash tar gzip sha256sum findmnt e2fsck pct pvesm; do
  command -v "$command" >/dev/null || {
    printf 'MISSING: %s\n' "$command" >&2
    exit 1
  }
done

command -v ddrescue >/dev/null || printf '%s\n' \
  'MISSING: ddrescue (Debian package: gddrescue; required for preferred CT 105 copy)'
command -v age >/dev/null || printf '%s\n' \
  'MISSING: age (install before the encryption step)'

Install gddrescue and age before the maintenance window if they will be used. Do not assume either command is present; neither was installed on pve02 during the June 14, 2026 audit.

  • BACKUP_ROOT contains a non-empty timestamp and is on network-backup-syn.
  • The write test and storage status checks pass.
  • Synology share/folder ACLs compensate for the CIFS mount's forced 0755 modes.
  • Available space is sufficient for all exports, the LXC 105 image, and the encrypted archive.
  • A second independent physical destination is identified.

2. Capture Proxmox Definitions

Run on pve02:

set -euo pipefail
source /root/rebuild-backup.env

out=/root/rebuild-capture/proxmox
rm -rf "$out"
install -d -m 0700 "$out/qm" "$out/pct"
cp /etc/network/interfaces /etc/hosts /etc/hostname \
  /etc/pve/storage.cfg "$out/"
pveversion -v > "$out/pveversion.txt"
pvesm status > "$out/storage-status.txt"
ip -br link > "$out/ip-link.txt"
ip -br address > "$out/ip-address.txt"
qm list > "$out/qm-list.txt"
pct list > "$out/pct-list.txt"

while read -r id; do
  qm config "$id" > "$out/qm/$id.conf"
done < <(qm list | awk 'NR > 1 {print $1}')

while read -r id; do
  pct config "$id" > "$out/pct/$id.conf"
done < <(pct list | awk 'NR > 1 {print $1}')

tar -C /root/rebuild-capture -czf \
  "$BACKUP_ROOT/proxmox/proxmox-definitions.tar.gz" proxmox
tar -tzf "$BACKUP_ROOT/proxmox/proxmox-definitions.tar.gz" >/dev/null
sha256sum "$BACKUP_ROOT/proxmox/proxmox-definitions.tar.gz"
  • VM 110 shows three NICs on vmbr0, vmbr1, and vmbr2.
  • The recorded startup order is pfSense 1, DNS 2, Docker 3.
  • interfaces, storage.cfg, every qm config, and every pct config are present.
  • Physical NIC names and MAC addresses are recorded for post-install mapping.

3. Export pfSense

In pfSense, open Diagnostics > Backup & Restore:

  • Select the complete configuration area.
  • Include package configuration.
  • Select Skip RRD Data.
  • Leave extra volatile data disabled.
  • Encrypt the export with a password stored in the approved password manager.
  • Download the raw dated XML to the administration workstation.
  • Generate a fresh redacted copy separately for documentation.

From the administration workstation, upload the downloaded file to a temporary path. Replace the local filename as needed:

scp ./pfsense-config-YYYYMMDD.xml pve:/root/pfsense-config-YYYYMMDD.xml

Then on pve02:

source /root/rebuild-backup.env
install -m 0600 /root/pfsense-config-YYYYMMDD.xml \
  "$BACKUP_ROOT/pfsense/pfsense-config-YYYYMMDD.xml"
rm /root/pfsense-config-YYYYMMDD.xml
test -s "$BACKUP_ROOT/pfsense/pfsense-config-YYYYMMDD.xml"
file "$BACKUP_ROOT/pfsense/pfsense-config-YYYYMMDD.xml"

Required restore test:

  • Create a temporary pfSense VM with three test NICs.
  • Restore the encrypted XML.
  • Confirm interfaces, VLAN 2, aliases, NAT, DHCP, WireGuard, packages, and gateway-group configuration appear.
  • Record the interface remapping required for the fresh VM.

STOP: A redacted XML is not a recovery backup.

4. Export Pi-hole and Cloudflared

Create a Teleporter export from the Pi-hole web UI and download it to the administration workstation. Upload it to /root on pve02 with scp, then move it into $BACKUP_ROOT/dns/ using the same two-stage process as the pfSense export. Do not use a workstation-side $BACKUP_ROOT; that path exists only on Proxmox.

Capture additional migration references without query history, on pve02:

set -euo pipefail
test "$(hostname -s)" = "pve02"
command -v pct >/dev/null
source /root/rebuild-backup.env
test -d "$BACKUP_ROOT/dns"

output="$BACKUP_ROOT/dns/dns-reference.txt"
output_tmp="${output}.tmp"
rm -f "$output_tmp"
trap 'rm -f "$output_tmp"' EXIT

pct exec 107 -- sh -s > "$output_tmp" <<'EOF'
set -eu
/usr/bin/cat /etc/os-release
/usr/local/bin/pihole -v
/usr/local/bin/cloudflared --version
/usr/bin/dpkg-query -W
printf '\n--- setupVars ---\n'
/usr/bin/sed -E 's/(WEBPASSWORD|API_KEY|TOKEN)=.*/\1=REDACTED/' \
  /etc/pihole/setupVars.conf 2>/dev/null || true
printf '\n--- custom DNS ---\n'
/usr/bin/cat /etc/pihole/custom.list 2>/dev/null || true
printf '\n--- enabled services ---\n'
for service in pihole-FTL cloudflared; do
  printf '%s: ' "$service"
  /usr/bin/systemctl is-enabled "$service" 2>/dev/null ||
    printf 'not enabled or not installed\n'
done
EOF

test -s "$output_tmp"
mv "$output_tmp" "$output"
trap - EXIT
test -s "$output"

pct exec ... sh -s supplies a minimal PATH that omits /usr/local/bin on this container, so the Pi-hole and Cloudflared commands deliberately use their absolute paths. The temporary output prevents a failed capture from being mistaken for a complete reference file.

  • Teleporter archive exists, is non-empty, and opens as an archive.
  • Custom DNS records, groups, clients, and adlists are visible in the export.
  • Cloudflared tunnel ID and account ownership are recorded.
  • The current embedded Cloudflared token is rotated and the replacement is stored only in the secret manager.
  • A temporary Pi-hole v6 installation successfully imports the Teleporter archive.

Do not copy pihole-FTL.db, downloaded gravity list files, or /var/log.

5. Preserve RustDesk Identity

LXC 106 is stopped. Mount it on Proxmox, copy the required files directly to the NAS, and guarantee that this procedure unmounts only a mount it created:

set -euo pipefail
source /root/rebuild-backup.env

out=/root/rebuild-capture/rustdesk
root=/var/lib/lxc/106/rootfs
mounted_here=0
cleanup() {
  if [ "$mounted_here" -eq 1 ]; then
    pct unmount 106
  fi
}
trap cleanup EXIT

rm -rf "$out"
install -d -m 0700 "$out"
if ! mountpoint -q "$root"; then
  pct mount 106
  mounted_here=1
fi

cp -a "$root/opt/rustdesk/db_v2.sqlite3"* "$out/"
cp -a "$root/opt/rustdesk/id_ed25519" \
  "$root/opt/rustdesk/id_ed25519.pub" "$out/"
cp -a "$root/root/.config/rustdesk/RustDesk.toml" "$out/"
cp -a "$root/etc/systemd/system/rustdeskrelay.service" \
  "$root/etc/systemd/system/rustdesksignal.service" "$out/"

tar -C /root/rebuild-capture -czf \
  "$BACKUP_ROOT/applications/rustdesk-migration.tar.gz" rustdesk
tar -tzf "$BACKUP_ROOT/applications/rustdesk-migration.tar.gz" >/dev/null
sha256sum "$BACKUP_ROOT/applications/rustdesk-migration.tar.gz"
  • Private/public identity key pair is present.
  • SQLite database and both systemd units are present.
  • An existing RustDesk client is identified for post-restore testing.

6. Recover LXC 105 Before Any Wipe

LXC 105 has a 580 GiB raw ext4 disk, approximately 571 GiB allocated, and currently fails to mount. Create a protected storage-level copy before repair.

Verify the source and destination on pve02:

set -euo pipefail
source /root/rebuild-backup.env

source_image=/mnt/pve/DIR01/images/105/vm-105-disk-0.raw
image_dir="$BACKUP_ROOT/proxmox/lxc105"
test -f "$source_image"
test "$(pct status 105 | awk '{print $2}')" = "stopped"
install -d -m 0700 "$image_dir"
ls -lh "$source_image"
du -h "$source_image"
df -h "$BACKUP_ROOT"

Preferred copy with resumable error handling:

set -euo pipefail
source /root/rebuild-backup.env
command -v ddrescue >/dev/null

source_image=/mnt/pve/DIR01/images/105/vm-105-disk-0.raw
image_dir="$BACKUP_ROOT/proxmox/lxc105"
ddrescue --force --sparse \
  "$source_image" "$image_dir/ct105.raw" "$image_dir/ct105.map"

If ddrescue cannot be installed, use this non-resumable fallback and record that no ddrescue map exists:

set -euo pipefail
source /root/rebuild-backup.env

source_image=/mnt/pve/DIR01/images/105/vm-105-disk-0.raw
image_dir="$BACKUP_ROOT/proxmox/lxc105"
dd if="$source_image" of="$image_dir/ct105.raw" \
  bs=64M conv=sparse,fsync status=progress
printf '%s\n' 'Copied with dd; no ddrescue map is available.' \
  > "$image_dir/COPY-METHOD.txt"

Do not run both copy methods against the same output. After the copy completes:

set -euo pipefail
source /root/rebuild-backup.env
image_dir="$BACKUP_ROOT/proxmox/lxc105"

test -s "$image_dir/ct105.raw"
stat "$image_dir/ct105.raw"
du -h "$image_dir/ct105.raw"
sha256sum "$image_dir/ct105.raw" | tee "$image_dir/ct105.raw.sha256"

set +e
e2fsck -fn "$image_dir/ct105.raw" 2>&1 |
  tee "$image_dir/e2fsck-read-only.txt"
e2fsck_status=${PIPESTATUS[0]}
set -e
printf 'e2fsck exit status: %s\n' "$e2fsck_status" |
  tee -a "$image_dir/e2fsck-read-only.txt"
case "$e2fsck_status" in
  0|1|2|4) ;;
  *) exit "$e2fsck_status" ;;
esac

e2fsck -f -n may return status 4 when it finds uncorrected filesystem errors. The block records that expected diagnostic status but still fails for operational, usage, cancellation, or library errors. Review the saved output.

  • ct105.raw exists under this run's BACKUP_ROOT, outside DIR01.
  • The copy method, apparent size, allocated size, and checksum are recorded.
  • The image is excluded from automatic backup pruning.
  • Filesystem repair and data inventory are performed on another copy, not the source.
  • An owner decides which recovered data must be retained.

STOP: Do not reinstall Proxmox while this is the only copy of unidentified data.

7. Capture Docker Deployment Definitions

Run on pve02. The capture is created inside CT 100, then streamed into the Proxmox-mounted NAS:

set -euo pipefail
source /root/rebuild-backup.env

pct exec 100 -- bash -s <<'EOF'
set -euo pipefail
umask 077
out=/root/rebuild-capture/docker
rm -rf "$out"
install -d -m 0700 "$out/projects" "$out/runtime"
docker version > "$out/docker-version.txt"
docker compose version > "$out/compose-version.txt"
docker compose ls -a > "$out/compose-projects.txt"
docker network inspect aproxy backend > "$out/runtime/networks.json"
docker ps -a --no-trunc > "$out/runtime/containers.txt"

while read -r container; do
  docker inspect "$container" > "$out/runtime/$container.json"
done < <(docker ps -a --format '{{.Names}}')

for directory in /root/*; do
  test -d "$directory" || continue
  name=${directory##*/}
  files=()
  for file in docker-compose.yml docker-compose.yaml compose.yml compose.yaml \
              .env Dockerfile; do
    test -f "$directory/$file" && files+=("$directory/$file")
  done
  if [ "${#files[@]}" -gt 0 ]; then
    install -d -m 0700 "$out/projects/$name"
    cp -a "${files[@]}" "$out/projects/$name/"
  fi
done
EOF

pct exec 100 -- tar -C /root/rebuild-capture -czf - docker \
  > "$BACKUP_ROOT/docker/docker-definitions.tar.gz"
tar -tzf "$BACKUP_ROOT/docker/docker-definitions.tar.gz" >/dev/null

The runtime JSON and .env files contain secrets. Keep them encrypted and restricted.

  • All active Compose projects are listed.
  • aproxy is recorded as 172.18.0.0/16.
  • backend is recorded as 172.19.0.0/16.
  • Image names and current tags are recorded.
  • Floating tags have planned, tested replacements.

8. Reconstruct Missing Compose Files

The following running projects had no Compose file on disk on June 14, 2026:

  • Appsmith
  • MeshCentral
  • Dashboard
  • Noticeboard
  • kh3-dev-site

For each project:

  1. Use its restricted docker inspect JSON to reconstruct image, environment, mounts, ports, networks, labels, and restart policy.
  2. Replace embedded secrets with .env references.
  3. Write a new Compose file under the matching /root/<project>/ in CT 100.
  4. Validate inside CT 100 without replacing the running container:
pct exec 100 -- docker compose \
  -f /root/<project>/docker-compose.yml config --quiet
pct exec 100 -- docker compose \
  -f /root/<project>/docker-compose.yml config --images
  1. Test the definition on an isolated temporary Docker host or with alternate project, network, and port names.

Appsmith's retained state is the bind mount /root/appsmith/data/stacks, even though /root/appsmith appeared empty during the initial file audit. Do not infer that a project has no state from an empty top-level directory; confirm its mounts with:

pct exec 100 -- docker inspect \
  -f '{{range .Mounts}}{{printf "%s\t%s\t%s\n" .Type .Source .Destination}}{{end}}' \
  <container>

STOP: Do not wipe while a retained container exists only as runtime metadata.

9. Export Databases

Run on pve02. Redirection happens in the Proxmox shell, so dumps are written directly to the NAS rather than inside the Docker LXC.

PostgreSQL:

set -euo pipefail
source /root/rebuild-backup.env

pct exec 100 -- docker exec postgresql \
  pg_dumpall -U postgres --globals-only \
  > "$BACKUP_ROOT/databases/postgresql/globals.sql"

pct exec 100 -- docker exec postgresql \
  psql -U postgres -Atc \
  'select datname from pg_database where datistemplate=false order by 1' \
  > "$BACKUP_ROOT/databases/postgresql/database-list.txt"

while IFS= read -r database; do
  test -n "$database"
  pct exec 100 -- docker exec postgresql \
    pg_dump -U postgres -Fc "$database" \
    > "$BACKUP_ROOT/databases/postgresql/$database.dump"
done < "$BACKUP_ROOT/databases/postgresql/database-list.txt"

MariaDB:

set -euo pipefail
source /root/rebuild-backup.env

pct exec 100 -- docker exec mariadb sh -lc \
  'mariadb -uroot -p"$MARIADB_ROOT_PASSWORD" -NBe "show databases"' \
  > "$BACKUP_ROOT/databases/mariadb/database-list.txt"

while IFS= read -r database; do
  case "$database" in
    information_schema|mysql|performance_schema|sys|'') continue ;;
  esac
  pct exec 100 -- docker exec -e TARGET_DB="$database" mariadb sh -lc \
    'mariadb-dump -uroot -p"$MARIADB_ROOT_PASSWORD" \
      --single-transaction --routines --events --triggers \
      --databases "$TARGET_DB"' |
    gzip > "$BACKUP_ROOT/databases/mariadb/$database.sql.gz"
done < "$BACKUP_ROOT/databases/mariadb/database-list.txt"

MongoDB:

set -euo pipefail
source /root/rebuild-backup.env

pct exec 100 -- docker exec mongodb mongodump --archive --gzip \
  > "$BACKUP_ROOT/databases/mongodb.archive.gz"

Verification uses database tools already present in the database containers:

set -euo pipefail
source /root/rebuild-backup.env

if find "$BACKUP_ROOT/databases" -type f -size 0 -print -quit | grep -q .; then
  printf '%s\n' 'ERROR: zero-length database export found' >&2
  find "$BACKUP_ROOT/databases" -type f -size 0 -print >&2
  exit 1
fi

find "$BACKUP_ROOT/databases" -type f -name '*.gz' -exec gzip -t {} +

while IFS= read -r dump; do
  pct exec 100 -- docker exec -i postgresql pg_restore --list \
    < "$dump" >/dev/null
done < <(find "$BACKUP_ROOT/databases/postgresql" -type f -name '*.dump' | sort)

pct exec 100 -- docker exec -i mongodb mongorestore \
  --archive --gzip --dryRun \
  < "$BACKUP_ROOT/databases/mongodb.archive.gz" >/dev/null

If the installed mongorestore does not support --dryRun, list the archive with the matching MongoDB Database Tools version or restore into an isolated temporary MongoDB instance. Do not restore into the production server.

  • Every retained PostgreSQL database has a non-empty dump.
  • MariaDB dumps exist for every retained application database.
  • MongoDB archive contains retained databases, including meshcentral.
  • At least one dump from each engine has been restored on a temporary instance.
  • Retired or unowned database dumps remain retained until owner review.

10. Export Application State

Run these blocks on pve02. Each tar process runs inside CT 100, while the shell redirection writes directly to BACKUP_ROOT on Proxmox. Required missing paths cause failure and must be investigated.

Platform and application archives:

set -euo pipefail
source /root/rebuild-backup.env

pct exec 100 -- tar -C /root -czf - \
  --exclude='traefik/data/logs' \
  traefik/docker-compose.yml traefik/.env traefik/data/traefik.yml \
  traefik/data/conf.d traefik/data/certs \
  > "$BACKUP_ROOT/applications/traefik.tar.gz"

pct exec 100 -- tar -C /root -czf - \
  vaultwarden/docker-compose.yml vaultwarden/.env vaultwarden/data \
  > "$BACKUP_ROOT/applications/vaultwarden.tar.gz"

pct exec 100 -- tar -C /root -czf - \
  homebox/docker-compose.yml homebox/.env homebox/data \
  > "$BACKUP_ROOT/applications/homebox.tar.gz"

pct exec 100 -- tar -C /root -czf - \
  --exclude='appsmith/data/stacks/logs' \
  appsmith/docker-compose.yml appsmith/.env appsmith/data/stacks \
  > "$BACKUP_ROOT/applications/appsmith.tar.gz"

pct exec 100 -- tar -C /root -czf - \
  --exclude='sterling-pdf/data/logs' \
  sterling-pdf/docker-compose.yml sterling-pdf/.env \
  sterling-pdf/data/config sterling-pdf/data/dat \
  > "$BACKUP_ROOT/applications/stirling-pdf.tar.gz"

pct exec 100 -- tar -C /root -czf - \
  portainer/docker-compose.yml portainer/data \
  > "$BACKUP_ROOT/applications/portainer.tar.gz"

pct exec 100 -- tar -C /root -czf - receiptapp \
  > "$BACKUP_ROOT/applications/receiptapp.tar.gz"

Forgejo:

set -euo pipefail
source /root/rebuild-backup.env

pct exec 100 -- tar -C /root -czf - \
  --exclude='gitea/data/gitea/log' \
  --exclude='gitea/data/gitea/actions_log' \
  --exclude='gitea/runner-data/.cache' \
  gitea/docker-compose.yml gitea/.env gitea/data \
  gitea/runner-data/config.yml gitea/runner-data/.runner \
  > "$BACKUP_ROOT/applications/forgejo.tar.gz"
  • PostgreSQL gitea dump exists.
  • Repositories, LFS objects, attachments, SSH host keys, and custom configuration are present.
  • Runner configuration and registration state are retained.
  • Service and Actions logs are absent.

Drone requires a consistent SQLite copy. This block restarts Drone even if the archive command fails:

set -euo pipefail
source /root/rebuild-backup.env

pct exec 100 -- docker stop drone >/dev/null
restart_drone() {
  pct exec 100 -- docker start drone >/dev/null
}
trap restart_drone EXIT
pct exec 100 -- tar -C /root -czf - \
  drone/docker-compose.yml drone/.env drone/data/database.sqlite \
  > "$BACKUP_ROOT/applications/drone.tar.gz"
restart_drone
trap - EXIT

MeshCentral:

set -euo pipefail
source /root/rebuild-backup.env

latest="$(
  pct exec 100 -- docker exec meshcentral sh -lc \
    'ls -1t /opt/meshcentral/meshcentral-backups/*.zip | head -1'
)"
test -n "$latest"
pct exec 100 -- docker exec meshcentral cat "$latest" \
  > "$BACKUP_ROOT/applications/meshcentral-autobackup.zip"

pct exec 100 -- tar -C /root -czf - \
  meshcentral/docker-compose.yml \
  meshcentral/data/config.json \
  meshcentral/data/user_files \
  meshcentral/data/agentserver-cert-private.key \
  meshcentral/data/agentserver-cert-public.crt \
  meshcentral/data/codesign-cert-private.key \
  meshcentral/data/codesign-cert-public.crt \
  meshcentral/data/mpsserver-cert-private.key \
  meshcentral/data/mpsserver-cert-public.crt \
  meshcentral/data/root-cert-private.key \
  meshcentral/data/root-cert-public.crt \
  meshcentral/data/webserver-cert-private.key \
  meshcentral/data/webserver-cert-public.crt \
  > "$BACKUP_ROOT/applications/meshcentral.tar.gz"
  • Compose has been reconstructed and included.
  • data/config.json, certificate/private-key files, and data/user_files are retained.
  • One recent auto-backup is retained and tested.
  • MongoDB meshcentral data is present in the logical export.

Verify every generated archive:

source /root/rebuild-backup.env
find "$BACKUP_ROOT/applications" -type f -name '*.tar.gz' -print0 |
  while IFS= read -r -d '' archive; do
    printf 'Checking %s\n' "$archive"
    tar -tzf "$archive" >/dev/null
  done

11. Preserve Websites and Source Code

  • website: archive the complete content tree because its Git object store is damaged.
  • kh3website: commit and push intended changes, or archive the complete content tree.
  • khy: commit and push intended changes, or archive the complete content tree.
  • khywebsite: commit and push intended changes, or archive the complete content tree.
  • kh3-dev-site: archive the complete tree.
  • receiptapp: preserve Dockerfile and local source.
  • dashboard: preserve reconstructed Compose; current content is empty.
  • noticeboard: preserve reconstructed Compose and confirm its source repository.
  • Documentation Nginx output is regenerated and does not need backup.

Fallback archive commands, run on pve02:

set -euo pipefail
source /root/rebuild-backup.env

for site in website kh3website khy khywebsite; do
  pct exec 100 -- tar -C /root -czf - \
    --exclude="$site/data/html/.git" "$site" \
    > "$BACKUP_ROOT/websites/$site.tar.gz"
done

pct exec 100 -- tar -C /root -czf - kh3-dev-site \
  > "$BACKUP_ROOT/websites/kh3-dev-site.tar.gz"

for project in dashboard noticeboard; do
  pct exec 100 -- tar -C /root -czf - \
    "$project/docker-compose.yml" \
    > "$BACKUP_ROOT/websites/$project-compose.tar.gz"
done

find "$BACKUP_ROOT/websites" -type f -name '*.tar.gz' -print0 |
  while IFS= read -r -d '' archive; do
    tar -tzf "$archive" >/dev/null
  done

12. Assemble, Encrypt, and Verify

First verify the unencrypted backup tree and create checksums relative to BACKUP_ROOT, so they remain usable after the directory is moved:

set -euo pipefail
source /root/rebuild-backup.env

(
  cd "$BACKUP_ROOT"
  find . -type f ! -path './checksums/SHA256SUMS' -print0 |
    sort -z |
    xargs -0 sha256sum > checksums/SHA256SUMS
  sha256sum -c checksums/SHA256SUMS
)

Create the encrypted archive on Proxmox. age -p obtains its passphrase from the terminal; store that passphrase outside Proxmox:

set -euo pipefail
source /root/rebuild-backup.env
command -v age >/dev/null

archive="${BACKUP_ROOT}.tar.gz.age"
archive_tmp="${archive}.tmp"
rm -f "$archive_tmp"
trap 'rm -f "$archive_tmp"' EXIT
tar -C "$(dirname "$BACKUP_ROOT")" -czf - "$(basename "$BACKUP_ROOT")" |
  age -p -o "$archive_tmp"
mv "$archive_tmp" "$archive"
trap - EXIT
test -s "$archive"
sha256sum "$archive" | tee "${archive}.sha256"

Copy the encrypted archive and its checksum to a second physical destination. Replace the placeholder with a mounted path that is not the same NAS or Proxmox local disk:

set -euo pipefail
source /root/rebuild-backup.env

second_root=/replace/with/second/physical/storage
test -d "$second_root"
test "$(findmnt -n -o SOURCE -T "$second_root")" != \
  "$(findmnt -n -o SOURCE -T "$BACKUP_ROOT")"

cp --preserve=timestamps "$BACKUP_ROOT.tar.gz.age" "$second_root/"
cp --preserve=timestamps "$BACKUP_ROOT.tar.gz.age.sha256" "$second_root/"

test "$(
  sha256sum "$BACKUP_ROOT.tar.gz.age" | awk '{print $1}'
)" = "$(
  sha256sum "$second_root/$(basename "$BACKUP_ROOT.tar.gz.age")" |
    awk '{print $1}'
)"

Test decryption and archive listing without writing an unencrypted archive:

set -euo pipefail
source /root/rebuild-backup.env

age -d "$BACKUP_ROOT.tar.gz.age" |
  tar -tzf - > /root/proxmox-rebuild-archive-list.txt
test -s /root/proxmox-rebuild-archive-list.txt
  • Checksums pass before encryption.
  • Two encrypted copies on different physical storage have matching hashes.
  • The encrypted archive decrypts and lists successfully.
  • At least one database, pfSense, Pi-hole, and one application restore have been tested.
  • The decryption password or key is available independently of Proxmox.

13. Final Go/No-Go Gate

All answers must be YES:

Gate YES/NO
Two verified encrypted backup copies exist outside Proxmox
pfSense full XML restored successfully in a temporary VM
Pi-hole Teleporter imported successfully into Pi-hole v6
Cloudflared replacement token is in the secret manager
Proxmox network/storage/guest definitions are captured
LXC 105 has a protected copy and recovery decision
RustDesk identity and database are preserved or retirement is approved
All retained Docker projects have tested Compose definitions
PostgreSQL, MariaDB, and MongoDB logical exports pass restore tests
Forgejo repositories/LFS/attachments and gitea database are preserved
Modified/corrupt website trees are pushed or archived
Owners approved retirement of VMs 102, 113, template 5000, and LXC 101 where applicable
Backup decryption material is accessible while Proxmox is offline
Rebuild installation media and a separate admin workstation are ready

If any answer is NO, postpone the reinstall.

14. Final Shutdown Capture

Freeze changes and stop user-facing writes. Repeat all three database export blocks plus every application or website archive whose source can change. Then regenerate checksums, recreate both encrypted copies, and repeat the decryption test.

Record final state from pve02:

pct exec 100 -- docker ps \
  --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'
date -Is
qm list
pct list
pvesm status

After final exports have completed, stop Docker workloads and shut down guests from the Proxmox host:

set -u

pct exec 100 -- bash -lc \
  'ids=$(docker ps -q); test -z "$ids" || docker stop $ids'
pct shutdown 100 --timeout 120 || pct stop 100
pct shutdown 107 --timeout 60 || pct stop 107
qm shutdown 110 --timeout 120 || qm stop 110
shutdown -h now

Use forced stops only after final exports and graceful shutdown attempts have failed. Run the commands from a local Proxmox console if shutting down pfSense will disconnect the administration workstation.