Skip to content

Back up and recover infrastructure documentation

Purpose

Protect the Git source, the generated publish artifact, and the configuration needed to reconstruct the publication path.

Current recovery status

The repository's scripts/backup/ suite targets the June 2026 Docker, Pi-hole, pfSense, CT 100, CT 107, and network-backup-syn architecture. That storage is absent from pvesm status, and the scripts do not capture the current CT 101 Quadlets, Technitium, CT 103 Caddy, or all runner-data. Do not run it as the current production backup without a separately reviewed update.

Prerequisites and risk

  • A restricted destination outside the repository and outside site/.
  • Sufficient space.
  • Read access to Git and, for artifact backup, ssh pvessh.

Copying source and generated artifacts is low risk. Restoring over a live publish directory changes production; use an approved window.

Back up documentation source

  1. Confirm the working tree and branch:
git status --short --branch
git log -1 --oneline --decorate
  1. Protect committed history with the approved Forgejo repository backup process.
  2. For an additional source-only archive:
python3 scripts/create_minimal_docs_backup.py \
  /secure/off-repo/it-docs-minimal-YYYYMMDD.tar.gz
tar -tzf /secure/off-repo/it-docs-minimal-YYYYMMDD.tar.gz | sed -n '1,20p'
  1. Record SHA-256 and store it separately:
sha256sum /secure/off-repo/it-docs-minimal-YYYYMMDD.tar.gz

Expected: archive contains Markdown, configuration, workflows, and scripts, but not Git history, site/, caches, secrets, or filled answer files.

Back up the publishing artifact

The artifact is rebuildable, but a snapshot accelerates recovery and preserves evidence:

ssh pvessh 'pct exec 101 -- tar -C \
  /opt/podman/volumes/forgejo/runner-data/publish/kh3-it-infra-docs \
  -czf - site' > /secure/off-repo/kh3-docs-site-YYYYMMDD.tar.gz
tar -tzf /secure/off-repo/kh3-docs-site-YYYYMMDD.tar.gz | sed -n '1,20p'

Also back up, through the approved host backup process:

  • docs-static.container;
  • Forgejo runner config.yml without exposing .runner;
  • CT 103 Caddyfile and service unit;
  • secret files through restricted backup, never this repository.

Restore source

  1. Restore or clone the Forgejo repository.
  2. Verify the intended commit and signature/review history.
  3. Install pinned dependencies.
  4. Run QA and strict build.
  5. Publish through Forgejo Actions. Do not manually treat a local site/ as production source.

Restore the generated artifact temporarily

Prefer rebuilding from known-good source. If CI is unavailable and an approved emergency requires the snapshot:

  1. Confirm docs-static is stopped or coordinate an atomic directory swap.
  2. Preserve the current publish directory as dated incident evidence.
  3. Extract into a new sibling directory, validate index.html, ownership, and modes.
  4. Swap directories atomically.
  5. Start/check docs-static, backend HTTP, ingress SNI, and public HTTPS.
  6. Rebuild through CI when available and remove temporary evidence only after approval.

Validation

python3 scripts/docs_qa.py
mkdocs build --strict
ssh pvessh 'curl -sS -o /dev/null -w "%{http_code}\n" http://192.168.2.20:30084/'
curl -sS -o /dev/null -w '%{http_code}\n' https://docs.kh3group.com/

Expected: both HTTP checks return 200 and the artifact timestamp matches the intended build.

Rollback and troubleshooting

Keep the previous publish directory until validation passes. Restore it atomically if the new snapshot gives 403, 404, or missing assets. A 502 usually means the static backend is unavailable, not that the snapshot is bad.