Skip to content

Validate the current infrastructure

Purpose

Refresh the canonical inventory without changing services. Run this after an incident, before documentation updates, and before planning a live change.

Prerequisites and impact

  • Repository access.
  • ssh pvessh for Proxmox and ssh ovps-me for the public VPS.
  • Optional authenticated ssh opn or OPNsense UI/API access.

All steps are read-only. Avoid raw environment files, firewall exports, database rows, private keys, and unrestricted configuration dumps.

Procedure

  1. Record UTC time and repository state:
date -u +%Y-%m-%dT%H:%M:%SZ
git status --short --branch
  1. Capture Proxmox version, guests, and storage:
ssh pvessh 'pveversion; qm list; pct list; pvesm status'

Expected: VM 100 and CTs 101–106 are running in the 2026-07-17 baseline. Investigate deviations; do not start or stop a guest merely to match documentation.

  1. Capture guest definitions:
ssh pvessh 'qm config 100; qm config 108'
ssh pvessh 'for id in 101 102 103 104 105 106 107; do
  printf "\nCT %s\n" "$id"
  pct config "$id"
done'
  1. Validate CT 101 without exposing secrets:
ssh pvessh 'pct exec 101 -- runuser -l podsvc -c \
  "podman ps --format \"table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}\""'
ssh pvessh 'pct exec 101 -- runuser -l podsvc -c \
  "XDG_RUNTIME_DIR=/run/user/2000 systemctl --user --failed --no-pager"'

Expected: no failed user units; PostgreSQL reports healthy.

  1. Validate DNS and ingress:
ssh pvessh 'pct exec 102 -- systemctl is-active dns'
ssh pvessh 'pct exec 102 -- dig +short @192.168.2.2 git.kh3group.com A'
ssh pvessh 'pct exec 103 -- systemctl is-active caddy'
ssh pvessh 'pct exec 103 -- curl -sS -o /dev/null -w "%{http_code}\n" \
  http://192.168.2.20:30084/'

Expected: active services, git.kh3group.com returns 192.168.2.3, and the documentation backend returns 200.

  1. Validate Headscale:
ssh ovps-me 'sudo -iu podsvc podman exec headscale headscale version'
ssh ovps-me 'sudo -iu podsvc podman exec headscale headscale nodes list'
ssh ovps-me 'sudo -iu podsvc podman exec headscale headscale nodes list-routes'
  1. Validate OPNsense only through an approved path:
ssh opn 'hostname; ifconfig -a'

If authentication fails, record Unverified, the error, and the next safe UI/API or corrected-key check. Do not use disk edits as a substitute.

  1. Update the inventory, network, service catalog, and evidence register in one change.

Validation and rollback

This procedure makes no live change, so rollback is not applicable. Validate the documentation with:

python3 scripts/docs_qa.py
mkdocs build --strict
git diff --check

Troubleshooting

Symptom Likely cause Remedy
socket: Operation not permitted Sandbox blocks a direct path Use ssh pvessh or an approved read-only path
OPNsense rejects keys Multiple/incorrect identities or stale authorization Record unverified; correct the approved SSH/UI access separately
QEMU guest command fails Guest agent absent or stopped Use qm config, console observation, and reachability evidence
Container command reads /root Wrong user environment Use runuser -l podsvc and the user runtime directory

Official references