Skip to content

Operate Caddy ingress

Purpose

Diagnose DNS, TLS, Caddy routing, and backend failures in the order traffic uses them.

Prerequisites and risk

Use ssh pvessh; CT 103 is the ingress guest. Read-only checks are safe. Caddyfile edits and restarts affect every published service and require an approved window plus a backup.

Ordered diagnosis

  1. Check name resolution:
ssh pvessh 'pct exec 102 -- dig +short @192.168.2.2 HOSTNAME A'

Expected for an internal Caddy route: 192.168.2.3.

  1. Check Caddy and listeners:
ssh pvessh 'pct exec 103 -- systemctl is-active caddy'
ssh pvessh 'pct exec 103 -- ss -H -lntp | grep -E ":(80|443)"'
  1. Find the route without printing environment files:
ssh pvessh 'pct exec 103 -- grep -nE "HOSTNAME|UPSTREAM" \
  /opt/caddy/config/Caddyfile'
  1. Test the upstream from CT 103:
ssh pvessh 'pct exec 103 -- curl -sS -o /dev/null \
  -w "%{http_code}\n" http://UPSTREAM/'
  1. Test Caddy locally with SNI:
ssh pvessh 'pct exec 103 -- curl -sS -o /dev/null \
  -w "%{http_code} verify=%{ssl_verify_result}\n" \
  --resolve HOSTNAME:443:127.0.0.1 https://HOSTNAME/'
  1. Test the client path and inspect only recent Caddy logs:
curl -Iv https://HOSTNAME/
ssh pvessh 'pct exec 103 -- journalctl -u caddy --since "10 minutes ago" --no-pager'

Symptom matrix

Symptom Likely layer Diagnosis Remedy
DNS wrong/NXDOMAIN Technitium or public DNS Query authoritative resolver and Technitium Correct the authoritative record in an approved change
404 Host matcher or static file Check Caddy route and index.html Correct matcher/path; preserve URL behavior
403 Deliberate policy or permissions Check route response and backend file modes Keep deliberate blocks; correct least-privilege read permissions
502 Backend down/wrong port Curl upstream from CT 103 Repair backend or route; do not weaken TLS globally
TLS name/chain error Certificate issuance or SNI Test with --resolve without -k; inspect issuance log Correct DNS-01 token scope/resolver or hostname
Timeout Routing/firewall/listener Check IP, route, listener, and OPNsense policy Change only the failed layer with authorization
Old content with 200 Build artifact Compare file and HTTP timestamps Use the publishing runbook; do not restart Caddy

Change, validation, and rollback

Before an approved edit, copy the Caddyfile to a dated restricted backup. Load the existing env only inside the shell and do not print it:

ssh pvessh 'pct exec 103 -- sh -c \
  "set -a; . /opt/caddy/env/caddy.env; set +a; \
  /usr/local/bin/caddy validate --config /opt/caddy/config/Caddyfile"'

Because the admin API is disabled, the deployed service uses a restart rather than reload. After authorization, restart only after validation, repeat DNS, backend, SNI, and public checks, and restore the dated Caddyfile on failure.

Official references