Skip to content

Network Production Hardening Plan

Dated planning and implementation record

This 2026-07-02 record contains useful findings but is not the canonical current network reference. Validate open decisions against Current network reference before implementation.

Updated: 2026-07-02

This plan reviews the current Proxmox, OPNsense, DMZ, DNS, Headscale, Caddy, and Podman network stack and defines a production-grade direction. It is a planning document only. Do not make live OPNsense, Proxmox, firewall, DNS, or service changes from this page without a fresh export, a maintenance window, and a rollback path.

Executive Findings

The current design has one critical circular dependency: Proxmox pve runs the OPNsense VM, but pve management currently lives on the routed DMZ path with default gateway 192.168.2.1, which is OPNsense. If OPNsense is stopped, late to boot, misconfigured, or unable to route, remote access to the host can fail before the operator can diagnose the firewall VM.

The second critical dependency is DNS. Infrastructure hosts, tunnel agents, and control-plane services must not depend on Technitium or internal DNS for their own recovery path. Technitium is the policy resolver for clients and internal names, not the bootstrap resolver for Proxmox, OPNsense, Headscale clients, or Cloudflare tunnels.

The third risk is that the environment is single-host and single-firewall. That is acceptable for a small site only if the design is honest about it: there is no true firewall HA, no true Proxmox HA, and no guaranteed remote break-glass path while all WAN egress depends on a firewall VM running on the same host.

Reference model: OPNsense's own Dnsmasq DNS & DHCP documentation describes Dnsmasq as the small/medium deployment DHCP service and shows DHCP options scoped by interface. The KH3 implementation follows that pattern with OPNsense Dnsmasq handing out DHCP and Technitium acting as the policy DNS resolver for LAN and DMZ clients. Firewall enforcement should then block client DNS bypass, as already done on DMZ.

Live validation on 2026-07-02 through ssh pve-kh3 confirmed a related bootstrap outage: pve could route to public IPs, but OPNsense blocked direct DNS from pve to 8.8.8.8:53 while /etc/resolv.conf used only 8.8.8.8. This broke both the pve Headscale/Tailscale control-plane connection and the cloudflared service on pve.

The persistent firewall-side fix applied later on 2026-07-02 was to add two narrow OPNsense DMZ pass rules for source 192.168.2.10 to Cloudflare bootstrap DNS (1.1.1.1:53 and 1.0.0.1:53) before the DMZ DNS bypass block. The persistent host-side fix is a dedicated localhost-only dnsmasq resolver on pve: /etc/resolv.conf points to 127.0.0.1, local KH3 service names are served from static overrides, and non-local names forward to Cloudflare. Technitium remains the client and DMZ policy resolver, not a Proxmox bootstrap dependency.

Current Verified Shape

Layer Current evidence
Physical NIC constraint Three physical interfaces are available. Two are required for WAN failover; the remaining interface is shared for LAN and DMZ.
Proxmox host pve, management address documented as 192.168.2.10/24 on vmbr0.2, default route via 192.168.2.1.
Firewall OPNsense VM 100 router, DMZ gateway 192.168.2.1/24.
OPNsense export Redacted export shows WAN1 on vtnet1 at 10.0.0.2/24, WAN2 on vtnet2 at 192.168.1.2/24, LAN on vtnet0, and DMZ VLAN 2 on vtnet0.2.
WAN failover Gateway group WANFailOver, WAN1GW tier 1 and WAN2GW tier 2, trigger downlosslatency.
DMZ 192.168.2.0/24, dynamic range 192.168.2.100-192.168.2.200, infrastructure reservations below the pool.
DNS Technitium CT 102 at 192.168.2.2; DMZ clients should receive DHCP option 6 = 192.168.2.2.
Ingress Caddy CT 103 at 192.168.2.3; CT 101 app services use high backend ports.
Tailnet The 2026-07-02 record covered only 192.168.2.0/24. A 2026-07-17 Headscale check showed CT 105 ts-router serving both 192.168.0.0/24 and 192.168.2.0/24; pve is not the primary subnet router.

Live Proxmox check on 2026-07-02:

Item Observed state
Proxmox version pve-manager/9.2.3, kernel 7.0.6-2-pve
Host network vmbr0 on nic0, VLAN-aware, vmbr0.2 static 192.168.2.10/24, default route 192.168.2.1
WAN attachment VM 100 router has hostpci0 0000:02:10.0 and hostpci1 0000:02:10.1, plus net0 on vmbr0
Guest startup OPNsense order=10,up=20; CT 102, 103, and 105 order=2; CT 101 order=3; CT 104 has onboot=1 but no explicit startup order
Host DNS before mitigation 8.8.8.8 only; direct TCP DNS to 8.8.8.8:53 failed from pve
Temporary host DNS mitigation 192.168.2.2 first, 1.1.1.1 second; backup /etc/resolv.conf.pre-pve-bootstrap-dns-20260702T120530Z
Persistent bootstrap DNS fix OPNsense allows 192.168.2.10 to 1.1.1.1:53 and 1.0.0.1:53 before the DMZ DNS bypass block
Host DNS after persistent fix /etc/resolv.conf points to 127.0.0.1; kh3-pve-dnsmasq.service listens on localhost, serves static KH3 bootstrap overrides, and forwards public names to 1.1.1.1 and 1.0.0.1
Host Headscale state after mitigation pve, archlinux, and ts-router visible in tailscale status; pve does not advertise routes
Host cloudflared state after mitigation cloudflared.service active and registered tunnel connections; timeout drop-in added at /etc/systemd/system/cloudflared.service.d/20-timeout.conf
LAN DHCP option 6 after fix OPNsense Dnsmasq advertises 192.168.2.2 on LAN and DMZ; backup /conf/config.xml.pre-lan-dhcp-option6-technitium-20260702T173046Z

Non-Negotiable Design Rules

  1. Proxmox host management must have a local break-glass path that does not require OPNsense, Technitium, Caddy, CT 105, or CT 101.
  2. Proxmox host DNS must use external resolvers or host-local static entries for bootstrap services. It must not use Technitium as its only resolver.
  3. OPNsense system DNS must use external resolvers for gateway monitoring, updates, and package operations. It must not depend on Technitium for firewall bootstrapping.
  4. Technitium should be advertised to clients and application hosts, but not be the only resolver for infrastructure recovery components.
  5. DHCP reservations are useful for ordinary service hosts, but core bootstrap addresses should be static or pinned at the guest and hypervisor layer.
  6. All management surfaces must be reachable by IP before relying on DNS.
  7. Tailnet access is a management convenience, not a substitute for a local host recovery path unless the host has independent egress.

Target Network Model

Physical and VLAN Model

With the current three-interface constraint:

Interface use Target
WAN1 NIC Dedicated to OPNsense WAN1. Prefer PCI passthrough if stable; otherwise a dedicated Proxmox bridge with no host IP.
WAN2 NIC Dedicated to OPNsense WAN2. Prefer PCI passthrough if stable; otherwise a dedicated Proxmox bridge with no host IP.
Shared LAN/DMZ NIC VLAN-aware trunk to the managed switch. Carry a native or tagged management VLAN, client LAN VLAN, and DMZ VLAN.

The production-preferred design adds one independent out-of-band management path. On an OptiPlex-class host without IPMI, that usually means a USB Ethernet adapter, a small dedicated management router, or moving the firewall off the Proxmox host. Without that extra path, remote access cannot be guaranteed when the OPNsense VM is down.

VLAN and Subnet Intent

Segment Purpose Notes
Management Proxmox UI/SSH, switch management, OPNsense admin UI Should be reachable locally without DNS. Ideally not routed only through OPNsense.
LAN User/admin clients DHCP from OPNsense; DNS through Technitium. Current CIDR still needs recapture.
DMZ Published services and infrastructure LXCs Current 192.168.2.0/24; keep servers here until a finer split is implemented.
Optional service VLAN Databases, app backends, storage Future split from DMZ if east-west control becomes necessary.
Guest/IoT Untrusted clients Future policy-controlled segment.

If only one switch-facing NIC exists, use a VLAN trunk. Do not collapse management, LAN, and DMZ policy into one flat segment.

Proxmox Host Baseline

Target state for pve:

  • Static host management IP, not DHCP-dependent.
  • Static /etc/hosts entries for critical local names used by host services.
  • /etc/resolv.conf points to a host-local resolver with external forwarders, not to Technitium or ordered fallback nameservers.
  • Tailscale/Headscale client on pve uses --accept-dns=false.
  • pve should not advertise production subnet routes except as a documented emergency rollback. CT 105 remains the primary subnet router.
  • cloudflared, if kept on pve, must proxy to local IP or loopback targets and use public DNS resolvers. It still cannot provide recovery if pve has no WAN path when OPNsense is down.
  • OPNsense now allows approved bootstrap DNS from pve to 1.1.1.1 and 1.0.0.1; the pve host-local resolver uses static KH3 overrides for internal service names and forwards all other names to Cloudflare.
  • Proxmox guest startup order:
  • OPNsense VM first.
  • Technitium, Caddy, and ts-router next.
  • Application Podman CTs after DNS and routing.

OPNsense Baseline

Target state for OPNsense:

  • WAN1 and WAN2 mappings documented with the physical NIC, Proxmox bridge or passthrough method, upstream device, gateway, and monitor IP.
  • Gateway group failover tested both ways, including state behavior and return to primary.
  • System DNS uses public upstream resolvers and is pinned to the intended WAN gateways where applicable.
  • DHCP backend and scopes recaptured from the live UI/API. The redacted export shows <dhcpbackend>kea</dhcpbackend> while recent operations describe Dnsmasq behavior, so this must be verified before the next DHCP change.
  • DMZ DHCP option 6 advertises 192.168.2.2 and, after a second resolver is built, both internal policy resolvers.
  • LAN DHCP option 6 advertises 192.168.2.2; the remaining task is LAN firewall enforcement so clients cannot bypass Technitium by manually choosing public DNS.
  • Remove or disable stale Pi-hole and 192.168.100.0/24 rules after confirming there is no remaining dependency.
  • Replace broad DMZ allow with ordered policy:
  • allow DMZ clients to internal DNS resolvers;
  • allow approved management sources to management surfaces;
  • allow Caddy to published backend ports;
  • allow app hosts required outbound egress;
  • block direct client DNS and DoT bypass;
  • log denies during tuning, then reduce noisy logging.
  • Install and validate the QEMU guest agent if supported by the OPNsense build, so Proxmox can inspect clean shutdown and IP state. Do not rely on guest agent commands until it is verified.

DNS Baseline

Target split:

Resolver consumer Resolver policy
Proxmox host Local dnsmasq bootstrap resolver on 127.0.0.1; static KH3 overrides plus Cloudflare forwarders.
OPNsense External resolvers for firewall operations and gateway health.
Caddy ACME Explicit public resolvers in Caddy TLS DNS-01 config.
Headscale VPS Public resolver path on the VPS; independent of site DNS.
DMZ clients and app hosts Technitium policy resolver, currently 192.168.2.2.
User LAN clients Technitium policy resolver through DHCP.

Production target is two internal policy resolvers on different failure domains. The current single Technitium resolver is a known SPOF. A second resolver can be another Technitium instance, Unbound with replicated local zones, or a small dedicated resolver VM/CT on a different host if one becomes available.

Headscale and Remote Access

Headscale on ovps-me is correctly outside the site and should remain outside the site. The site-side dependency is the subnet router and the Proxmox host's own client.

Target state:

  • Keep CT 105 ts-router as the primary route advertiser for 192.168.2.0/24.
  • Keep pve enrolled as its own tailnet node with --accept-dns=false.
  • Do not make pve the routine subnet router.
  • Approve only narrow, validated routes. Do not advertise 192.168.100.0/24.
  • Treat remote access over Headscale as available only when site routing and WAN egress are up. It is not an out-of-band path unless pve has independent egress that bypasses the OPNsense VM.

Podman, Caddy, and DMZ Services

Target state:

  • Keep public low ports on Caddy CT 103.
  • Keep CT 101 rootless Podman services on high backend ports.
  • Caddy should reach backends by fixed IP and port or by resolver behavior that does not create a hidden startup dependency.
  • Regenerate the stale rootless Podman network namespace resolver during a maintenance window so it stops carrying old DNS state.
  • Keep Forgejo Actions runner job options with explicit DNS and host mapping until the rootless namespace is refreshed and validated.
  • Do not expose Adminer, Dozzle, Technitium UI, Proxmox UI, or OPNsense UI publicly without source-IP, VPN/tailnet, or OIDC controls.

Implementation Batches

Batch 1: Evidence and Backups

No changes.

  1. Export OPNsense config from the live UI.
  2. Capture Proxmox /etc/network/interfaces, ip -br addr, ip route, bridge vlan show, qm config 100, pct config 101-105, and guest startup order.
  3. Capture switch VLAN configuration for the shared LAN/DMZ port.
  4. Capture current pve resolver, Headscale client prefs, and cloudflared service config without printing secrets.
  5. Record WAN physical mapping: cable label, upstream device, Proxmox NIC, VM NIC, OPNsense interface, gateway, monitor.

Batch 2: Bootstrap Recovery

Goal: make host diagnosis possible when DNS or guests are broken.

  1. Set pve to static management addressing. Done before this pass: 192.168.2.10/24 on vmbr0.2.
  2. Temporary mitigation was set first: pve resolver to 192.168.2.2, then 1.1.1.1, because OPNsense blocked direct DNS from pve to 8.8.8.8:53.
  3. Persistent firewall-side fix applied: OPNsense pass rules allow pve to query 1.1.1.1:53 and 1.0.0.1:53 before the DMZ DNS bypass block.
  4. Persistent host-side fix applied: pve now runs kh3-pve-dnsmasq.service, a dedicated localhost-only dnsmasq resolver. /etc/resolv.conf points to 127.0.0.1; dnsmasq serves static KH3 bootstrap overrides and forwards public names to Cloudflare.
  5. Unbound was tested first and rejected because the Proxmox/Debian AppArmor profile denied required Unix socket creation. The idempotent script rolled back the resolver during failed attempts, and dnsmasq-base was selected as the simpler production-proven fit for this host-local use case.
  6. Ensure pve Headscale/Tailscale uses --accept-dns=false. Live prefs show CorpDNS=false and no advertised routes.
  7. cloudflared on pve was repaired by restoring host DNS and adding a timeout drop-in; it is active as of the 2026-07-02 validation.
  8. Document the remaining hard limit: if OPNsense is the only WAN router, no tunnel can survive OPNsense being fully down.

Batch 3: OPNsense Control Plane

Goal: remove stale policy and confirm failover.

  1. Verify DHCP backend and active scopes.
  2. Confirm DMZ option 6, reservations, and dynamic pool boundaries.
  3. Confirm OPNsense system DNS and gateway-monitor behavior.
  4. Test WAN1 failure, WAN2 failure, and recovery during a window.
  5. Confirm QEMU guest agent feasibility and install only if supported.
  6. Resolve the live LAN DHCP option 6 discrepancy: current Dnsmasq config advertises 1.1.1.1 on LAN while the description says LAN clients use Technitium. Completed July 2, 2026: LAN now advertises 192.168.2.2.
  7. Decide and implement LAN DNS bypass enforcement. The DMZ already blocks direct client DNS and DoT bypass; LAN does not yet mirror that policy.

Batch 4: DMZ Policy

Goal: replace broad DMZ allow with auditable intent.

  1. Inventory each DMZ host and required flows.
  2. Add explicit allow rules above the broad allow rule.
  3. Add logging to candidate deny rules temporarily.
  4. Remove stale Pi-hole, Squid, and 192.168.100.0/24 rules only after confirming no dependency.
  5. Retire or document inactive reservations such as old k3s, Squid, RustDesk, and Windows service hosts.

Batch 5: DNS Resilience

Goal: make internal DNS policy redundant and prevent another listener outage.

  1. Add a second internal resolver on a separate failure domain where possible.
  2. Replicate internal zones or records.
  3. Advertise both resolvers through DHCP.
  4. Keep listener health checks on each resolver.
  5. Validate direct external DNS and DoT blocks still work for clients.

Batch 6: Service and Podman Cleanup

Goal: remove hidden DNS and routing drift.

  1. Refresh CT 101 rootless Podman network namespace during a no-job window.
  2. Validate Forgejo, runner, Vaultwarden, Adminer, Dozzle, docs-static, and PostgreSQL after namespace refresh.
  3. Move any service route still depending on stale Traefik/Pi-hole assumptions to the Caddy/Technitium pattern.
  4. Clean the remotely managed Cloudflare tunnel configuration. Many current tunnel origins still point at legacy https://proxy.kh3group.com; this is a Docker/Traefik-era compatibility name, not the desired Podman/Caddy naming model. Replace those origins with direct Caddy 192.168.2.3 targets or validated service IP:port targets, then remove the temporary Proxmox host-local proxy.kh3group.com resolver entry.

Batch 7: True Break-Glass Upgrade

Goal: remove the remaining single-host circular dependency.

Choose one:

  • Add a fourth management interface, even a dedicated USB Ethernet adapter, to a small independent management network.
  • Move OPNsense to dedicated hardware.
  • Add a second small router/firewall for management-only access.
  • Add a second Proxmox node and move routing out of the only virtualization host's dependency chain.

Without one of these, the design can be improved but cannot be called fully production-grade for remote recovery.

Validation Matrix

Failure test Expected result
Technitium stopped pve, OPNsense, cloudflared, Headscale client control-plane checks still resolve required public names. Clients lose policy DNS until backup resolver is available.
CT 101 stopped Caddy routes to CT 101 fail, but DNS, OPNsense, Proxmox, and tailnet routing remain available.
CT 103 stopped Public HTTPS apps fail, but DNS, Proxmox, OPNsense, and direct backend checks remain available.
CT 105 stopped Tailnet subnet route fails, but pve remains reachable as its own tailnet node if WAN egress works.
OPNsense rebooted Local management path to pve still works. Remote tunnel works only if host egress does not depend on OPNsense.
WAN1 failed OPNsense shifts to WAN2 and Headscale/cloudflared reconnect after convergence.
WAN2 failed WAN1 remains active.
Switch trunk mis-tagged Proxmox local console or out-of-band management remains available for rollback.

Lessons Learned

  • Verify the live configuration tree before editing OPNsense XML. Current Dnsmasq DHCP options live under top-level <dnsmasq>, while older exports and stale assumptions may point elsewhere.
  • Prefer small copied scripts over complex SSH heredocs for OPNsense changes. The default shell can reinterpret quoting before /bin/sh or Python receives the intended script.
  • Plain /etc/resolv.conf fallback is not split DNS. Proxmox host bootstrap DNS needs a local resolver with explicit KH3 overrides plus external forwarders, not ordered nameservers that mix Technitium and public DNS.
  • Unbound is technically appropriate for many resolver designs, but on this Proxmox host it failed because AppArmor denied required Unix socket creation. The selected host bootstrap resolver is therefore dedicated dnsmasq-base unless the AppArmor profile is deliberately redesigned.
  • proxy.kh3group.com is a legacy Docker/Traefik-era origin. It remains only because Cloudflare's remotely managed tunnel config still references it. Treat the local resolver entry as compatibility, not as a production naming pattern.
  • LAN DNS bypass enforcement is more urgent than DMZ DNS bypass enforcement. LAN contains unmanaged or semi-managed client behavior; DMZ hosts are servers with narrower, more predictable flows. LAN enforcement still needs an approved change window because it can affect client devices immediately.

Open Decisions

  • Final current LAN CIDR after the 192.168.100.0/24 retirement.
  • Whether the shared NIC trunk has a separate management VLAN today.
  • Whether the WAN NICs are PCI passthrough or Proxmox bridges in live OPNsense.
  • Whether cloudflared is truly intended to run on pve, CT 103, or elsewhere.
  • Whether Cloudflare tunnel origins should all terminate at Caddy 192.168.2.3 or only the selected public hostnames that are still active.
  • Whether a second physical or logical management path can be added.
  • Whether stale Squid and k3s reservations represent active systems or old recovery material.

References