fix(cluster): resolve DNS failures on systemd-resolved hosts

[brianwtaylor]

@drew thank you for approving my vouch, had this brewing as a small fix for the repo, look like this change could help some folks. doing a qa pass for this change, will add a visual diagram or two as well.

Summary

• Mount host systemd-resolved config into the gateway container so the entrypoint can extract real upstream DNS resolvers
• Bypass the broken cross-namespace DNAT proxy that silently fails on Ubuntu/systemd-resolved hosts (including DGX Spark)
• Preserve existing DNAT and public DNS fallbacks for non-systemd environments

Closes #437
Related: #415

Changes

crates/openshell-bootstrap/src/docker.rs — Bind-mount /run/systemd/resolve/resolv.conf (read-only) into the container when the path exists on the host. Skipped on macOS/Windows where the path doesn't exist.

deploy/docker/cluster-entrypoint.sh — Add get_upstream_resolvers() that extracts non-loopback nameservers from the mounted systemd-resolved config (or falls back to /etc/resolv.conf). When upstream resolvers are found, write them directly to the k3s resolv.conf instead of attempting the DNAT proxy. Also improves DNS verification logging on failure.

Root Cause

Docker's embedded DNS at 127.0.0.11 is only reachable from the container's own network namespace. The existing DNAT rules forward to this loopback address, but k3s pods run in child network namespaces where the forwarded packets are dropped as martian packets. On systemd-resolved hosts, /etc/resolv.conf contains 127.0.0.53 (another loopback), so the fallback also fails silently.

Testing

• Tested on DGX Spark (Ubuntu 24.04, systemd-resolved, Docker with cgroupns=host)
• Verified DNS resolution works from k3s pods after the fix
• Verified the mount is skipped on hosts without systemd-resolved

Automated Tests

cargo test -p openshell-bootstrap

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[brianwtaylor]

I have read the DCO document and I hereby sign the DCO.

[drew]

I'm just a little worried about mounting existing system files into the cluster container (this would be the first time we do it).

How about as an alternative approach we sniff the resolvers from the boostrap crate and then pass them into the cluster container as parameters.

[brianwtaylor]

let me explore this, i have some qa going right now as well. happy to chat to ensure this change aligns with your standards.

update: great call out on extending the existing architecture

[brianwtaylor]

@drew after your feedback, here is a breakdown of the DNS FLOW before and after and the validation toplogy.

DNS FLOW — BEFORE vs AFTER
──────────────────────────

BEFORE (broken on systemd-resolved hosts):

  Pod ──→ CoreDNS ──→ resolv.conf ──→ iptables DNAT ──→ Docker DNS
          (cache)      127.0.0.11       PREROUTING       127.0.0.11
                            │                                │
                            └──── FAILS ─────────────────────┘
                            loopback DNAT from pod namespace
                            dropped as martian packet

AFTER (PR #478):

  Pod ──→ CoreDNS ──→ resolv.conf ──→ upstream resolver ──→ response
          (cache)     e.g. 192.168.1.1    (direct UDP)
                       ▲
                       │
                  set by Rust bootstrap:
                  resolve_upstream_dns()
                  reads /run/systemd/resolve/resolv.conf
                  passes via UPSTREAM_DNS env var
                  entrypoint writes to k3s resolv.conf

NON-SYSTEMD HOSTS (macOS, WSL2, Alpine) — unchanged:

  Pod ──→ CoreDNS ──→ resolv.conf ──→ iptables DNAT ──→ Docker DNS ──→ host
          (cache)     container IP      PREROUTING       127.0.0.11

  /run/systemd/resolve/resolv.conf absent → UPSTREAM_DNS not set →
  entrypoint falls back to existing DNAT proxy path. Zero behavior change.

I also did some hardware validation with these changes, below is a breakdown of the validation topology i used.

OPENSHELL DNS FIX (PR #478) — VALIDATION TOPOLOGY
══════════════════════════════════════════════════

                  ┌─────────────────────────┐
                  │     Node A (Linux)      │
                  │    aarch64 · GPU        │
                  │                         │
                  │  BASELINE + ORCHESTRATOR │
                  │  (read-only, runs all   │
                  │   tests from here)      │
                  └─────┬──────────┬────────┘
                        │          │
          high-speed    │          │ LAN
          interconnect  │          │
                        │          ├──────────────┐
              ┌─────────▼──┐   ┌───▼──────────┐  ┌▼──────────────┐
              │  Node B    │   │  Node C      │  │  Node D       │
              │  Linux     │   │  macOS       │  │  Windows/WSL2 │
              │  aarch64   │   │  Apple Si    │  │  x86_64       │
              │            │   │  no systemd  │  │  no systemd   │
              │ TEST TARGET │   │              │  │               │
              │ (DNS-fixed │   │  CONTROL     │  │  CONTROL      │
              │  gateway   │   │  ✓ verified  │  │  ✓ verified   │
              │  deployed) │   │              │  │               │
              └────────────┘   └──────────────┘  └───────────────┘

WHAT EACH NODE PROVED
═════════════════════

Node A ─── "Does the fix break anything that already works?"
            Captured baseline iptables, TLS certs, and DNS state.
            All comparisons showed zero drift.

Node B ─── "Does the new code handle edge-case input safely?"

Node C ─── "Does the fix affect macOS hosts?"
            No systemd-resolved → no UPSTREAM_DNS set → no change.
            Existing DNAT proxy path untouched.

Node D ─── "Does the fix affect Windows/WSL2 hosts?"
            No systemd-resolved → no UPSTREAM_DNS set → no change.
            Existing DNAT proxy path untouched.

[brianwtaylor]

sorry trying not to rush with this fix.