Skip to content

fix(proxies): advertise an address jobs can actually reach - #208

Merged
luthermonson merged 1 commit into
mainfrom
fix/proxy-advertise-host
Oct 2, 2026
Merged

luthermonson merged 1 commit into
mainfrom
fix/proxy-advertise-host

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Every cache proxy advertises the address it was asked to bind: the CNI bridge gateway. That is right only where job containers share this host’s bridge — on Linux. On macOS and Windows hosts, jobs run inside a VM, and from in there the bridge gateway is the VM’s own bridge, where nothing listens. The proxies are out on the host.

Measured, on this fleet’s Mac (169 days uptime)

192.168.64.1:8082  -> 200  (0.195s)      host, on the Vz NAT
10.88.0.1:8082     -> 000  (4s timeout)  the advertised value
10.88.x interfaces on the host: 0

caches: gomod 0B   cargo 0B   ghrel 0B   composer 0B

The proxies were healthy the entire time. Only the advertised address was unreachable — and the existing health gate could not see it, because it probes loopback, which always answers. That node pulls ~29 GB/day and none of it was cacheable.

resolveJobProxyEnv, per platform

  • linux — identity. The bridge gateway is correct, and deliberately not probed: CNI creates the bridge with the first job container (see pkg/proxies/listen.go), so a startup probe would fail every time and disable proxies that work fine.
  • darwin — find this host’s address on the guest NAT, rewrite the env to it, then probe every advertised address and advertise nothing if any fails.
  • windows — nothing, as today. Same defect, but an unreachable GOPROXY hangs Windows builds rather than failing over, so suppression stays until the Hyper-V host address is confirmed on metal. Guessing it would reintroduce hung builds, which is worse than the bandwidth.

Two deliberate choices:

Rewrite, don’t re-render. Each proxy already knows how to build its own env (GOPROXY, CARGO_*, GHREL_PROXY, COMPOSER_REPO_PACKAGIST); only the authority was ever wrong. Re-deriving it per platform would duplicate seven proxies’ logic and let the copies drift.

The NAT range locates, it does not authorise. 192.168.64.0/24 is only used to find a candidate among local interfaces — the same range pkg/vm already sweeps to find guests in ARP. The address is then probed. If Apple moves the range, the outcome is "no proxy env" (bandwidth) rather than a black hole (red builds).

Testing

9 tests, all platform-independent, in pkg/proxies. The logic lives there rather than in the darwin file on purpose: the darwin build needs macOS cgo and cannot be compiled off a Mac, so code in that file is unverifiable by CI’s other platforms. The darwin file is now a thin wrapper.

Revert-verified — making RewriteEnvHost a no-op fails the rewrite tests.

linux and windows cross-build clean. darwin is CI-only: its Code-Hex/vz dependency does not cross-compile, and fails identically on untouched main.

Scope

This recovers Go modules, cargo/rustup, GitHub release artifacts and Packagist on the Mac. It does not cover Homebrew, Xcode or macOS system downloads, which may be most of that 29 GB/day — I would rather measure what the node actually fetches than predict a saving.

Not rolled to the fleet.

Every cache proxy advertises the address it was asked to bind: the CNI
bridge gateway. That is right only where job containers share this host's
bridge — on Linux. On macOS and Windows hosts, jobs run inside a VM, and
from in there the bridge gateway is the VM's OWN bridge, where nothing
listens. The proxies are on the host.

Measured on this fleet's Mac, 169 days uptime:

  192.168.64.1:8082 -> 200 (0.195s)      host, on the Vz NAT
  10.88.0.1:8082     -> 000 (4s timeout)  the advertised value
  10.88.x interfaces on the host: 0
  caches: gomod 0B, cargo 0B, ghrel 0B, composer 0B

The proxies were healthy the whole time. Only the advertised address was
unreachable, and the existing health gate could not see it because it
probes loopback, which always answers. The node downloads ~29 GB/day,
and none of it was cacheable.

resolveJobProxyEnv is per-platform:

- linux: identity. The bridge gateway is correct, and deliberately NOT
  probed — CNI creates the bridge with the first job container, so a
  startup probe would fail every time and disable working proxies.
- darwin: find this host's address on the guest NAT, rewrite the env to
  it, then probe every advertised address and advertise nothing if any
  fails.
- windows: nothing, as today. Same defect, but an unreachable GOPROXY
  HANGS Windows builds instead of failing over, so suppression stays
  until the Hyper-V host address is confirmed on metal.

Rewriting rather than re-rendering keeps each proxy's own env logic as
the single source of truth; only the authority was ever wrong.

The NAT range locates a candidate, it is not trusted: the address is
probed before use, so if Apple moves the range the result is "no proxy
env" (bandwidth) rather than a black hole (broken builds).

Logic lives in pkg/proxies, not the darwin file, because the darwin build
needs macOS cgo and cannot be compiled off a Mac — code there is
unverifiable by CI's other platforms. 9 tests, all platform-independent.
Revert-verified: making RewriteEnvHost a no-op fails the rewrite tests.

linux and windows cross-build clean; darwin is CI-only (its vz dependency
does not cross-compile, same on untouched main).
@ephpm

ephpm Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

ePHPm Preview — removed

Preview deployment has been torn down.

@ephpm
ephpm Bot temporarily deployed to preview-pr-208 October 1, 2026 05:45 Inactive
@ephpm
ephpm Bot temporarily deployed to preview-pr-208 October 1, 2026 05:45 Inactive
@luthermonson
luthermonson merged commit 9ed0002 into main Oct 2, 2026
4 checks passed
@luthermonson
luthermonson deleted the fix/proxy-advertise-host branch October 2, 2026 04:17

This branch was previously deployed

1 inactive deployment
preview-pr-208 — 69952c27 Deployed Oct 1, 2026 by ephpm[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant