fix(proxies): advertise an address jobs can actually reach - #208
Merged
Merged
Conversation
Every cache proxy advertises the address it was asked to bind: the CNI bridge gateway. That is right only where job containers share this host's bridge — on Linux. On macOS and Windows hosts, jobs run inside a VM, and from in there the bridge gateway is the VM's OWN bridge, where nothing listens. The proxies are on the host. Measured on this fleet's Mac, 169 days uptime: 192.168.64.1:8082 -> 200 (0.195s) host, on the Vz NAT 10.88.0.1:8082 -> 000 (4s timeout) the advertised value 10.88.x interfaces on the host: 0 caches: gomod 0B, cargo 0B, ghrel 0B, composer 0B The proxies were healthy the whole time. Only the advertised address was unreachable, and the existing health gate could not see it because it probes loopback, which always answers. The node downloads ~29 GB/day, and none of it was cacheable. resolveJobProxyEnv is per-platform: - linux: identity. The bridge gateway is correct, and deliberately NOT probed — CNI creates the bridge with the first job container, so a startup probe would fail every time and disable working proxies. - darwin: find this host's address on the guest NAT, rewrite the env to it, then probe every advertised address and advertise nothing if any fails. - windows: nothing, as today. Same defect, but an unreachable GOPROXY HANGS Windows builds instead of failing over, so suppression stays until the Hyper-V host address is confirmed on metal. Rewriting rather than re-rendering keeps each proxy's own env logic as the single source of truth; only the authority was ever wrong. The NAT range locates a candidate, it is not trusted: the address is probed before use, so if Apple moves the range the result is "no proxy env" (bandwidth) rather than a black hole (broken builds). Logic lives in pkg/proxies, not the darwin file, because the darwin build needs macOS cgo and cannot be compiled off a Mac — code there is unverifiable by CI's other platforms. 9 tests, all platform-independent. Revert-verified: making RewriteEnvHost a no-op fails the rewrite tests. linux and windows cross-build clean; darwin is CI-only (its vz dependency does not cross-compile, same on untouched main).
|
ePHPm Preview — removed Preview deployment has been torn down. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every cache proxy advertises the address it was asked to bind: the CNI bridge gateway. That is right only where job containers share this host’s bridge — on Linux. On macOS and Windows hosts, jobs run inside a VM, and from in there the bridge gateway is the VM’s own bridge, where nothing listens. The proxies are out on the host.
Measured, on this fleet’s Mac (169 days uptime)
The proxies were healthy the entire time. Only the advertised address was unreachable — and the existing health gate could not see it, because it probes loopback, which always answers. That node pulls ~29 GB/day and none of it was cacheable.
resolveJobProxyEnv, per platformpkg/proxies/listen.go), so a startup probe would fail every time and disable proxies that work fine.GOPROXYhangs Windows builds rather than failing over, so suppression stays until the Hyper-V host address is confirmed on metal. Guessing it would reintroduce hung builds, which is worse than the bandwidth.Two deliberate choices:
Rewrite, don’t re-render. Each proxy already knows how to build its own env (
GOPROXY,CARGO_*,GHREL_PROXY,COMPOSER_REPO_PACKAGIST); only the authority was ever wrong. Re-deriving it per platform would duplicate seven proxies’ logic and let the copies drift.The NAT range locates, it does not authorise.
192.168.64.0/24is only used to find a candidate among local interfaces — the same rangepkg/vmalready sweeps to find guests in ARP. The address is then probed. If Apple moves the range, the outcome is "no proxy env" (bandwidth) rather than a black hole (red builds).Testing
9 tests, all platform-independent, in
pkg/proxies. The logic lives there rather than in the darwin file on purpose: the darwin build needs macOS cgo and cannot be compiled off a Mac, so code in that file is unverifiable by CI’s other platforms. The darwin file is now a thin wrapper.Revert-verified — making
RewriteEnvHosta no-op fails the rewrite tests.linux and windows cross-build clean. darwin is CI-only: its
Code-Hex/vzdependency does not cross-compile, and fails identically on untouchedmain.Scope
This recovers Go modules, cargo/rustup, GitHub release artifacts and Packagist on the Mac. It does not cover Homebrew, Xcode or macOS system downloads, which may be most of that 29 GB/day — I would rather measure what the node actually fetches than predict a saving.
Not rolled to the fleet.