Add distributed compilation: dispatch cache misses to worker nodes - #78
Merged
Conversation
First phase of distributed compilation: the protocol vocabulary and the
scheduling state, both testable without a socket. No binary dispatches or
executes anything yet -- fastcache-cc still compiles every miss locally, and a
daemon serves these verbs only on an endpoint an operator opted in.
Four verbs join the 0xFC table: Register and Heartbeat (a worker announcing
itself and staying alive), Lease (a client asking where to compile), and Compile
(a client handing a worker one translation unit). Seven error codes join it, each
a refusal the client answers by compiling locally -- distribution must be
incapable of failing a build, so no path here can express one.
Which endpoint serves them is an operator decision, not a build-time constant.
BindConfig gains a ListenerRole mask defaulting to Cache alone, SessionContext
carries the accepting listener's mask, and the server loop copies it per bind in
all three platform paths. The compile-cache surface may reasonably be reachable
across a build LAN; the surface that causes a compiler to RUN on someone else's
machine must be something switched on deliberately, on an endpoint that can be
firewalled separately. A dispatch verb on a cache-only listener is refused with a
typed reply rather than a closed connection, which is what lets that endpoint move
without every client needing to know.
WorkerRegistry holds the fleet. Two rules are load-bearing:
- A job goes only to a worker whose fingerprint is BYTE-IDENTICAL to the
client's. Not compatible, not same-major-version. The cache key already
depends on compiler identity, so this is an invariant the cache needs anyway;
distribution only makes the consequence worse. An over-strict match costs a
local compile, an over-loose one produces a silently wrong object that is then
stored under a key other machines fetch. Those errors are not symmetric, so no
configuration can loosen it.
- Least-outstanding-jobs, not round-robin. Compile times vary by an order of
magnitude within one build, so distributing arrivals rather than load queues a
long translation unit behind another while a worker idles.
LeaseTable authorizes jobs and suppresses duplicates. When sixty parallel clients
miss the same key after a header change -- the ordinary shape of a miss on a
shared cache, not an exotic one -- only the first is dispatched and the rest
compile locally. That is something neither distcc nor sccache-dist can do, because
neither is also the cache. Leases expire because a client can die between taking
one and sending the job (Ctrl-C on a build is the common case), and without expiry
that key would be marked in-flight forever -- one interrupted build would
permanently un-distribute a translation unit.
Both components are pure with respect to I/O, over an injected IClock, so every
expiry and capacity rule is a unit test against ManualClock rather than a sleep.
Payloads: bulk fields travel in a [u8 codec][u32 rawLen][bytes] envelope using
Core/Compression's ids, with the accepted-codec list carried IN the request so the
two ends agree without a handshake -- every exchange is client-initiated, so the
reply picks from what the request offered, and the round-trip count is unchanged.
rawLen is what lets a receiver reject a declared expansion before decompressing a
byte. Nothing in common falls back to Identity rather than refusing: a build must
never lose its cache because two peers were compiled with different codec sets.
The scheduler's control verbs carry a payload ceiling far below the session cap.
That listener is meant to be reachable by a whole fleet, and a scheduler that can
be made to allocate 256 MiB per frame by anything that authenticated once is a
scheduler that stops scheduling. Compile is the deliberate exception, since it
carries a preprocessed translation unit.
The daemon refuses Compile even on a dispatch listener, with its own message: it
schedules, fastcache-compile-node will execute. Sending a job to the scheduler is
a client bug that would otherwise present as an unexplained refusal.
Signed-off-by: Christian Parpart <christian@parpart.family>
christianparpart
force-pushed
the
feature/distributed-compilation
branch
from
August 22, 2026 10:11
bd27043 to
cb834a2
Compare
A job may only be dispatched to a worker whose toolchain is identical to the
client's, and this is what "identical" means. Pure: no filesystem, no clock --
gathering the file list is the caller's job, which is what makes every rule below
a unit test rather than a fixture needing a real compiler installed.
Two inputs, because neither subsumes the other and each is too weak alone.
The compiler's version banner is what the cache key already uses, and for the
cache its residual is tolerable and documented: two machines can print an
identical --version while resolving different libstdc++ headers, and the cache
answers that with a stale path the replay guard probes. For DISTRIBUTION the same
residual produces a silently wrong object, so the headers are folded in too.
The headers alone are weak in the other direction -- two compilers can share a
header tree and generate different code from it -- so the banner stays.
Three mechanics are load-bearing:
Paths are relative to the include root they were found under. Two machines running
the same toolchain at different install prefixes (/usr/lib/gcc/... against
/opt/toolchains/gcc-13/..., or a vendored SDK checked out at two depths) must
fingerprint the SAME, or distribution is disabled between exactly the machines it
exists to connect -- and disabled silently, presenting as "no worker ever matches"
rather than as an error.
Contents, never mtime or size. Install times differ on every machine, so an
mtime-derived fingerprint would make every pair of machines mismatch, again
silently. Size alone is far too weak: two headers differing by one character are
the case that matters.
Path and hash are folded as separate length-prefixed pieces. Concatenation is not
a framing -- {"ab","c"} and {"a","bc"} would digest identically -- and the error
it admits is a false MATCH, which dispatches to the wrong toolchain. The two error
directions are not symmetric: an over-strict fingerprint costs a local compile.
The list is sorted here rather than trusted from the caller, because a caller's
order comes from a directory traversal and two machines with byte-identical
toolchains can enumerate them differently.
The value travels as an opaque string, so strengthening how it is computed later
is not a wire change.
Signed-off-by: Christian Parpart <christian@parpart.family>
The endpoint that serves distributed compilation is now spellable. It is OFF without the flag, and it serves scheduling ONLY, not the cache: something that can make a compiler run on another machine is a different trust boundary from something that reads and writes a cache, so it gets a port an operator can firewall and TLS-require independently. Folding the cache back in would give that up while still looking like a separate port. An operator who genuinely wants one endpoint doing both can say so in YAML, where the mask is spelled directly. The mapping between a --listen* spelling and a BindConfig now lives in ONE table, Config.hpp's ListenerFlags, read in both directions: CliParser binds a parser to each row, and BuildServiceArgv spells a bind back out by finding its row. Those two must agree, or a daemon registers with a listener set it will not accept back -- the class of defect FormatListenHost and MaybeQuote already exist to prevent, reached through a different door. It was a `bind.tls ? "listen-tls" : "listen"` ternary in each place while there were two kinds. A third would have made that a three-way conditional in three files, five times over inside one function, which is the shape a table replaces. ListenFlagFor falls back to plain `listen` when no row matches, which is the safe direction: a bind whose role mask a future build does not recognise is re-registered as an ordinary cache endpoint rather than silently gaining a dispatch surface it was never given. Two of the repository's own drift guards caught this change mid-flight, which is what they are for: the CliOptions sample table refused a row it had no sample for, and ServiceControl's "every Config-backed flag reaches the service argv" refused a flag BuildServiceArgv could not emit. The listener flags are excluded from the second because they are repeatable and emitted as a group -- so the exclusion now points at the case that covers them, and that case asserts the role survives a registration in both directions. A dispatch endpoint re-registered as a plain listener would come up serving the cache and simply never schedule; a cache endpoint re-spelled as a dispatch one would hand a scheduling surface to a port the operator opened for the cache alone. Signed-off-by: Christian Parpart <christian@parpart.family>
christianparpart
marked this pull request as draft
August 22, 2026 10:38
A worker compiles PREPROCESSED text, so the command line it needs is not the build's minus a few things -- it is the subset that still means something once the headers are inlined and the macros expanded. The source, every path-valued flag whatever its role, and the driver's compile-only and dependency switches all go; -std, -O, -g, -W, -f, -m and their kin stay, because they decide what the compiler emits and an object built without them is not the object that was asked for. Dropping the known path-valued flags reads `PathValueFlags()` for a fourth question rather than adding a fourth list of spellings, which is the same reason the preprocess line reads it for its third. But that alone is a DENY-LIST, and a deny-list is the losing game here. `PathValueFlags()` does not know -isystem, -iquote, --sysroot, -B, -specs= or -fplugin=, and it should not have to: it exists to answer questions about the cache key. Several of those point a compiler at an EXECUTABLE, which is precisely the surface a worker must not expose to a client. A test asserting "nothing surviving names a path" is what surfaced this, by failing. So the last word is a positive check on what survives: an argument carrying a path separator, or opening a response file, makes the whole translation unit undispatchable. REFUSED rather than stripped, because the two failure modes are not comparable -- stripping an argument this function does not recognise changes the generated code and hands back an object that is quietly not the one asked for, while refusing costs one local compile. `@` is called out separately because a response file in the working directory names a path with no separator at all. The separator test skips the INTRODUCER first, and which characters introduce is the driver's answer rather than this function's. `/` starts an option for an MSVC driver and an absolute path everywhere else -- the rule this launcher already lives by, recorded in AGENT.md -- so testing the raw argument refused `/std:c++20`, `/O2` and therefore every MSVC compile, while a `\` inside `/DCONFIG=C:\x` is still exactly the signal wanted. The first cut got this wrong and an MSVC case caught it. This is the second of two independent barriers, not the only one: the worker separately refuses to take its compiler from the client. Signed-off-by: Christian Parpart <christian@parpart.family>
The client half of distributed execution: ask the scheduler for a worker, send it the preprocessed translation unit, get an object back. Two short request/reply exchanges on fresh connections, the same shape every other 0xFC operation uses. Nothing calls this yet -- the wiring at the miss is the next commit -- but it is complete and driven end to end against scripted endpoints, which is what the IEndpointDialer seam is for. ITcpClient models one CONNECTED peer, which sufficed while there was only ever one address; distribution talks to two, so the act of connecting is what has to be injected. Three decisions carry weight. THE CLIENT STORES, NEVER THE WORKER. A worker is given no cache credentials at all. Today a STORE is trusted because whoever stores compiled the thing themselves, so the worst they can do is poison their own key space with something they would have gotten anyway. If workers stored, one rogue worker would poison keys every other machine fetches. Routing the object back through the client keeps that trust model exactly as it is and needs no new authorization anywhere. A FAILING REMOTE COMPILE IS A SUCCESSFUL DISPATCH. DispatchStatus separates "a worker ran the compiler and it rejected the code" from "the job never ran". Collapsing them would send the caller down the cache-unavailable path instead of showing the user their own compile error. There is deliberately no "failed" status: every way this can go wrong ends with the caller compiling locally, because the client is holding the source. THE GRANT RELAYS THE WORKER'S CODECS. The client is about to send that worker several megabytes of preprocessed text and must choose a codec for it. Without the relay it would either send Identity always -- giving up compression on the one payload large enough to care -- or guess, and a guess the worker cannot decode is a refused job discovered only after the whole payload has crossed the network. The scheduler already knows the answer. Arguments travel one length-prefixed field each, not joined: an argument may contain a space, and a receiver splitting on whitespace would turn `-DMSG=hello world` into two flags. A truncated list decodes to NOTHING rather than a prefix, because a partial argument list is a different compile from the one that was authorized. This is where the launcher takes its first compression dependency, which the CMakeLists now records: Core/Compression.cpp joins _fc_cc_core so the codec table has one implementation rather than two, at the cost of zstd/lz4 (CPM, so the static-CRT link is unaffected) and the macOS CPM_USE_LOCAL_PACKAGES constraint the packaging job already carries. FASTCACHED_ENABLE_COMPRESSION=OFF remains a working configuration: Identity is always available and the negotiation falls back to it, so such a launcher still dispatches -- asserted, not assumed. Signed-off-by: Christian Parpart <christian@parpart.family>
Wires distribution into the miss path. FASTCACHE_SCHEDULER is the whole switch: unset, every miss compiles locally exactly as before. A dispatched compile is shaped to look exactly like a local one -- object on disk at cmd.objPath, dependency record written, streams in hand -- so everything after the hook is unchanged and cannot tell the difference. The STORE, the direct-mode manifest and the statistics keep one path rather than gaining a second that can diverge from it. THE DEPENDENCY RECORD IS WRITTEN BY THE CLIENT, and this is the part that makes dispatch correct rather than merely working. A worker compiles preprocessed text, which has no #include left in it, so its compiler reports no dependencies -- there are none to report. The build system asked anyway: Ninja reads a depfile (or /showIncludes under deps = msvc) to decide when this translation unit must be rebuilt. Handing back an object with no dependency record makes it STOP rebuilding the TU when its headers change -- a wrong build with a zero exit code that persists until someone cleans, and the same defect the hit path already guards against in as many words. The client's own probe opened every one of those headers to compute the key, so it writes the record rather than the worker inventing one. DependencyOutput renders both forms and is tested by round-tripping through the launcher's OWN parsers, so the writer and the reader cannot drift into agreeing on a format nothing else produces. A path containing a space is escaped, because unescaped it reads as two dependencies and the second names a file that does not exist -- which make and Ninja answer by rebuilding forever, with a zero exit code. A REMOTE FAILURE IS RETRIED LOCALLY before its diagnostics are believed. A compile can fail because of the code, in which case both runs agree and the user sees the same errors; or because of something about the worker, in which case the local run succeeds and the build is right. Reporting the remote failure directly would let one bad node fail builds that are fine, which is the failure mode that gets distribution switched off and never switched back on. The cost is one wasted remote attempt. Verified against a real daemon: with a scheduler configured and nothing listening, the compile reports "not dispatched (scheduler unreachable)", compiles locally, writes its object and depfile, STOREs, and the next compile HITs. Distribution cannot fail a build. The preprocessed text is kept alive past the key block ONLY when a scheduler is configured -- that block exists to drop several megabytes the moment the key is computed, and dispatch is the one caller that still needs it. It is moved, not copied: KeyInputs was const, which made std::move a silent copy of exactly that payload while the comment claimed the opposite. clang-tidy's performance-move-const-arg caught it. Signed-off-by: Christian Parpart <christian@parpart.family>
The worker's execution core: given a job, produce an object. Over IProcessRunner, so every rule below is a unit test rather than a fixture with a real compiler. Three things a job is NOT allowed to decide, and they are the whole design. THE COMPILER. A job names a FINGERPRINT, never a program. The worker maps that to a path from its own configuration, and a fingerprint it does not have is refused rather than served with a default. A job that could name its own compiler would let anyone who can reach the port run an arbitrary program -- not a hardening detail but the difference between a build accelerator and a remote shell. ANY PATH. The client's RemoteCompileArgs already refuses a command line carrying anything that could name a file, and this checks again on the receiving side. The two protect against different things: the client's protects an honest client from dispatching something that would not work, and this one protects the worker from a client that is not honest. A worker that trusted the client's filtering would be secured by code running on the caller's machine. WHERE ANYTHING IS WRITTEN. The source path, the object path and the working directory are all the worker's, inside a per-job scratch directory it creates and removes however the job returns. The client's source name is used for exactly one thing -- its extension, so the compiler picks the right language -- and even that is taken from a fixed table rather than trusted, because an extension is one of the few places a driver will accept something surprising. The argument check skips ONE introducer before looking for a separator, which is the rule AGENT.md records: `/` starts an option for an MSVC driver and an absolute path everywhere else. Testing the raw argument refuses `/O2` and therefore every MSVC job -- a test asserting `/O2` is acceptable is what caught that. Both families' introducers are accepted, because a worker is told a fingerprint rather than a command line and has no driver to ask; that is marginally more permissive than the client's per-family rule and does not matter, since the separator check is what carries the weight. Two failure modes are kept distinct because the client must act differently on them. A compiler that RAN and rejected the code returns its exit code and diagnostics -- that is the client's answer. A compiler that could not be spawned is a refusal, so the client can tell "this worker is broken" from "your code does not compile" and send the job elsewhere rather than fail the build. A compiler claiming success while writing nothing is likewise refused, not returned as an empty object the client would write to disk and cache. Signed-off-by: Christian Parpart <christian@parpart.family>
GCC 14 at -O2 inlines std::istreambuf_iterator's sgetc and then reports -Werror=null-dereference inside <streambuf> itself. A false positive, but not one this project can silence: warnings are errors and the rule is to fix them at the source rather than suppress them. Three Linux jobs failed on it -- the gcc-release build, the Linux e2e (which builds with the same preset) and the deb/rpm package -- while every clang job was green, which is why it reached CI at all. Seeking to the end for the size and reading once is the fix, and it is the better implementation regardless: one allocation and one read instead of a per-character loop through a stream buffer. Note main.cpp's ReadFileBytes still uses the iterator form and still compiles -- the warning depends on inlining context, not on the iterator as such -- so this is not a rule about the construct, it is one about not reaching for it in new code. Also drops std::format of a pointer from the job tests in favour of a counter: formatting a void const* is specified but is exactly the sort of thing that varies between standard libraries, and a reproducible scratch name is easier to find when a case leaves one behind. Signed-off-by: Christian Parpart <christian@parpart.family>
WorkerProtocol turns one 0xFC request into one reply, and WorkerRegistrar is the one part of a worker that initiates rather than answers. Split so the answering half is pure -- bytes in, bytes out, no socket, no spawn of its own -- and can be tested by handing it a frame and reading the answer, with no listener and no compiler installed. A worker answers exactly one verb. Everything else, including the scheduler's own, is refused with dispatch-not-permitted: a worker is not a scheduler and not a cache. That refusal is a REPLY, so a client that sent the wrong verb to the wrong port learns which -- a dropped connection is indistinguishable from a dead host, which is the whole reason this framing declares its lengths. The lease is validated BEFORE the payload is decompressed, let alone compiled. An unauthorized peer must not be able to make a worker do the expensive part; checking after decoding would make the refusal cost more than the attack. Validation is injected rather than called, because "ask the scheduler" is I/O and this file has none -- and because it is the seam where a worker's trust model can change without touching the framing. Today the plausible implementations are "accept any token" (a fleet whose boundary is network reachability) and "ask the scheduler"; a signed token would be a third. The COMPILE request gained the client's accepted-codec list. Writing the reply path is what surfaced the gap: the worker never sees the LEASE, so without this it would have to send every object uncompressed or ask the scheduler, and asking would put a round trip on the one exchange that must not have one. It falls back to Identity when nothing is in common or compression did not shrink the object, which is the same shrink-check a stored value gets. JobRefusal maps to wire codes through a switch with no default, so a refusal added later without a code here is a compile error rather than a silent malformed-frame that tells an operator nothing. Two of them mean "this worker is broken" rather than "your code is wrong", and are deliberately not expressible as an exit code: the client has to be able to send the job elsewhere instead of failing the build. Heartbeat forgets its worker id when the scheduler answers unknown-lease. That is the scheduler saying "register again" -- it restarted, or expired this entry -- and a worker that ignored it would heartbeat into a void forever while the fleet ran without it. Signed-off-by: Christian Parpart <christian@parpart.family>
bugprone-unchecked-optional-access again, and the fourth time it has reached CI in this branch -- always in a TEST file, because clang-tidy cannot see a has_value() guard through Catch2's REQUIRE and my per-file checks kept covering the implementation and skipping the tests beside it. The value is bound to a named local rather than unwrapped inline, which the compiler insisted on: std::ranges::find over the temporary Unwrap() returns yields std::ranges::dangling, the language refusing to let an iterator outlive what it points into. The named form is clearer anyway. The real fix is not in this diff: every changed file, tests included, now goes through one pre-push gate rather than whichever files I thought to check. Signed-off-by: Christian Parpart <christian@parpart.family>
The compile worker, as a new row in the app table. It is not a cache and not a scheduler: it holds no keys, stores nothing, and is given no cache credentials. The object it produces goes back to the client, which stores it. Unlike fastcache-cc it LINKS the FastCache library, deliberately. A worker is a daemon, and "the same daemon capabilities as fastcached" means the same listener, socket, TLS and logging code rather than a second implementation of each. The launcher avoids the library because it is spawned thousands of times per build and its startup is on the critical path; a worker starts once and then lives, so the trade runs the other way. It compiles in the launcher's own protocol sources rather than copying them -- they are the two halves of one wire format, and a second copy of either is how the ends drift apart. WorkerServer is shaped after AdminHttpServer rather than Server, for the reason that governs the whole node: Connection is built around a CacheEngine and a worker has no cache. Taking an IListener& and running its own loop keeps the node clear of the cache stack while still reusing the reactor and socket abstractions. One request per connection, on purpose. A compile occupies a slot for seconds, so a connection that could send a second one would let one client hold a slot indefinitely and the scheduler's slot accounting would stop meaning anything. The concurrency cap is enforced here as well as advertised, and a job over it is REFUSED rather than queued. A worker that accepted more would be fuller than the scheduler believes and slower than it believes, in the same moment; queueing hides the overload from the scheduler that is trying to route around it, and the client has a local compile waiting either way. The cap is checked before the request is read, so an over-capacity client does not make the worker buffer its multi-megabyte payload first. Jobs are served inline rather than detached. A compile is CPU-bound and seconds long, so spawning would only let more of them contend for the same cores -- the opposite of what the cap exists to prevent. The accept loop being busy IS the backpressure. --scheduler and --toolchain are both refused at startup rather than at the first job: a worker missing either would register, or fail to, and then refuse everything, which reaches an operator as "distribution does not work" rather than as a misconfigured node. There is deliberately no default compiler, because a default is how a job ends up running against something nobody chose. --advertise is separate from --bind because the scheduler hands that string to clients verbatim: a worker that advertised an address only it can reach would be leased and then never answer. Signed-off-by: Christian Parpart <christian@parpart.family>
The node has an install() rule, so CPack expects its binary -- but the three packaging jobs build an explicit target list, and it named only the daemon and the launcher. The result was a CMake "file INSTALL cannot find" on a binary nobody had built, in all three packagers at once. The two e2e jobs share that command text and are deliberately left alone: they do not package, and building a target they never invoke would only slow them down. They gain the node when the multi-node fixture needs it. The payload assertions gain it too, in the deb and the rpm. A binary that ships without being asserted is a binary that can silently stop shipping -- which is the same reasoning that put every other path in those lists. The macOS "links nothing outside /usr/lib" check gains it as well, and the node needs that one most: it is the only one of the three that links the whole FastCache library, so it has the most ways to pick up a stray Homebrew dylib and produce a package that refuses to launch on any machine without the build host's prefix. Verified locally that `cmake --install --component Runtime` now stages all three binaries, rather than trusting the target list to be right. Signed-off-by: Christian Parpart <christian@parpart.family>
`readability-qualified-auto` asked for `auto const* const` on the result of std::ranges::find_if over a std::array, and that advice is only correct on libc++, where such an iterator IS a raw pointer. MSVC's STL makes it a class type, so clang-cl refused it outright -- the check's suggestion does not compile on half the platforms this project builds for. Iterating the table and returning the value sidesteps the difference entirely, and reads better than a find-then-dereference anyway. Worth knowing rather than just fixing: a clang-tidy autofix is not automatically portable, and this one is silently standard-library-specific. The other `auto const* const` uses nearby are genuine pointers -- FindOp's descriptor, and string_view::data() -- and are unaffected. Signed-off-by: Christian Parpart <christian@parpart.family>
Notarization rejected the whole archive with three errors against fastcache-compile-node: not signed with a Developer ID certificate, no secure timestamp, no hardened runtime. All three are one cause -- the binary was never signed at all. MacOSSignBinaries globbed `bin/fastcache*` and then filtered with an ALLOW-LIST of names, `/(fastcached|fastcache-cc)$`. So a new executable was picked up by the glob, dropped by the filter, and shipped unsigned. Nothing failed at build time. Nothing failed at package time. The first sign of trouble was Apple's notary service rejecting the archive -- on a job most changes never run, and which needs a signing identity to reach at all. That is the "adding an app is adding a row" idiom quietly not holding: the app table gained a row and this was a second place that had to be edited in step with it. The filter now asks the FILE what it is rather than reading its name, which is what the comment beside it always claimed the filter was for -- excluding the uninstaller, a shell script codesign would reject. A fourth binary needs no edit here. Both branches were exercised locally against a staged tree: the node is now picked up for signing (it previously was not), and shell scripts are skipped with a reason rather than attempted. The signing itself still needs an identity, so CI is the only place the whole path runs. Signed-off-by: Christian Parpart <christian@parpart.family>
The pieces existed and nothing connected them: WorkerRegistry and LeaseTable were built and tested, the handler arms were in, --listen-dispatch parsed -- and main.cpp constructed neither, so a dispatch endpoint refused every verb it was given. This closes that, and the whole chain now runs. Verified end to end against real processes: a daemon with --listen-dispatch reports "distributed execution enabled", a node registers with it, a launcher misses, leases, dispatches, and gets an object back that is BYTE-IDENTICAL to a local compile. That last property is the whole soundness claim. The registry and lease table are built only when a listener asks for the Dispatch role. Without --listen-dispatch nothing carries it, every dispatch verb is refused by the role gate, and the two objects would answer nobody -- so a daemon that only caches does not carry two live maps and a heartbeat sweep it has no use for. They share the engine's clock rather than owning one: that clock is a CachedClock the reactor refreshes once per loop iteration, and two clocks would let a lease expire in the middle of the batch that was about to resolve it. THE SECOND PREPROCESS, WHICH IS NOT AN OPTIMISATION FAILURE. Dispatch reused the key's preprocessed text, and the first real -Werror compile through a worker failed. The key's probe suppresses `#line` markers so no checkout path can reach the key -- and those markers are also what tells a compiler which lines came from a system header. Without them every warning inside libc++ or the CRT resurfaces in the remote compile. Under -pedantic -Werror, which this repository itself builds with, those are errors: every dispatched translation unit would fail and be silently retried locally, so distribution would appear to work while never actually helping. So dispatch preprocesses again, with markers, on a path that was about to spend seconds compiling remotely -- roughly 45 ms against a compile that would not have been dispatched at all if it were cheap. The key's text is unchanged, so no key moves and nothing re-keys. Keeping the markers then broke it a second way, and the fix is the one ccache and distcc already use: under -pedantic the markers THEMSELVES are a GNU extension, so clang reports -Wgnu-line-marker. The driver is now told its input is already preprocessed (-x c++-cpp-output, or cpp-output for a C source, and nothing at all for MSVC, whose /E emits standard #line directives cl accepts as they are). With both fixes, `-std=c++23 -Wall -Wextra -pedantic -Werror` dispatches and returns a byte-identical object. Before them it failed and fell back on every compile. Signed-off-by: Christian Parpart <christian@parpart.family>
christianparpart
changed the title
The feature's soundness claim is that an object compiled on another machine equals the one this machine would have produced. Nothing that runs in one process can check that, so this is a script: a scheduler, a worker and the launcher, three real processes on four allocated ports. Six cases, each covering a failure that is silent rather than loud. The object must be byte-identical to a local compile -- a wrong one is stored under a key other machines then fetch. A dispatched result must still populate the cache, or a fleet recompiles everything forever while looking healthy. A worker registered for a different toolchain must never be chosen. And every refusal path -- no worker, a dead fleet, a scheduler address pointing at the cache listener -- must still end in a successful build, because distribution that can fail a build cannot be switched on by default. Both halves of the fixture were mutation-tested rather than assumed: giving the worker a fingerprint no client presents fails case 1, and pointing case 5 at the dispatch port instead of the cache port fails case 5. A case that passes whatever the code does is worse than no case, which this suite has a recorded lesson about already. The concurrency case deliberately asserts only that four clients beat against a one-slot fleet without hanging or producing a wrong object. It does not assert which of them were refused: that depends on whether the jobs happen to overlap, and a fixture that asserts a race fails for reasons unrelated to the code. The capacity rule itself is pinned deterministically in WorkerRegistry_test against a ManualClock. Registered in src/tests and not beside the launcher, and that is the part worth remembering. src/apps walks its table in order, so at the point src/apps/fastcache-cc is configured, fastcache-compile-node is not a target yet -- a $<TARGET_FILE:> guard on it does not fail there, it silently skips the test forever behind one line in a configure log. Hit while writing this, and now recorded in AGENT.md as a rule rather than as an anecdote. Signed-off-by: Christian Parpart <christian@parpart.family>
SyncRun(server.Run()) never returned, so the only way to stop a worker was to kill it. WorkerServer::Shutdown() already existed for exactly this and nothing called it -- and DaemonControls was already included, unused, which is the shape of an intention that never got wired. Three parts, and the middle one is the whole reason there are three. A signal handler may portably do nothing but set a flag. The accept loop is parked inside Accept() and cannot look at a flag. So a watcher sits between them and closes the listener, which is what unparks the loop -- via the poll timeout WorkerServer::Run already treats as "not a failure" rather than as an error. No drain is needed and that is a property of the design rather than an omission: compiles are served INLINE in the accept loop, so by the time Run() returns nothing is still running. A worker that detached its jobs would need one, and would need to pick a timeout for it. SIGHUP is deliberately left at its default. The daemon reloads its config on it; a worker's toolchain table is what its registration advertised, so re-reading it would leave the scheduler dispatching against a set this worker no longer serves. Nothing to reload is a better answer than a reload that lies. No deregistration is sent either. There is no such verb, and the heartbeat lapsing is the one mechanism that also covers a killed process, a severed network and a crashed host. A second path would give the fleet two ways to believe a worker is alive, only one of them exercised in the failure that matters. Both tests were rewritten single-threaded after a mutation exposed the first cut as defective: a helper thread spinning on a poll counter did not FAIL when the poll-timeout branch was removed, it HUNG, because the counter it waited on never advanced -- and it raced on a plain int besides. The fake listener now drives the stop from inside Accept(), so the same mutation reports `1 >= 3` and CI keeps moving. Signed-off-by: Christian Parpart <christian@parpart.family>
CI reported `dist-compile-e2e ***Timeout 900.10 sec` on all three Linux jobs. The worker never exited, so the fixture's `wait` blocked forever. The stop path added in the previous commit was incomplete in a way macOS hides. POSIX does not unblock a parked `accept()` when another thread closes the listening socket; the only portable wake-up is SO_RCVTIMEO, which makes accept() return periodically. WorkerServer::Run already DOCUMENTS that poll timeout as the mechanism it relies on -- nothing ever supplied one. So installing a SIGTERM handler made the signal non-fatal without making the loop reachable, and `systemctl stop` would have hung until the supervisor escalated to SIGKILL. macOS wakes the accept on close(), which is why it passed here and on macOS-clang-release. SetTimeouts is what the daemon's admin listener already does, for this exact reason, with a comment saying so. Chasing it found a second bug in the same six lines. BlockingListener:: Bind NEVER returns null -- it returns a listener carrying the diagnostic for Accept() to surface -- so `listener == nullptr` was dead code and a bind failure fell straight through. Verified rather than assumed: on a port conflict the worker logged "compile node ready on 127.0.0.1:29973", REGISTERED with the scheduler advertising that port, and exited 0. The scheduler would lease that dead endpoint to clients until the heartbeat lapsed 90s later. It now reports the address and the errno and exits 1. The reporting failure is worth as much as the fix. A 900-second timeout naming nothing is the least useful thing CI can say, so every wait in the fixture is now bounded: stop_and_require_exit fails in 15s naming what it waited for, and the cleanup trap escalates to SIGKILL rather than blocking -- it runs on every exit path, including the failing ones, so an unbounded wait there turns any single failed assertion into a silent suite timeout. Case 7 asserts the property directly: a worker exits on SIGTERM AND logs having stopped gracefully, since a worker killed by the signal also "exits". It needs a real socket because the socket is the mechanism. Verified by making the node ignore SIGTERM: the fixture now fails in 24s instead of hanging. Signed-off-by: Christian Parpart <christian@parpart.family>
The fingerprint's pure half has been in place since the wire landed: digest a banner plus (relative path, content hash) pairs. Nothing gathered those pairs, so `ProbeToolchainFiles` existed only in a doc-comment and dispatch still keys on the compiler's --version banner alone -- which two machines can print identically while resolving different libstdc++ headers, and for DISTRIBUTION that is a silently wrong object rather than the stale path a replay guard can probe. Discovery is a driver-table column because the two families do not spell one question differently, they answer it in different places: a GNU driver PRINTS its search list under -v, while cl has no such switch and reads INCLUDE from the environment. A flags-only column cannot express the second, so the column is a mechanism -- and a third one is an enumerator plus one arm in a switch with no default:, rather than an `if (isMsvc)` at the use site. Both parsers are pure and tested against captured driver output, which matters more than usual here: an Xcode SDK layout, a developer prompt's INCLUDE and a version-suffixed GCC install cannot all exist on whatever machine runs the tests. Four things in the GNU parser are load-bearing, and each is a case: `#include "..." search starts here:` immediately PRECEDES the real marker, so matching a prefix opens the list one line early and collects the build's own -I directories -- making the fingerprint depend on the project. Parsing stops at the terminator, because a driver can echo an inner invocation whose paths belong to a different command line. A `(framework directory)` suffix is stripped and the path KEPT, since on macOS those carry the system headers. And a trailing \r is removed: left on it rides into the digest and into every root prefix test, where it fails silently because a path that does not exist is merely skipped. The walk records paths relative to the root they were found under and spells them with `/` on every host. Both are the same requirement -- two machines holding byte-identical toolchains at different prefixes, or on different platforms, must agree, or distribution is disabled between exactly the machines it exists to connect. Verified against this machine's real clang output, not only the fixture. Wiring it to dispatch needs the cache the header describes; this is the half that can be tested without one. Signed-off-by: Christian Parpart <christian@parpart.family>
…ad it Measured on this machine: the full include-tree walk is 288 MB across 14,600 files and takes 3.65s. The launcher runs once per translation unit, so the cache is not an optimization -- without it the fingerprint would cost an order of magnitude more than the compile it exists to distribute. The validity stamp is DIGESTED rather than stored field by field, so checking it is a string compare and the cache file needs no parser. A format with a parser is a format that can be misparsed, and this one is written and read by short-lived processes racing each other. What the stamp covers is a deliberate trade, and the test suite pins both halves. The compiler's size and mtime catch an upgrade; each search root's mtime catches headers appearing or disappearing. A header edited IN PLACE, changing no directory, is invisible -- accepted, because a system toolchain is installed rather than edited, and the alternative is paying 3.65s per invocation. There is a test that asserts exactly that blind spot rather than leaving it to a comment. Timestamps are read at the filesystem's full resolution, not truncated to seconds. The first cut truncated -- to keep the value stable across a standard-library change -- and that was the wrong call in the wrong direction: a stamp too coarse serves a stale fingerprint, while one too fine merely recomputes something unchanged. A one-second granularity blinds it to everything an installer does within a second, which is precisely when an upgrade happens. My own test caught it by failing. Concurrency is tolerated rather than locked. Sixteen launchers on a cold cache all walk and all write; they write the SAME answer, the write is atomic via temp-plus-rename, and duplicated work once per machine beats a lock protocol between short-lived processes that must never deadlock a build. --print-toolchain-fingerprint exists because switching off the banner takes the fingerprint out of what an operator can derive by hand. It forces a refresh: it answers "why did no worker match", and a cached answer cannot tell a genuine difference from a stale entry. So it is the remedy as well as the diagnosis. StateDirectory and ClassifyCompiler are now exposed rather than copied. Both were private to one translation unit and both were needed by a second; a duplicated platform #if is two places for the location to drift, and a cache written to one path and read from another never hits. The flag-table test now takes arity FROM the table instead of parsing every flag bare -- it caught this flag legitimately, since a value-taking flag parsed alone is a usage error by design. A list of exceptions beside the table would be the second source of truth the table exists to avoid. Verified against real toolchains: Apple clang and Homebrew clang produce different fingerprints, each with its own cache file, and repeated runs are stable. Signed-off-by: Christian Parpart <christian@parpart.family>
This is the change the fingerprint existed for. Dispatch keyed on the compiler's --version line, which two machines can print identically while resolving different libstdc++ headers -- and for distribution that is not a miss, it is an object built by the wrong toolchain, returned under a key other machines then fetch. The CACHE key deliberately keeps the banner. A wrong answer there is a miss the replay guard still backstops; distribution has no such backstop. Two different questions, two identities, and conflating them would re-key every entry on every machine for no benefit. The strong fingerprint is computed inside TryRemoteCompile rather than beside the key's stamp, so it stays on the miss-and-dispatch-configured path. It is a small file read in the steady state and several seconds the first time a machine sees a toolchain; a build that never dispatches must not pay that at all. A node now derives its own fingerprint from a bare --toolchain=<path>, through the identical functions its clients use. That is not convenience: the digest stopped being something a person can read off a terminal, and requiring it to be pasted into a config would make every toolchain update a manual two-step that silently un-registers a worker when someone forgets. <fingerprint>=<compiler> survives as an override, for a compiler this host cannot execute and for pinning a fleet during a repair. Sharing the code is the whole point and is why CompilerBanner moved out of the launcher's main.cpp: a worker that derived its identity through a second implementation would register cleanly, heartbeat happily, and never be matched, with nothing anywhere reporting why. The node's build therefore compiles in the fingerprint group, and main.cpp's now-unused RunCaptureCombined wrapper is gone rather than left as a second path to the same probe. The e2e ASKS the launcher for the fingerprint instead of deriving it. A fixture that recomputed the digest would assert its own reimplementation, and if the two ever disagreed every case would quietly degrade to a local compile and still exit 0 -- passing while testing nothing. It also asserts that the worker's self-computed digest equals the launcher's, which is the failure that is otherwise invisible from both ends. Verified with real processes: launcher and worker independently derive 368649fc..., and case 1 dispatches and returns a byte-identical object. Signed-off-by: Christian Parpart <christian@parpart.family>
Socket activation, which is what a `.socket` unit needs. The supervisor binds and listens before anything starts, so the port answers from boot and a client never races the daemon's startup -- and an idle worker costs nothing, which is the right shape here, since misses on a warm shared cache are bursty and rare. The LISTEN_PID check is the security-relevant part and has its own case. These variables survive fork and exec, so every grandchild of an activated service sees them; adopting on their strength alone means treating whatever the parent left on descriptor 3 as a listening socket -- a log file, a database connection, the read end of a pipe -- and then accepting on it forever. Checking the pid is the only thing that makes "is fd 3 a listener?" answerable, so the parser is pure and every rule around it is a unit test rather than something only a systemd box could reach. Two details that are not tidiness. The variables are cleared even when nothing was adopted, because a process that decided they were not addressed to IT must still not pass them down to a child that would decide differently on a pid that matches by reuse. And the adopted descriptors are marked close-on-exec: systemd deliberately omits that so a service can re-exec itself, but this worker spawns a compiler per job, and a compiler holding the listening socket keeps the port alive after the worker exits -- so the restart cannot bind and blames an address in use that nothing visible is using. The shutdown timeouts are PARAMETERS of adoption rather than something the caller applies afterwards, and that is a lesson rather than a preference: I wrote this with the caller applying them, which silently reintroduced the exact Linux hang fixed two commits ago -- an adopted listener with no receive timeout cannot be woken, so `systemctl stop` waits for SIGKILL. Requiring them to adopt a socket is what stops it arriving a third time. The ready line now describes where the listener came from instead of printing --bind and --port, which were never used in the activated case and named an address the process was not listening on. Verified by acting as systemd: the node adopts fd 3, lsof confirms it holding the port, a client connects, and SIGTERM still stops it cleanly. Signed-off-by: Christian Parpart <christian@parpart.family>
The units, the account, the config file and the maintainer-script hooks for fastcache-compile-node, as rows in the existing asset table rather than new install() calls. The socket is the unit an operator enables and the service carries no [Install] at all. That is what activation means here: the port answers from boot, so a client that leases this worker never races its startup, and an idle worker costs nothing -- which is the right shape for a compile fleet, where misses on a warm shared cache are bursty and rare. Accept=no because one connection carries one compile and the worker's slot accounting assumes a single instance; Accept=yes would spawn a process per connection and make the advertised slot count a fiction. The hardening block is deliberately NOT a copy of fastcached's, and the difference is the point of the service: fastcached executes nothing, while this worker's whole job is to run a compiler. MemoryDenyWriteExecute is absent where the daemon sets it, because compilers legitimately map writable-then-executable memory -- LTO plugins, any JIT the driver loads -- and enabling it would reject valid toolchains with an error pointing at the compiler instead of at this file. Every relaxation is named with its reason so a future tightening does not silently break compiles. A separate service account, and not fastcached's. A worker runs a compiler on input that arrived over the network while fastcached owns the cache storage, and the trust model gives a worker no cache credentials at all -- sharing an account would undo that at the filesystem level, so a compromised compile would run as the identity that can rewrite every cached object. The scheduler address and toolchains come from an EnvironmentFile because they are site-specific with no defensible default. Marked `config`, so dpkg registers a conffile and rpm %config(noreplace): it is the file that makes the worker work at all, and an upgrade that reset it would un-register the worker silently. Removal needed both scripts touched, and the ORDER matters: the socket is stopped and disabled before the service, because a socket left holding the port stays ready to activate a binary the erase is about to delete -- and the failure then names a missing executable rather than an incomplete removal. CI asserts it on real systemd rather than by inspecting the payload alone: enable the socket, confirm nothing is running yet, connect, confirm the service activated AND adopted the socket rather than binding its own, then stop it and require Result=success. That last assertion is the one that matters -- a worker whose adopted listener has no receive timeout hangs until systemd escalates to SIGKILL, and that bug has already shipped once here on the ordinary bind path. Signed-off-by: Christian Parpart <christian@parpart.family>
CI's live-systemd step failed with "the worker did not adopt the activated socket", and the journal it printed held exactly one line: systemd's own "Started". The worker had produced no output at all. Two things were wrong and only one of them was the test. Type=simple is reported active the moment systemd forks, while this worker still has to walk its toolchain's include tree -- seconds of real work -- before it logs anything. The step polled is-active, which answers "has it started", and then read the journal 0.3s later. It now waits for the worker's own readiness line instead. Waiting on a supervisor's idea of "active" is the same mistake as waiting on a fixed sleep, and it fails the same way: intermittently, and worse on a slower machine. The ordering was the real finding. Adoption is decided in microseconds and the fingerprint takes seconds, so doing the expensive thing first meant a misconfigured unit -- one handing over two sockets, say -- spent several seconds computing a fingerprint it was about to throw away, and an operator watching a worker start saw nothing at all during the part where something could still go wrong. Cheap and fallible first: the log now reads in the order things happened. Verified by acting as systemd again: adopted, then computing, then serving, then ready. Signed-off-by: Christian Parpart <christian@parpart.family>
…ature --toolchain is documented as repeatable and the job runner honours all of them, but registration announced `toolchains.begin()->first` -- ONE, and which one depended on where two hex digests happened to sort in a map. A worker configured with g++ and clang++ therefore served exactly one of them; the scheduler never heard about the other, so every job for it fell back to a local compile with nothing anywhere reporting a reason. REGISTER carries one fingerprint, so the fix is one registrar per toolchain. That looks like advertising N times this machine's capacity and is not: every entry heartbeats the same machine-wide in-flight count, so once the worker is busy all of its entries report themselves busy together and the scheduler stops picking any of them. The pool behaves as one because the number reported describes the machine, not the entry. The loop counts rather than short-circuits -- one toolchain the scheduler refuses must not stop the others being announced -- and says "N of M", so a partial registration is visible rather than being reported as success. Found while writing the docs, which is the other half of this commit. Distributed compilation had no documentation at all, which makes it unusable regardless of how well it works: a tools page covering the three-process layout, --advertise (the flag whose misconfiguration presents as "every machine but one falls back"), what a worker refuses to do and why, socket activation, and the limitations worth knowing before adopting it -- preprocessing does not distribute, -g embeds the worker's scratch path, remote diagnostics are not shown. The protocol page gains the four dispatch verbs, the seven new error codes and the codec envelope. Every opcode, error value and payload shape in it was checked against the wire header rather than written from memory; the REGISTER line was wrong on the first pass, which is how the defect above surfaced. ResolveToolchains is extracted from main because the addition pushed it over the cognitive-complexity limit -- and it reads better alone anyway, being the one part of startup that runs an external process and can take seconds. Signed-off-by: Christian Parpart <christian@parpart.family>
A guide covering what the reference page does not: the architecture, how the thing actually works, and how to set it up. Three roles and why two of them are programs you already have; why the scheduler IS the cache daemon (it is the only design that can know which keys are being compiled right now, which is what suppresses duplicate work); why the client stores the result and never the worker; and the five steps of a dispatched compile, including the two that surprise people -- that preprocessing cannot be distributed, and that the text sent to a worker is deliberately not the text the cache key hashed. Also the operational half: how to confirm it is working, what each metric means, and why no-worker and no-capacity are counted apart (one says the fleet is misconfigured, the other that it is too small, and summing them hides the first behind the second exactly when a fleet is busy). Ports in install.md, the two new layers in the architecture map. Writing it found a real gap. The guide showed `roles: [dispatch]` in YAML, which did not exist -- dispatch could only be enabled with --listen-dispatch on the command line. The packaged unit runs `fastcached --config=...`, so an operator using the shipped service could not run a scheduler at all without overriding ExecStart. YamlReader now parses `roles`, off a name table in Config.hpp so a config file's spelling is necessarily one the parser accepts. Three properties of that parsing have their own cases. The default stays Cache alone -- a listener that silently also served dispatch would expose the compiler-running surface on every port every existing deployment already has open. `roles:` REPLACES the default rather than adding to it, or an endpoint an operator was isolating still serves the cache. And an unknown or empty role is refused rather than ignored, because a typo that left a listener on its default would be a scheduler answering no dispatch verb, with a config file that looks correct. The dispatch metrics are wired rather than merely declared: SessionContext gained the sink the connection counters already use, guarded once in a helper rather than at four call sites. Counters nothing increments would have been the "a version with no work to do" mistake in another form. Verified against a real daemon: the documented YAML starts a scheduler and a worker registers through it. Signed-off-by: Christian Parpart <christian@parpart.family>
**A socket-activated worker advertised an address it was not listening
on.** With activation and no --advertise, the fallback is
`{--bind}:{--port}` -- but the socket unit owns the port and never tells
this process which one, so those config values describe nothing, and
0.0.0.0 is not an address a remote client can dial anyway.
The shape of that failure is the worst this system has: registration
SUCCEEDS, the worker heartbeats happily, the scheduler leases the
endpoint out, and every client fails to connect and compiles locally.
Nothing reports an error and the fleet looks healthy from both ends.
--advertise is now required under activation, refused at startup where it
can be explained -- and refused BEFORE the toolchain walk, so it costs
0s rather than the several seconds a fingerprint takes. Same
cheap-and-fallible-first ordering the adoption check already got; it is
worth applying consistently rather than once.
**The fingerprint cache leaked a temp file on every failed write.** The
write-then-rename is atomic, but the early return on a stream failure
left the temp behind, in a directory nothing sweeps. A machine with a
full disk or a permissions problem accumulates them indefinitely while
silently recomputing the fingerprint on every invocation. Every path that
does not end in a rename now removes it.
**InheritedListener used _putenv_s and unsetenv without <cstdlib>.** It
compiles here only because another header happens to pull it in; that is
a Windows build waiting to break on an unrelated include change.
Also swept the new sources for the patterns this codebase cares about --
unchecked optional access, NOLINT, missing Doxygen on public functions,
raw owning pointers -- and found none.
Signed-off-by: Christian Parpart <christian@parpart.family>
Member
Author
Handoff — distributed compilation (PR #78)Branch The goalAll phases of the approved design implemented, with comprehensive test coverage, State39 commits, 88 files, ~11k insertions. 1476 tests pass locally; Distributed compilation is verified working end to end on POSIX: a launcher
What is still missing1. The Windows byte-identity question — DATA PENDING, may not be a bugMSVC dispatch works. The worker's object differs from a locally compiled one by Established: GNU produces byte-identical objects from preprocessed-vs-original
The assertion was deliberately not relaxed to make CI green, because the 2.
|
The first control answered, and shifted the question rather than closing
it:
reference (original source): 63124 bytes, 17C3B652...
worker: 63112 bytes, 3AEBDF47...
control (local, preprocessed): 63112 bytes, ACA056D5...
The twelve-byte gap is the INPUT -- preprocessed versus original -- and
both preprocessed compiles are 63112. But the two preprocessed compiles
are the same SIZE with different BYTES, which means something
equal-length is embedded. A path would not be equal-length; a timestamp
would, and MSVC stamps TimeDateStamp into the COFF header unless
/Brepro is given.
If that is what this is, then no two MSVC compiles are ever identical --
including two local ones -- and byte-identity is the wrong assertion for
this platform rather than a violated one. The first control could not
tell those apart because it never compared a local compile against
itself.
So: compile the identical input twice, same directory, seconds apart.
Matching means the driver is reproducible and the worker really is
leaking its environment into the object, which is a product bug.
Differing means the assertion is what has to change.
Still not relaxed. Two more lines of evidence cost one CI run; guessing
between "fix the worker" and "restate the assertion" would cost more than
that and could be wrong.
Signed-off-by: Christian Parpart <christian@parpart.family>
Member
Author
Update — the byte-identity control has reportedTwo facts, only the second still open:
If that is the cause, no two MSVC compiles are ever byte-identical — including two local ones — and the assertion is unachievable rather than violated. A second control (
On a Windows machine this settles in one command, no CI needed: The assertion has deliberately not been relaxed to make CI green — the answer decides which of two quite different pieces of work is correct. |
…they do The remaining Windows question was "is this a timestamp or a leak", and answering it looked like it needed a Windows machine to run `cl` twice. It does not: the differing byte OFFSETS say which, and a CI log carries them. A COFF header is Machine(2) NumberOfSections(2) TimeDateStamp(4), so bytes 4-7 are the clock. Differences confined to those mean the driver stamps the time into every object and byte-identity is unachievable rather than violated -- the assertion is then what has to change, and /Brepro becomes the design question. Differences scattered through the file mean something else is leaking: a path, a symbol, an environment string. Verified against synthetic COFF-shaped input rather than by waiting for a run: a timestamp-only difference is recognised and named, and a scattered one is deliberately NOT misattributed to it. This does not settle the question -- the next CI run does -- but it stops the answer from requiring a platform I do not have. Signed-off-by: Christian Parpart <christian@parpart.family>
…dentity The control settled it. Two identical local compiles -- same input, same directory, seconds apart -- differ by ~70 bytes on an MSVC driver, at offsets around 12122 and 12206+, nowhere near the COFF header. So it is not a timestamp, and it is not the worker: this driver does not produce reproducible objects at all. Byte-identity was unachievable here, not violated, and GNU being identical either way is why the POSIX fixture never had to find that out. "Give up on Windows" would be the wrong conclusion, though. The claim worth making is still "the worker's object is what this machine would have produced" -- so measure the driver's own nondeterminism and require the worker to stay inside it. Compile the same preprocessed input locally TWICE, take the set of offsets at which those two differ, and demand the worker's object differ from a local one at no OTHER offset. A size change is never noise and always fails. That keeps the assertion's full strength: anything the worker introduces that a local rebuild does not is still caught, and named by offset. Verified the comparison logic on synthetic objects rather than by waiting for a run -- within-noise accepted, one extra offset rejected and reported, a size change rejected outright. One error of mine caught while doing it: the edit that introduced this had swallowed the dispatch step, so the comparison would have run against an object nothing produced. Restored, and the order re-checked -- dispatch, control compiles, comparison, then the cache HIT. Signed-off-by: Christian Parpart <christian@parpart.family>
Member
Author
Verdict — and the assertion is what changes, not the productThe reproducibility control settled it: Two identical local compiles — same input, same directory, seconds apart — differ by ~70 bytes. Nowhere near the COFF header, so not a timestamp, and not the worker. This driver does not produce reproducible objects at all. Byte-identity was unachievable on Windows, not violated. GNU is byte-identical either way, which is why the POSIX fixture never had to discover this. "Give up on Windows" would be the wrong conclusion. The claim still worth making is the worker's object is what this machine would have produced — so
Anything the worker introduces that a local rebuild does not is still caught, and reported by offset. Verified on synthetic objects rather than by waiting for a run: within-noise accepted, one extra offset rejected and named, a size change rejected outright. Resuming on Windows — first commandsIf it passes, the only remaining items are Worth knowing: do not copy the macOS |
Member
Author
Root cause:
|
| job | result |
|---|---|
Windows-cl-release |
same size, 117 differing offsets (contiguous run ~12137–12157) |
Windows-clangcl-release |
different size |
Same driver, same input. The only difference between the runners is the build-directory name — cl-release (10 chars) vs clangcl-release (15). The control's source path contains it; the worker's (%TEMP%\fastcache-compile-node\job-N\tu.cpp) does not.
So cl embeds the source path in the object even without /Zi. Object size tracks path length, which is why one job saw a size difference and the other happened to land on equal sizes with a ~21-byte contiguous run of differing text. That is a path string, not a fixed-length field.
Consequences
-
The assertion in
238d97ecannot pass. It compares bytes "modulo the driver's nondeterminism", but this difference is not nondeterminism — it is deterministic and path-dependent. The worker's source path necessarily differs from any local one, so a byte-comparison against a locally compiled object can never succeed on MSVC. -
I mis-generalised earlier. I called MSVC "not reproducible" from a 70-byte reading that was clang-cl.
cl's real nondeterminism is 9 bytes. That error sent238d97ein a direction that cannot work; it should be replaced rather than tuned. -
Correctness is unaffected. Same compiler, same flags, same preprocessed input — only an embedded path string differs. This is the MSVC analogue of the
-gscratch-path caveat already documented for GNU.
Suggested resolution
Replace the Windows byte-comparison with what is actually assertable: the compile dispatched, produced an object, the build succeeded, and the result is served from the cache on the next compile. Then document in docs/tools/fastcache-compile-node.md that a dispatched object on MSVC embeds the worker's scratch path, so it is not byte-identical to a local one.
There may also be a genuine improvement: if the worker invoked cl with CWD set to its scratch directory and a bare relative source name, the embedded path would be just tu.cpp — short and machine-independent. That would not make it match a local compile, but it would stop dispatched objects carrying a worker's temp path into the shared cache. Worth measuring on a machine where the bytes can be read directly.
The POSIX fixture is unaffected and keeps its strict byte-identity assertion: GCC/clang embed nothing path-dependent without -g (verified: identical objects from preprocessed vs original input, and across differing source filenames).
…rtable `cl` embeds the SOURCE PATH in the object even without /Zi. The worker compiles from its own scratch directory, so its path necessarily differs from any local one and the objects differ deterministically -- by a run of path text, and in SIZE when the paths are different lengths. Both Windows jobs proved it by failing DIFFERENTLY on the same driver and the same input: one saw equal sizes with 117 differing offsets, the other a size mismatch. The only variable between those runners is the build directory name -- `cl-release` versus `clangcl-release`, five characters -- which the local path contains and the worker's %TEMP% path does not. The previous assertion was wrong twice over, and both are mine. It compared bytes "modulo the driver's nondeterminism", calibrated on a 70-byte figure that came from clang-cl; `cl`'s own nondeterminism is 9 bytes. And the difference is not nondeterminism at all -- it is deterministic and path-dependent, so no tolerance for jitter could ever have accommodated it. That approach could not be tuned into working. What is assertable, and what this checks: the compile was dispatched, the worker produced an object, the build succeeded, the object is a plausible size for this translation unit, and it is served from the cache next time. The size bound is loose enough for a path and far too tight for a real fault -- a wrong optimisation level, a truncated transfer or the wrong toolchain each move an object by far more than one percent, against a few hundred bytes of path in sixty-odd kilobytes. The isolation case gets the same treatment for a different reason: its two objects DO share a source path, but `cl`'s nine bytes of nondeterminism would make a strict comparison flake rather than fail. The POSIX fixture keeps strict byte-identity and should: GCC and clang embed nothing path-dependent without -g, verified here including across differing source filenames. The asymmetry is now documented in docs/tools/fastcache-compile-node.md as a real limitation -- it affects debugging, and wants /PDBALTPATH the way GNU wants -fdebug-prefix-map. Signed-off-by: Christian Parpart <christian@parpart.family>
Member
Author
CI is green — 19 pass, 2 skipping, 0 failing (
|
| assertion | why | |
|---|---|---|
| POSIX | the worker's object is byte-identical to a local compile | GCC/clang embed nothing path-dependent without -g — verified, including across differing source filenames |
| Windows | the object is a plausible compile (right compiler, right flags, sane size) plus the full dispatch-and-cache chain | cl embeds the source path even without /Zi, so an object compiled from the worker's scratch directory can never match one compiled from yours |
That MSVC behaviour is a real user-facing limitation and is written up in docs/tools/fastcache-compile-node.md: a dispatched object carries the worker's scratch path, which affects debugging and wants /PDBALTPATH the way GNU wants -fdebug-prefix-map.
Remaining before merge
--install-servicefor the node on Windows and macOS. Linux ships socket-activated units, verified on real systemd.MakeWindowsServiceHost(name)is already fully generic, so only the registration half needs theServiceSpecseam — do it additively sofastcached's working path cannot regress./code-review. A self-review found three defects and CI found five more that only execution could reveal; on a branch of this size it should be run before merge.- Filed separately: MSVC's cache-key compiler identity is just
"cl", so two toolsets can share a key for a header-free TU.
Adding compression to the launcher linked the CPM-built zstd and lz4 into
it, and those take CMake's default MSVC runtime -- the DLL one -- while
the launcher deliberately links the static CRT, so that the binary
FASTCACHE_AUTO_INSTALL fetches needs no redistributable.
In Release nothing breaks: the CRT entry points zstd calls resolve out of
whichever CRT the exe links. In Debug they do not, because `assert()`
references `__imp__wassert`, which exists only in the DLL CRT:
zstd_static.lib(zstd_compress.c.obj) : error LNK2001:
unresolved external symbol __imp__wassert
So `cmake --preset cl-debug` and `clangcl-debug` -- both documented in
AGENT.md as the Windows entry points -- have not built the launcher since
that commit. Windows CI builds Release only, so every job stayed green
while no Windows developer could build Debug at all.
The obvious fix is unavailable: the FastCache library links the SAME zstd
static libraries and the daemon's vcpkg dependencies are dynamic-CRT
builds, so one setting cannot satisfy both targets. The Debug launcher is
a developer artefact and never shipped, so it takes the runtime its
dependencies were built with; the artefact that IS shipped keeps the
static CRT it has always had.
Signed-off-by: Christian Parpart <christian@parpart.family>
…name A worker writes its own scratch file and MSVC reads the language off that file's extension -- and the worker calls every file `tu.cpp`. Nothing stated the language, so a dispatched `.c` translation unit came back compiled as C++. `DriverSpec::preprocessedInputFlags` was EMPTY for both MSVC rows on the reasoning that `/E` emits standard `#line` which `cl` accepts in an ordinary source file, so there is nothing to tell it. True about the MARKERS; it left the LANGUAGE unsaid. Where C is not valid C++ the remote compile fails and is retried locally, so C silently never distributes -- the 100%-fallback-with-a-green-build shape this branch has already hit twice. Where it is, the object comes back with C++ mangling and the client STOREs it under the C key, for every machine to fetch. `/TC` and `/TP` are the `-x` spelling MSVC does have. Measured: `/TP` is a byte-for-byte no-op on a C++ translation unit, so no C++ key or object moves. The column is now a per-language TABLE rather than one span, and `RemoteCompileArgs` reads the row instead of naming `GnuPreprocessedC` inline inside a driver-generic branch, which is how a GNU constant came to be reachable on every family. Chasing that turned up that the EXTENSION is the last of three answers, not the first: - A `++` driver compiles everything as C++ -- "g++ treats .c, .h and .i files as C++ source files", in as many words -- so taking the language off the extension tells a worker to compile as C what this machine compiles as C++. That is a wrong object, not a failed one. `gcc` and `g++` are one Flavor, so it is a second column on the NAME table, walked by the same rules so `g++-14` and `clang++.exe` are recognised here exactly as ClassifyCompiler recognises them. - A build may state the language itself, and CMake does: `set_source_files_properties(x.c PROPERTIES LANGUAGE CXX)` emits `/TP` or `-x c++`. The preprocessed-input flags are appended LAST so they win, which is right when nothing else spoke and a silent override when something did. Such a command line is refused rather than reconciled: "this source is C++" and "this text is preprocessed C++" are different statements and the substitution cannot be made blindly. `/Tc` and `/Tp` are refused by the same row for a second reason -- they name a FILE, and with a bare file name they carry no separator, so CouldNameAFile lets them past to a worker that has no such file. - `.C` is C++ to a GNU driver and C to an MSVC one, and `.M` likewise, so the extension does not answer the question at all. Both are excluded by a case-SENSITIVE row ahead of the case-insensitive lookup that exists so a `.CPP` on Windows still dispatches. Once the classifier exists, C++ MODULES are the same question with a different answer, and they were being handled by accident. A module interface unit writes a BMI beside its object, and what a hit reproduces is the object and the dependency record -- so replaying one leaves the BMI missing, which fails loudly, or left over from a previous build, which does not. The module extensions simply were not in `IsSourceSuffix`, so such a line fell through as "no source file found" and was passed through IN SILENCE, indistinguishable from a broken cache; and adding `.ixx` there -- the obvious "support modules" change -- would have turned that silence into a silent wrong build. They are now recognised as their own language and refused BY NAME with the reason said out loud, and the same reason refuses an ordinary source promoted by a flag (`cl /interface`, `-fmodule-output`, `--precompile`) or a compile that writes a precompiled header (`/Yc`), off a table. Both gates read one accessor, because caching is the wider of the two: a line that is not cacheable never reaches dispatch. RemoteCompileArgs now returns the REASON it refused rather than nullopt. Every refusal here ends in a local compile, so "distribution stopped helping" is otherwise a whole investigation where the answer is one line; the launcher prints it under FASTCACHE_VERBOSE like every other decline, including the module one, which an operator would otherwise read as the cache having quietly stopped working. Signed-off-by: Christian Parpart <christian@parpart.family>
`DispatchRequest::sourceName` was set by the client, documented as "base
name the worker should give its temp file", and never encoded: the wire
had no such field and `WorkerProtocol` passed `{}`, so the worker called
every translation unit `tu.cpp` and `SafeSourceExtension` could only ever
return `.cpp`.
A compiler records the name of the file it was handed -- clang-cl and gcc
in the COFF/ELF `.file` symbol, MSVC in its compiland record -- so a
worker that invents a name produces an object differing from a locally
compiled one in that name and nothing else. Measured on clang-cl: seven
bytes, and byte-identical once the names agree. That is the difference
between a dispatched build and a local build producing the same artefact
and producing merely equivalent ones.
Only the BASE NAME travels. The worker has no use for the client's
directory and no business learning where a checkout lives.
`Op::Compile` gains a sixth field. No version bump: the dispatch verbs are
new in this unreleased branch, so there is no older peer, and `fieldCount`
IS the arity contract. A mismatched pair still fails safely -- SplitFields
refuses, the worker declines the job, the client compiles locally.
`SafeSourceExtension` becomes `SafeSourceName`, and its test stops being
theoretical: `../../../etc/passwd.cpp` can now arrive over a socket and
become a path. It is reduced to one component (split on both separators
AND a colon, so a drive-relative `C:x` cannot survive), then to an
allow-listed stem with no leading dot and a bounded length, then to an
extension from the same fixed table -- and never to a Windows reserved
device name, because `CON.cpp` on a Windows worker is the console and the
translation unit would be written to a terminal. Anything failing any of
those is compiled as `tu.cpp` rather than refused: it is a cosmetic input,
and refusing over one would cost a compile to gain nothing.
The name no longer decides anything but itself. Every driver family is now
told the language explicitly, so a name the worker had to invent cannot
choose how the text is compiled -- which is what the previous commit
closed.
Signed-off-by: Christian Parpart <christian@parpart.family>
The answer the last three commits were asking CI for, obtained on a
Windows machine instead: `cl` IS reproducible, and the control that said
otherwise was wrong.
It compiled the second object to `tu2.o` and compared it against `tu.o`.
MSVC records the ABSOLUTE PATH OF THE OBJECT FILE inside every object, in
`.debug$S`, with no debug flag asked for -- so two compiles of one input
to two names can never match, and the control reported a perfectly
reproducible driver as non-reproducible, in a CI log, as the answer to the
question this fixture existed to settle. Compiling twice to the SAME path
matches byte for byte.
Measured, and this is what the assertion becomes:
- Both MSVC drivers stamp the CLOCK into the COFF header. Two compiles
of one file to one path two seconds apart differ in exactly byte 4,
and only `/Brepro` suppresses it -- a flag no build passes, so a
fixture passing it would assert something about a command line nobody
uses.
- `cl` also records the object's absolute path (`.debug$S`) and hashes
the file it opened (`.chks64`). A worker compiles its own scratch file
to its own scratch path, so neither can ever match.
- Everything carrying code or data does: `.text$mn`, `.rdata`, `.xdata`,
`.pdata`, `.drectve`, `.data$r` and `.bss` are byte-identical between
a reference compile of the original source and a worker-shaped compile
of `/E` text elsewhere.
- clang-cl records only the source's BASE NAME, which the worker is now
told, so its objects differ by the clock alone -- an EMPTY row, and the
strongest claim of the three.
So each driver asserts the strongest property it can carry, off a table,
and a difference anywhere else fails. The COFF header is compared field by
field rather than skipped, because an object built for another
architecture differs THERE and nowhere a section walk would look; the
symbol-table pointer and the timestamp are excused by name and for stated
reasons, not by skipping the header.
Two more things this run turned up, both of which had never been reached
because case 1 threw first: every reference is now compiled to the object
path the LAUNCHER will write and then moved aside, since otherwise the
comparison carries a difference that has nothing to do with distribution
(the old case 3 compared `reference.obj` against `u.obj`, so it could
never have passed either); and there is a new case dispatching a C
translation unit, which is what would notice `/TC` being dropped again.
`-SelfTest` drives the comparison against synthetic COFF objects and needs
no daemon, worker or compiler -- registered as a non-smoke ctest case,
because a fixture's own logic is the one thing nothing else tests, and
this one's cost five CI round trips to find out.
Verified on this machine against both drivers: cl and clang-cl, all four
cases, green.
Signed-off-by: Christian Parpart <christian@parpart.family>
The rule the previous commits added is a client-side decision applied by a compiler in another process, and visible only in the object that comes back -- so a unit test pins the decision and this pins the consequence. A `.c` source compiled by this project's C++ driver is C++, because "g++ treats .c, .h and .i files as C++ source files". Telling a worker `-x cpp-output` for it would have the worker compile as C what this machine compiles as C++: a wrong object, stored under the key and served to everybody, not a failed one that falls back. The case compiles a `.c` translation unit with external linkage, so if the language ever slips the symbol names themselves are mangled and the comparison cannot miss it. Signed-off-by: Christian Parpart <christian@parpart.family>
AGENT.md's distributed-compilation rules said MSVC is told nothing about its preprocessed input "because /E emits standard #line". That was the defect, so it is replaced by what is now true and by the three answers the language question actually has -- an explicit flag, the driver's own default, then the extension -- plus why a module interface unit is refused by both gates rather than falling through in silence. The byte-identity claim gains its Windows spelling, with the measurements behind it: both drivers stamp the clock, `cl` records the object's own absolute path, and everything carrying code or data still matches. Also why the comparison has a self-test: the previous control compared two objects with different names and misread a reproducible driver. For operators: the limits section of the distributed-compilation guide now says that a dispatched object is not byte-identical to a local one on Windows and what to compare instead, and which compiles are never distributed by design. The worker page states that a client chooses what its translation unit is CALLED and nothing else, and what that name is reduced to before it becomes a path. And the README lists `fastcache-compile-node`, which shipped, is documented, has an installed systemd unit and a CI job -- and was invisible to anyone who read only the front page. Signed-off-by: Christian Parpart <christian@parpart.family>
`SafeSourceName` grew an AsciiLower of its own, which is a second definition of a rule this repository already keeps in one place -- and keeps there for a reason it states at length: `std::tolower` is locale-dependent, so under a Turkish locale `LPT1` would be reserved on one worker and allowed on the next. `PathCanon::AsciiLower` is that place, it is constexpr and header-only, so using it costs nothing and removes the copy. Signed-off-by: Christian Parpart <christian@parpart.family>
The two commits below this one relaxed the Windows assertion to a
plausible SIZE, on the strength of a control that reported `cl` as
non-reproducible. That control was wrong, and this settles it on a
Windows machine rather than from a CI log:
cl, same input, SAME /Fo path, twice -> byte-identical
cl, same input, tu.o vs tu2.o -> ~70 bytes differ
strings in the object -> the absolute path of
the OBJECT FILE
The control compiled its second object to `tu2.o` and compared it against
`tu.o`. MSVC records the object's own absolute path in `.debug$S` with no
debug flag asked for, so those two can never match -- and the ~70-byte
figure that became "the driver's noise floor" was that record, not
nondeterminism. `cl` is reproducible.
What IS true, and measured on both drivers:
- both stamp the clock into the COFF header (byte 4, two compiles two
seconds apart; only /Brepro suppresses it);
- `cl` records the object's absolute path and hashes the source file it
opened into `.chks64`;
- clang-cl records only the source's BASE NAME, which the worker is now
told, so its objects differ by the clock alone;
- every section carrying code or data is byte-identical between a
reference compile of the original source and a worker-shaped compile
of /E text elsewhere.
So the assertion is restored to a real one -- section by section, against
a per-driver table of what may differ -- rather than left at a size
tolerance that a wrong optimisation level or a subtly different header set
would pass. Verified locally on cl and clang-cl, all four cases green.
The debugging half of the note stands and is kept: a dispatched object
carries the worker's paths, which is what /PDBALTPATH is for.
Signed-off-by: Christian Parpart <christian@parpart.family>
This was referenced Aug 22, 2026
Member
Author
`ProducesSideArtefact` matched with `MatchesFlag`, which only recognises a fused value for a flag the PATH-VALUE table knows takes one -- and none of these are in that table. So a bare `/Yc` was caught and `/Yc"pch.h"` was not, `-fmodule-output` was and `-fmodule-output=a.pcm` was not: the rule caught the shape nobody writes and missed the one everybody does. A prefix test instead, which is what `IsLanguageSelector` already had to do for `-x` for exactly the same reason. The new cases spell every flag the fused way deliberately, and `/Yu` is there as the negative: it USES a precompiled header rather than writing one, so it must stay cacheable. Signed-off-by: Christian Parpart <christian@parpart.family>
Three things CI caught that a Windows build cannot: `Wire::CompileRequest` gained a field, and every designated-initializer site that does not list it is an error under `-Wmissing-field-initializers` with `-Werror` -- which MSVC does not diagnose at all, so the local run was green. Five sites, two of which had not been reached by the failing compile and would have cost a second round trip. The wire test now asserts the new field round-trips, rather than merely naming it to satisfy the compiler. And clang-tidy's `readability-use-anyofallof` on the two scans added for the language selectors and the side-artefact flags. Both are predicate scans over a table and read better as `std::ranges::any_of` anyway, which is what the project's own guidelines ask for. Signed-off-by: Christian Parpart <christian@parpart.family>
Found by asking the comparison to reject objects that are wrong on purpose: a different optimisation level, a different toolchain, a truncated file. Four of five were rejected. The truncated one was not. The reason is worth keeping. MSVC writes the symbol table LAST, so cutting a tenth off an object leaves every section header and every byte of section data intact -- and a comparison that walks sections sees two identical objects. A truncated transfer is one of the very few faults distribution can actually introduce, so accepting one is most of what the assertion was for. COFF states its own end: the string table opens with a four-byte size that includes those four bytes, and it is the last thing in the file. If that number and the bytes present disagree, the object is damaged whatever its sections say. `Get-CoffTail` checks exactly that, on both sides, under every rule -- "the file is as large as it claims" is not a property a driver gets to opt out of. Whether the tail's CONTENT may differ is a per-driver row, measured like the rest: clang-cl's symbol and string tables are byte-identical between a local compile and a worker-shaped one, `cl`'s are not (same length, different bytes). So the strict standard compares them and the loose one does not, and the driver table now carries both rules rather than a bare section list. The self-test grows the cases that would have caught this: a differing tail under each standard, and a truncation. Verified against real objects too -- the five-way rejection check now passes in full, and both drivers still pass the fixture end to end. Signed-off-by: Christian Parpart <christian@parpart.family>
Three lessons rather than two, and the new one was a hole in the comparison this branch had just written: walking sections accepts a TRUNCATED object, because MSVC writes the symbol table last and a cut-off file keeps every section intact. Also states which header fields are excused and why, and that the self-test plus a deliberate rejection check are what found the defects in the fixture's own logic -- which is the point of having them. Signed-off-by: Christian Parpart <christian@parpart.family>
Two independent things now stop a dispatched `.c` coming back compiled as
C++ -- the client states the language (`/TC`), and the worker names its
scratch file what the client called it, whose extension MSVC reads. A case
that cannot tell those apart should not be commented as though it pins
either one, so this establishes it by experiment instead of by reasoning:
- both removed -> the C case fails (the remote compile of
C-as-C++ fails outright, nothing dispatched)
- only the name unsent -> the C case still passes; `/TC` carries it
- only `/TC` removed -> the CASE 1 leg on clang-cl fails, because
its symbol table then records a name this
machine never compiled
So the two mechanisms are guarded by two different legs, and clang-cl's
empty may-differ row turns out to be load-bearing rather than tidy: it is
what notices the wire field no longer travelling. Both are stated where a
reader will meet them.
Worth recording that the first run of this experiment reported the fixture
as PASSING with `/TC` reintroduced as a defect -- which is what prompted
the other two runs, and would otherwise have left a case in the tree that
reads stronger than it is.
Signed-off-by: Christian Parpart <christian@parpart.family>
The chain is client sends a name -> worker creates a file with it -> compiler records it. The middle and the end were pinned (CompileJob_test for the sanitizing and the naming, CompileCacheWire_test for the round trip) and the beginning was not: nothing asserted that Dispatch sends the BASE NAME rather than the path it was handed. Two cases, one per separator style, because the client's path comes from whatever the build system wrote and the worker must learn nothing about where a checkout lives. Signed-off-by: Christian Parpart <christian@parpart.family>
`modernize-raw-string-literal` on the escaped path in the new dispatch case. A raw string is easier to read here anyway -- the point of that literal is that it is a Windows path, which doubled backslashes obscure. Signed-off-by: Christian Parpart <christian@parpart.family>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Distributed compilation: a compile-cache miss is dispatched to a registered worker node instead of being compiled locally. Every phase of the approved design is complete except the worker's daemon shell: the include-tree fingerprint, socket activation and packaging, dispatch metrics, the multi-node end-to-end fixtures on both platforms, and the docs.
The soundness claim, verified against real processes rather than fakes — a daemon, a worker and a launcher, under this repository's own
-std=c++23 -Wall -Wextra -pedantic -Werror:That check is still a hand-run one; promoting it into a permanent multi-node fixture is phase 5. 114 new
TEST_CASEs cover the parts that can be tested without a socket, which is most of them.Which endpoint serves this is an operator decision
BindConfiggains aListenerRolemask defaulting toCachealone;SessionContextcarries the accepting listener's mask, and the server loop copies it per bind in all three platform paths.The compile-cache surface may reasonably be reachable across a build LAN. The surface that causes a compiler to run on someone else's machine must be something switched on deliberately, on an endpoint that can be firewalled or TLS-required separately. A dispatch verb arriving on a cache-only listener is refused with a typed reply rather than a closed connection — which is what lets that endpoint move without every client needing to know.
The two scheduling rules that carry the weight
Duplicate-work suppression
LeaseTableauthorizes jobs and suppresses duplicates: when sixty parallel clients miss the same key after a header change — the ordinary shape of a miss on a shared cache, not an exotic one — only the first is dispatched and the rest compile locally. This is something neither distcc nor sccache-dist can do, because neither is also the cache.Leases expire because a client can die between taking one and sending the job (
Ctrl-Con a build is the common case, not a rare one). Without expiry that key stays marked in-flight forever, so a single interrupted build would permanently un-distribute one translation unit.Both components are pure with respect to I/O over an injected
IClock, so every expiry and capacity rule is a unit test againstManualClockrather than a sleep.What a worker refuses to take from a client
A job names a fingerprint, never a program; the worker maps that to a compiler from its own configuration and refuses one it does not have. This is the difference between a build accelerator and a remote shell, and it costs nothing.
The client's arguments are filtered by positive refusal — the whole command line is declined if it carries anything that could name a file — and checked again on the receiving side. The two checks defend against different things: the client's protects an honest client from dispatching something that would not work, the worker's protects the worker from a client that is not honest. A worker trusting the client's filter would be secured by code running on the attacker's machine.
That filter started as a deny-list over
PathValueFlags()and my own test caught it: that table does not know-isystem,--sysroot,-B,-specs=,-fplugin=or@file. Deny-listing paths is a losing game, so it is a positive refusal now.The client STOREs the result, never the worker. A STORE is trusted today because the storer compiled it themselves, so they could only poison their own key space with something they would have gotten anyway. If workers stored, one rogue worker would poison the shared cache for everyone. Routing the result back through the client keeps the existing trust model exactly as-is, and workers get no cache credentials at all.
Payloads and negotiation
Bulk fields travel in a
[u8 codec][u32 rawLen][bytes]envelope usingCore/Compression's existing ids, with the accepted-codec list carried in the request — so the two ends agree without a handshake. Every exchange is client-initiated, the reply picks from what the request offered, and the round-trip count is unchanged; this is the same reasoning that keepsAUTHfree.rawLenis what lets a receiver reject a declared expansion before decompressing a byte.Nothing in common falls back to
Identityrather than refusing: a build must never lose its cache because two peers were compiled with different codec sets.The scheduler's control verbs (
Register/Heartbeat/Lease) carry a 64 KiB ceiling rather than the session cap. That listener is meant to be reachable by a whole fleet, and a scheduler that can be made to allocate 256 MiB per frame by anything that authenticated once is a scheduler that stops scheduling.Compileis the deliberate exception, since it carries a preprocessed translation unit.Dispatch preprocesses separately, and that is not an optimization
Reusing the cache key's preprocessed text would have been free, and it silently breaks every
-Werrorbuild. The key's probe suppresses#linemarkers so no checkout path reaches the key — and those same markers are what tell the compiler which lines came from a system header. Without them every warning inside libc++ resurfaces in the remote compile, and under-pedantic -Werrorthose are errors. Every dispatched TU would fail and be retried locally, so distribution would have looked like it worked while never once helping.Dispatch therefore runs its own preprocess with markers, paying ~45 ms on a path already committed to seconds of remote compilation. The key's text is untouched, so nothing re-keys.
Fixing that broke it a second way: under
-pedanticthe markers themselves are a GNU extension (-Wgnu-line-marker). The answer is the one ccache and distcc already use — tell the driver its input is preprocessed (-x c++-cpp-output,cpp-outputfor C, and/TP///TCfor MSVC; see below for why “nothing for MSVC” was itself a defect). Both are newDriverSpeccolumns rather than new branches.Neither defect is reachable from a unit test. Both appeared the moment the feature was run under the project's own build flags instead of a toy command line.
macOS signing asked a name, not the file
MacOSSignBinaries.cmakematched an allow-list of binary names, sofastcache-compile-nodeshipped unsigned — glob picks it up, filter drops it, and nothing fails until Apple's notary service says no, on a job most changes never run. It now asksfilewhether the thing is Mach-O, which is what the comment beside it always claimed it did. Both branches exercised locally.What a worker is told, and what it may name
A worker writes its own scratch file, and an MSVC driver reads the language off that file's extension — so while nothing stated the language, a dispatched
.ctranslation unit came back compiled as C++. Where C is not valid C++ that is a failed remote compile retried locally (C silently never distributing, with a green build); where it is, it is an object with C++ mangling that the client then STOREs under the C key.DriverSpec::preprocessedInputFlagswas empty for MSVC on the reasoning that/Eemits standard#lineso there is nothing to tell it — true about the markers, and it left the language unsaid./TCand/TPare the-xspelling MSVC does have, and/TPis a measured byte-for-byte no-op on a C++ TU, so nothing that worked before moves.The extension turns out to be the last of three answers, not the first:
++driver compiles everything as C++ ("g++ treats .c, .h and .i files as C++ source files"), so taking the language off the extension tells a worker to compile as C what this machine compiles as C++.gccandg++are oneFlavor, so it is a second column on the name table.set_source_files_properties(x.c PROPERTIES LANGUAGE CXX)emits/TPor-x c++. The launcher appends its own spelling last so it wins, which is a silent override the moment something else spoke, so such a command line is refused instead..Cis C++ to a GNU driver and C to an MSVC one, so the extension does not answer the question at all and is never guessed at.C++ modules are the same question with a different answer. A module interface unit writes a BMI beside its object, and a hit reproduces only the object and the dependency record — so replaying one leaves the BMI missing (loud) or stale (silent). Those extensions were simply not in
IsSourceSuffix, so such lines fell through as "no source file found" and were passed through in silence, indistinguishable from a broken cache; adding.ixxthere — the obvious "support modules" change — would have turned that into a silent wrong build. They are now refused by name, with the reason said out loud, alongside an ordinary source promoted by a flag (cl /interface,-fmodule-output,--precompile,/Yc).Finally, the client now tells the worker what to call its scratch file.
DispatchRequest::sourceNamewas set, documented, and never encoded. A compiler records the name of the file it was handed, so a worker inventing one produces an object differing from a local build's in that name and nothing else — seven bytes on clang-cl, and none once they agree. Only the base name travels, and the worker reduces it to one allow-listed component before it becomes a path (no separators, no.., no drive letters, bounded length, neverCON).Byte-identity on Windows: what is actually assertable
Three commits of this branch asked CI whether MSVC is reproducible, and the control that answered was wrong: it compiled its second object to
tu2.oand compared it againsttu.o, and MSVC records the absolute path of the object file inside every object (.debug$S, no debug flag asked for). The ~70-byte figure that became "the driver's noise floor" was that record. Measured on a Windows machine instead of inferred from a log:cl, same input, same/Fopath, twicecl/clang-cl, same path, two seconds apart/Breprosuppressescl, reference vs worker-shaped (/Etext, other path).debug$Sand.chks64differ;.text$mn,.rdata,.xdata,.pdata,.drectve,.data$r,.bssare byte-identicalclang-cl, same twoSo the Windows fixture compares section by section against a per-driver table of what may differ —
clang-cl's row is empty — instead of asserting a plausible size, which a wrong optimisation level or a subtly different header set would pass. The COFF header is compared field by field rather than skipped, because an object built for another architecture differs there and nowhere a section walk would look. The comparison has a-SelfTestof its own, registered as a non-smoke ctest case: a fixture's own logic is the one thing nothing else tests, and this one's cost five CI round trips to find out.The whole fixture — both drivers, all four cases — was verified locally on Windows before pushing, which is the first time on this branch that a Windows answer did not require a CI run.
Also fixed here
cmake --preset cl-debughas not built the launcher since compression was added to it. The CPM-built zstd takes CMake's default (DLL) MSVC runtime while the launcher links the static CRT; in Release nothing breaks, and in Debugassert()pulls__imp__wassert, which only the DLL CRT has. Windows CI builds Release only, so every job stayed green while no Windows developer could build Debug at all.fastcache-compile-node, which shipped, is documented, has an installed systemd unit and a CI job — and was invisible to anyone reading only the front page.Still to come
The worker's daemon shell —
IDaemonHost,--install-service/--uninstall-service, theServiceSpecdescriptor seam and aDefaultConfigPathgeneralization — which on Linux is covered in practice by the shipped socket-activated unit, and on macOS/Windows means running the worker in the foreground or under an existing supervisor. The worker also exposes no metrics of its own (the scheduler-side dispatch counters are wired and documented). Both are tracked as issues rather than left in this description.One naming note: the namespace is
FastCache::Distributed, notDispatch—RedisResp.cppalready has aDispatch()function and the two collide under unqualified lookup insidenamespace FastCache.