Fix race: delete worker options file only after workers exit - #22020
Open
Dextheking1 wants to merge 1 commit into
Open
Dextheking1 wants to merge 1 commit into
Dextheking1 wants to merge 1 commit into
Conversation
Dextheking1
force-pushed
the
fix/worker-options-race
branch
from
September 21, 2026 22:10
d4770e6 to
e304174
Compare
This comment has been minimized.
This comment has been minimized.
Dextheking1
force-pushed
the
fix/worker-options-race
branch
from
September 21, 2026 22:42
e304174 to
34ae133
Compare
This comment has been minimized.
This comment has been minimized.
Dextheking1
force-pushed
the
fix/worker-options-race
branch
from
September 21, 2026 23:04
34ae133 to
24002c1
Compare
This comment has been minimized.
This comment has been minimized.
…21974) When a parallel build gives up on a worker that never managed to start up (common when several mypy invocations run concurrently and starve the workers of CPU), the coordinator used to delete the serialized options file before waiting for the worker subprocess to exit. A worker that was still starting up would then fail to read its options file and crash with a confusing FileNotFoundError. Delete the options file only after every worker process is known to have exited, and make WorkerClient.close() reap the process (terminating, then killing if it ignores SIGTERM) instead of leaving it running.
Dextheking1
force-pushed
the
fix/worker-options-race
branch
from
September 21, 2026 23:26
24002c1 to
8ba74f7
Compare
Contributor
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #21974.
Root cause
When several
mypyinvocations run concurrently with--num-workers, the workersubprocesses can be slow to start up because they are starved of CPU. The coordinator
gives up on such a worker after
WORKER_START_TIMEOUTand tears the build down.The teardown in
build()did this:The serialized options file (
.mypy_cache/.worker_options.<id>.data) was deletedbefore the worker subprocesses were confirmed to have exited. A worker that was
still starting up therefore woke up to find its options file gone and crashed with the
error reported in the issue:
This is a genuine race, not a filesystem visibility delay: the coordinator itself removes
the file out from under the worker. The retry added in #21980 helps when the file is
merely not visible yet, but it cannot help once the file has actually been deleted.
The fix
Two small changes in
mypy/build.py:build(), unlink the options file only after every worker has been closed, so noworker can still be reading it.
WorkerClient.close(), actually reap the worker process instead of returning whileit may still be running: terminate (immediately for workers we never connected to),
wait, and
kill()if it ignoresSIGTERM. This is what makes (1) meaningful, and italso avoids leaving orphaned worker processes behind.
The happy path is unchanged: connected workers are still sent their shutdown message and
given a chance to exit gracefully before being terminated.
Verification
mypy/test/testworker.py, a deterministic unit test that drivesbuild()with aworker process that never starts up, and asserts the options file is unlinked only once
the worker process is dead. It fails on unpatched
master(AssertionError: unexpectedly None) and passes with this fix.pytest mypy/test/testcheck.py mypy/test/testipc.py mypy/test/testworker.pypasses(8200+ tests, including the existing parallel-checking tests).
delayed past
WORKER_START_TIMEOUT: onmasterthe worker observes the options filedisappearing underneath it; with this fix the worker is terminated before the file is
removed and no
FileNotFoundErroroccurs.The one failure I saw in
mypy/test/testdaemon.py::...testAttrsTypeIgnoreAfterUnknownImportis pre-existing and reproduces identically on pristine
master(unrelated output ordering).