Skip to content

Fix race: delete worker options file only after workers exit - #22020

Open
Dextheking1 wants to merge 1 commit into
python:masterfrom
Dextheking1:fix/worker-options-race
Open

Dextheking1 wants to merge 1 commit into
python:masterfrom
Dextheking1:fix/worker-options-race

Conversation

@Dextheking1

Copy link
Copy Markdown

Fixes #21974.

Root cause

When several mypy invocations run concurrently with --num-workers, the worker
subprocesses can be slow to start up because they are starved of CPU. The coordinator
gives up on such a worker after WORKER_START_TIMEOUT and tears the build down.

The teardown in build() did this:

finally:
    for thread in connect_threads:
        thread.join()
    if options_data is not None:
        os.unlink(options_data)          # <-- options file deleted first
    ...
    for worker in workers:
        worker.close()                   # <-- worker processes reaped later

The serialized options file (.mypy_cache/.worker_options.<id>.data) was deleted
before the worker subprocesses were confirmed to have exited. A worker that was
still starting up therefore woke up to find its options file gone and crashed with the
error reported in the issue:

FileNotFoundError: [Errno 2] No such file or directory: '.mypy_cache/.worker_options.<id>.data'
Failed to establish connection with worker: No status file found

This is a genuine race, not a filesystem visibility delay: the coordinator itself removes
the file out from under the worker. The retry added in #21980 helps when the file is
merely not visible yet, but it cannot help once the file has actually been deleted.

The fix

Two small changes in mypy/build.py:

  1. In build(), unlink the options file only after every worker has been closed, so no
    worker can still be reading it.
  2. In WorkerClient.close(), actually reap the worker process instead of returning while
    it may still be running: terminate (immediately for workers we never connected to),
    wait, and kill() if it ignores SIGTERM. This is what makes (1) meaningful, and it
    also avoids leaving orphaned worker processes behind.

The happy path is unchanged: connected workers are still sent their shutdown message and
given a chance to exit gracefully before being terminated.

Verification

  • Added mypy/test/testworker.py, a deterministic unit test that drives build() with a
    worker process that never starts up, and asserts the options file is unlinked only once
    the worker process is dead. It fails on unpatched master (AssertionError: unexpectedly None) and passes with this fix.
  • pytest mypy/test/testcheck.py mypy/test/testipc.py mypy/test/testworker.py passes
    (8200+ tests, including the existing parallel-checking tests).
  • Manually reproduced the original race by launching a build whose worker is artificially
    delayed past WORKER_START_TIMEOUT: on master the worker observes the options file
    disappearing underneath it; with this fix the worker is terminated before the file is
    removed and no FileNotFoundError occurs.

The one failure I saw in mypy/test/testdaemon.py::...testAttrsTypeIgnoreAfterUnknownImport
is pre-existing and reproduces identically on pristine master (unrelated output ordering).

@Dextheking1
Dextheking1 force-pushed the fix/worker-options-race branch from d4770e6 to e304174 Compare September 21, 2026 22:10
@github-actions

This comment has been minimized.

@Dextheking1
Dextheking1 force-pushed the fix/worker-options-race branch from e304174 to 34ae133 Compare September 21, 2026 22:42
@github-actions

This comment has been minimized.

@Dextheking1
Dextheking1 force-pushed the fix/worker-options-race branch from 34ae133 to 24002c1 Compare September 21, 2026 23:04
@github-actions

This comment has been minimized.

…21974)

When a parallel build gives up on a worker that never managed to start up
(common when several mypy invocations run concurrently and starve the workers
of CPU), the coordinator used to delete the serialized options file before
waiting for the worker subprocess to exit. A worker that was still starting
up would then fail to read its options file and crash with a confusing
FileNotFoundError.

Delete the options file only after every worker process is known to have
exited, and make WorkerClient.close() reap the process (terminating, then
killing if it ignores SIGTERM) instead of leaving it running.
@Dextheking1
Dextheking1 force-pushed the fix/worker-options-race branch from 24002c1 to 8ba74f7 Compare September 21, 2026 23:26
@github-actions

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent parallel invocations can lose worker-options files

1 participant