Conversation
|
I think I see the issue now. It's a combination of 2 things, which is why 3.14.0 worked, but 3.14.2 broke. First: In 3.14 they changed the default process pool start method
However, there was an issue that prevented forkserver from setting sys.argv correctly, and when that was fixed:
Then the failure started showing up. |
…iprocessing / Py3.14 regression) (#5) * Fix explicit profiler ownership for multiprocessing * Avoid helper process atexit registration * Skip atexit output in forked children * Refine ownership checks for explicit profiler * Add debug hooks and reduce CI matrix * Skip orphaned forkserver output * Restore full CI matrix
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #415 +/- ##
==========================================
+ Coverage 87.56% 87.98% +0.41%
==========================================
Files 18 20 +2
Lines 1641 2180 +539
Branches 348 467 +119
==========================================
+ Hits 1437 1918 +481
- Misses 149 200 +51
- Partials 55 62 +7
... and 3 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
|
I think overall this method looks fine. The idea is when a script starts up and the explicit profiler is enabled, it records the pid the process and puts it in an environment variable I'm leaving the debug statements in as they are disabled by default, and I think they could be useful if this causes more problems in the future. |
Fix Python 3.14.2 multiprocessing shutdown behavior by ensuring only the owner process registers the
atexitreporting hook.Problem
When running the complex example with
LINE_PROFILE=1andPROFILE_TYPE=expliciton Python 3.14.2, multiprocessing using theforkserverstart method can cause non-owner processes (e.g., the forkserver process / child processes) to also register theatexithook. At interpreter shutdown, those processes may try to print/write profiling results, resulting in:profile_output.txt,Root cause
atexit.register(...)was executed in processes that should not own reporting. Even ifshow()includes guards, shutdown ordering and partially-torn-down globals can make those guards unreliable. The safest approach is to avoid registering the hook outside the owner process.Fix
Move/guard
atexit.register(...)so it occurs only after establishing ownership (PID/parent/process role) and only in the designated owner process. Non-owner processes never register the shutdown reporter.The above is a GPT description of the patch, which I used heavily to find the problem. I believe the diagnosis is correct, but I don't understand what changed in 3.14.2 to cause the problem. Pushing up the patch as is to see if it fixes the dashboard.