Repository navigation
Fix intermittent hang of parallel_map when called from several threads - #566
Conversation
Forked workers of a ProcessPoolExecutor inherit the registry of "wakeup" objects of all process pools in the parent (concurrent.futures.process. _threads_wakeups), and wake them up when they exit. In recent Python versions (e.g. 3.12.15, 3.13, 3.14) these objects have a lock; if another thread held such a lock while a worker was forked, the worker hung when exiting, and parallel_map hung in executor.shutdown(). This made test_overlapping_calls in tmulti_proc.py hang intermittently in CI. - Clear the inherited registry in the worker initializer (CPython does the same for thread pools in forked processes). - Add a regression test that triggers the race deterministically. - Add a 30-minute timeout to the CI job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
darafferty
left a comment
darafferty
left a comment
There was a problem hiding this comment.
Not sure I completely understand the problem, but I think I might have encountered some hanging myself on recent runs. At any rate, good to have a test for it!
It's subtle, indeed, like with all bugs related to race conditions. Apparently, this is a bug in the Python library |
#566) Forked workers of a ProcessPoolExecutor inherit the registry of "wakeup" objects of all process pools in the parent (concurrent.futures.process. _threads_wakeups), and wake them up when they exit. In recent Python versions (e.g. 3.12.15, 3.13, 3.14) these objects have a lock; if another thread held such a lock while a worker was forked, the worker hung when exiting, and parallel_map hung in executor.shutdown(). This made test_overlapping_calls in tmulti_proc.py hang intermittently in CI. - Clear the inherited registry in the worker initializer (CPython does the same for thread pools in forked processes). - Add a regression test that triggers the race deterministically. - Add a 30-minute timeout to the CI job. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Problem
CI runs hang intermittently in
test/tmulti_proc.py, attest_overlapping_calls. One #564 run hung there for 27 minutes, until it was cancelled, and the run for #560 has been stuck in the same test step for hours. Jobs then run until GitHub's 6-hour limit.The cause is a race in
concurrent.futures.processwhen forking from a multi-threaded process:_threads_wakeups) of the "wakeup" objects of all running process pools. A forked worker inherits it, including the entries of other pools.threading._shutdown()runs_python_exit, which callswakeup()on every entry, andwakeup()takes the object's lock.parallel_maphangs inexecutor.shutdown()while waiting for it.test_overlapping_callscallsparallel_mapfrom two threads at once, so occasionally one pool forks while the other thread is insidewakeup(). The stacks of a reproduced hang (with py-spy) show exactly this: the parent waits inexecutor.shutdown(), and the worker is blocked in_ThreadWakeup.wakeupcalled from_python_exit.The lock in
_ThreadWakeuponly exists in recent Python versions. It's present in 3.12.15, 3.13 and 3.14, but not in 3.12.3 or in 3.10/3.11. CI uses 3.12.16, and the hang couldn't be reproduced with 3.12.3.Fix
multi_proc.py: the worker initializer clears the inherited registry. Those entries belong to the parent's pools, which are of no use in a worker. CPython already does the same for thread pools in forked processes (concurrent/futures/thread.py:os.register_at_fork(after_in_child=_threads_queues.clear)), but not for process pools. Real PyBDSF runs callparallel_mapfrom one thread, so they are affected only when PyBDSF is used from several threads at once.tmulti_proc.py: a newtest_fork_while_other_pool_is_busytriggers the race deterministically. It holds another pool's wakeup lock whileparallel_mapforks its workers. The test runs in a subprocess with a 60-second timeout, so a regression fails the test instead of hanging the job. It's skipped on Python versions without the lock.ci.yml:timeout-minutes: 30for the job (normal runs take 2–4 minutes).Testing
tmulti_proc.py, Python 3.11, 3.12, 3.13, 3.14Unrelated and not changed here: on Python 3.10,
test_interrupt_stops_workersfails intermittently ("workers still running"), also onmaster(in 1 of 3 runs in both cases). The wheel builds (cibuildwheel) run this test on cp310.🤖 Generated with Claude Code