How is single-worker recycling supposed to work after max_tasks_per_child is reached?

[aysanorhun]

I am trying to understand the intended worker recycle/restart model. From reading the worker, receiver, and process manager code, this is how it looks to me:

• when max_tasks_per_child is reached, the worker stops prefetching, waits for running tasks to finish, and goes through shutdown, but the process itself does not exit
• the process manager restarts a worker either through a reload action, or when it detects that a specific worker process is no longer alive

What I want to clarify is how restarting a single worker is supposed to work. As far as I can tell:

• SIGINT / SIGTERM are shutdown signals, not restart signals
• SIGHUP reloads all workers
• ReloadOneAction restarts one worker, but I do not see a supported way to trigger that for a specific worker from within worker code
• if a worker process exits, the process manager detects it and respawns that specific worker

Right now, the only way I was able to make a specific worker get recycled was to add os._exit(1) in the worker shutdown path so the process exits and the manager respawns it. So my questions are:

1. When max_tasks_per_child is reached, is the intended behavior that the worker only goes through shutdown, or that the worker process exits and the parent respawns it?
2. Is there a supported way to recycle or restart one specific worker manually, without relying on something like os._exit(1)?

My use case is that I do not want to restart all workers. I want to recycle a specific worker once max_tasks_per_child is reached.

[juanmicl]

I traced this through the code and reproduced the behavior locally, so here is what actually happens and where the hang lives.

The respawn machinery in core works

When max_tasks_to_execute is reached:

1. Receiver._should_stop_prefetch (taskiq/receiver/receiver.py) stops the prefetcher, which sends _QueueSignal.DONE and closes the broker's listen generator.
2. The runner drains running tasks ("No more tasks to wait for. Shutting down.") and listen() returns.
3. start_listen (taskiq/cli/worker/run.py) then calls shutdown_broker(broker, timeout), which is the "Shutting down the broker." line.
4. After that returns, start_listen returns and the worker process exits.
5. ProcessManager.start polls worker.is_alive() every second and queues a ReloadOneAction for dead workers.

I verified this end to end with a minimal custom broker (yields N messages, then holds the connection open like a real broker) run with taskiq worker app:broker app --workers 1 --max-tasks-per-child 2. With a no-op shutdown() the cycle repeats continuously: exit, dead detection within about a second, respawn with a new pid. 26 task executions over a 14 second run, all through respawns.

The reported hang starts below core

In your logs the last worker line is "Shutting down the broker.", and the process stays alive, so the hang is inside or after broker.shutdown(). I reproduced that exact signature with two different broker-level defects:

1. A shutdown() that blocks the event loop thread. shutdown_broker wraps the call in asyncio.wait_for(timeout), but the timeout only fires if the loop can run callbacks. A blocking call freezes the loop and the timeout never triggers. I verified this: with a 5 second shutdown timeout, no "Broker.shutdown cannot be completed" warning was logged even after the process sat there for 50 more seconds.
2. A non-daemon thread left behind by the broker. shutdown() returns, the worker's main function returns, but the interpreter waits for the thread and the process never exits.

In both cases worker.is_alive() stays True, so no reload is ever scheduled. There is also a side effect worth knowing: the manager itself hangs when it tries to shut down, because multiprocessing joins non-daemon children at exit. In my repro the manager had to be SIGKILLed while the worker was stuck.

How to tell which one you are hitting

With the real taskiq-redis setup, run py-spy dump --pid <stuck-worker-pid> (or enable faulthandler and send SIGABRT):

• main thread inside run_until_complete -> wait_for -> broker.shutdown means case 1,
• main thread in threading._shutdown or a join means case 2.

Design question for the maintainers

Independent of what taskiq-redis does, should the worker child guarantee its own exit after a graceful shutdown, the way gunicorn workers do? For example, after shutdown_broker completes, give the process a short grace period and call os._exit(0) if anything is still holding it. That would make --max-tasks-per-child self-healing regardless of broker behavior, and it would also unblock the manager's shutdown join. The docs describe this option as a restart ("maximum number of tasks to be executed by a single worker process before restart"), so the current behavior reads like a bug against that promise.