From 35b6ded0e6db6d03eeaca7217cc8e09963766e47 Mon Sep 17 00:00:00 2001 From: Rosa Gutierrez Date: Mon, 31 Aug 2026 10:17:12 +0200 Subject: [PATCH 1/2] Exit forked processes immediately, skipping Ruby's at-exit cleanup When a thread is killed while it waits in SQLite's busy handler, the kill unwinds through SQLite's C frames and leaves the connection mutex locked (the sqlite3 gem invokes the handler with a bare rb_funcall). Ruby kills every leftover thread at process exit -- a heartbeat or maintenance timer mid-query, a pool thread running a job past the shutdown timeout -- and then finalizes every remaining object, so a forked process on SQLite under write contention could deadlock inside sqlite3_close_v2 during exit and linger forever instead of exiting. That's where the orphaned processes in the test suite and the long-standing Puma plugin hang on SQLite came from: about 1 in 10 runs of the plugin's "supervisor dies" test left a zombie scheduler behind, stuck with this backtrace: ruby_cleanup -> rb_objspace_call_finalizer -> rb_data_free -> sqlite3_close_v2 -> pthread_mutex_lock (never returns) Exit forked processes with exit!, like Puma's cluster workers and our own QUIT handler already do, skipping at-exit hooks and finalizers: everything the process needs to do on shutdown has already run by then. Only the success path exits this way, so an error raised out of the fork's block still gets reported and exits non-zero. The Puma plugin's supervisor fork gets the same treatment, which is what lets Puma's Process.wait return during shutdown. With this, that same test leaves 0 orphans in 15 runs, and the full suite, the three lifecycle suites (which assert the forks' exit statuses) and the Puma plugin tests all stay green. The busy-handler mutex leak itself is a sqlite3-ruby bug (reported separately with a standalone reproduction). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CACej2M9mLVDwpE3Bk8V41 --- lib/puma/plugin/solid_queue.rb | 10 +++++++++ lib/solid_queue/processes/supervised.rb | 8 +++++++ test/unit/supervised_test.rb | 29 +++++++++++++++++++++++++ 3 files changed, 47 insertions(+) create mode 100644 test/unit/supervised_test.rb diff --git a/lib/puma/plugin/solid_queue.rb b/lib/puma/plugin/solid_queue.rb index 8a7aea28b..a0acfdbba 100644 --- a/lib/puma/plugin/solid_queue.rb +++ b/lib/puma/plugin/solid_queue.rb @@ -33,6 +33,11 @@ def start_forked(launcher) @solid_queue_pid = fork do Thread.new { monitor_puma } SolidQueue::Supervisor.start(mode: :fork) + + # Same as Processes::Supervised#create_fork: skip at-exit + # finalization, which can deadlock on database handles when a + # thread was killed while inside a query + exit!(0) end end @@ -43,6 +48,11 @@ def start_forked(launcher) @solid_queue_pid = fork do Thread.new { monitor_puma } start_solid_queue(mode: :fork) + + # Same as Processes::Supervised#create_fork: skip at-exit + # finalization, which can deadlock on database handles when a + # thread was killed while inside a query + exit!(0) end end diff --git a/lib/solid_queue/processes/supervised.rb b/lib/solid_queue/processes/supervised.rb index 5638026d2..9ad02be66 100644 --- a/lib/solid_queue/processes/supervised.rb +++ b/lib/solid_queue/processes/supervised.rb @@ -29,6 +29,14 @@ def create_fork(&block) fork do register_signal_handlers block.call + + # Exit skipping at-exit hooks and finalizers, like Puma's cluster + # workers do: Ruby would finalize every object still alive in the + # fork, including SQLite database handles whose mutex can be left + # locked when a thread is killed while waiting in SQLite's busy + # handler, deadlocking the exit. Everything the process needs to do + # on shutdown has already run by now. + exit!(0) end end diff --git a/test/unit/supervised_test.rb b/test/unit/supervised_test.rb new file mode 100644 index 000000000..d79ccd1fc --- /dev/null +++ b/test/unit/supervised_test.rb @@ -0,0 +1,29 @@ +# frozen_string_literal: true + +require "test_helper" + +class SupervisedTest < ActiveSupport::TestCase + class FakeProcess + include SolidQueue::Processes::Supervised + + def stop + end + end + + test "forked processes exit immediately, without running inherited at-exit hooks" do + reader, writer = IO.pipe + + pid = FakeProcess.new.send(:create_fork) do + at_exit { writer.write("at_exit ran") } + writer.write("block ran") + end + + writer.close + _, status = Process.waitpid2(pid) + + assert_equal 0, status.exitstatus + assert_equal "block ran", reader.read + ensure + reader.close + end +end From ff53fe19089062e90daad7bbfecc837cb126f048 Mon Sep 17 00:00:00 2001 From: Rosa Gutierrez Date: Mon, 31 Aug 2026 10:17:12 +0200 Subject: [PATCH 2/2] Bind the Puma plugin tests only to their dynamically allocated port The test harness passes -b tcp://127.0.0.1: since the dynamic port allocation was added, but the plugin test configs still set port 3000, so every test Puma bound the default port as well and collided with whatever else was using it on the same machine. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CACej2M9mLVDwpE3Bk8V41 --- test/dummy/config/puma_async.rb | 4 ---- test/dummy/config/puma_fork.rb | 4 ---- 2 files changed, 8 deletions(-) diff --git a/test/dummy/config/puma_async.rb b/test/dummy/config/puma_async.rb index beb652595..925bde385 100644 --- a/test/dummy/config/puma_async.rb +++ b/test/dummy/config/puma_async.rb @@ -13,10 +13,6 @@ # worker_timeout 3600 if ENV.fetch("RAILS_ENV", "development") == "development" -# Specifies the `port` that Puma will listen on to receive requests; default is 3000. -# -port ENV.fetch("PORT") { 3000 } - # Specifies the `environment` that Puma will run in. # environment ENV.fetch("RAILS_ENV") { "development" } diff --git a/test/dummy/config/puma_fork.rb b/test/dummy/config/puma_fork.rb index 4cdbbfd1d..08422974f 100644 --- a/test/dummy/config/puma_fork.rb +++ b/test/dummy/config/puma_fork.rb @@ -13,10 +13,6 @@ # worker_timeout 3600 if ENV.fetch("RAILS_ENV", "development") == "development" -# Specifies the `port` that Puma will listen on to receive requests; default is 3000. -# -port ENV.fetch("PORT") { 3000 } - # Specifies the `environment` that Puma will run in. # environment ENV.fetch("RAILS_ENV") { "development" }