Merge latest upstream solid_queue v1.5.0 (adopts PR #11 + adds a semaphore guard)

.github/workflows/main.yml

@@ -26,20 +26,13 @@ jobs:
       fail-fast: false
       matrix:
         ruby-version:
-        - 3.1
         - 3.2
         - 3.3
         - 3.4
         - 4.0
         database: [ mysql, postgres, sqlite ]
         gemfile: [ rails_7_1, rails_7_2, rails_8_0, rails_8_1, rails_main ]
         exclude:
-          - ruby-version: "3.1"
-            gemfile: rails_8_0
-          - ruby-version: "3.1"
-            gemfile: rails_8_1
-          - ruby-version: "3.1"
-            gemfile: rails_main
           - ruby-version: "3.2"
             gemfile: rails_main
     services:

Appraisals

@@ -1,13 +1,10 @@
 # frozen_string_literal: true
 
 appraise "rails-7-1" do
-  # rdoc 6.14 is not compatible with Ruby 3.1
-  gem 'rdoc', '6.13'
   gem "railties", "~> 7.1.0"
 end
 
 appraise "rails-7-2" do
-  gem 'rdoc', '6.13'
   gem "railties", "~> 7.2.0"
 end
 

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    solid_queue (1.4.0)
+    solid_queue (1.5.0)
       activejob (>= 7.1)
       activerecord (>= 7.1)
       concurrent-ruby (>= 1.3.7)

README.md

@@ -23,6 +23,7 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite,
   - [Threads, processes, and signals](#threads-processes-and-signals)
   - [Database configuration](#database-configuration)
   - [Other configuration settings](#other-configuration-settings)
+  - [Validating the configuration](#validating-the-configuration)
 - [Lifecycle hooks](#lifecycle-hooks)
 - [Errors when enqueuing](#errors-when-enqueuing)
 - [Concurrency controls](#concurrency-controls)
@@ -45,7 +46,7 @@ Solid Queue is configured by default in new Rails 8 applications. If you're runn
 1. `bundle add solid_queue`
 2. `bin/rails solid_queue:install`
 
-(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.1.6.)
+(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.2.)
 
 This will configure Solid Queue as the production Active Job backend, create the configuration files `config/queue.yml` and `config/recurring.yml`, and create the `db/queue_schema.rb`. It'll also create a `bin/jobs` executable wrapper that you can use to start Solid Queue.
 
@@ -339,7 +340,7 @@ FROM solid_queue_ready_executions
 WHERE queue_name LIKE 'beta%';
 ```
 
-This type of `DISTINCT` query on a column that's the leftmost column in an index can be performed very fast in MySQL thanks to a technique called [Loose Index Scan](https://dev.mysql.com/doc/refman/8.0/en/group-by-optimization.html#loose-index-scan). PostgreSQL and SQLite, however, don't implement this technique, which means that if your `solid_queue_ready_executions` table is very big because your queues get very deep, this query will get slow. Normally your `solid_queue_ready_executions` table will be small, but it can happen.
+This type of `DISTINCT` query on a column that's the leftmost column in an index can be performed very fast in MySQL thanks to a technique called [Loose Index Scan](https://dev.mysql.com/doc/refman/8.0/en/group-by-optimization.html#loose-index-scan). PostgreSQL doesn't implement this technique natively, so Solid Queue uses a [recursive CTE](https://www.postgresql.org/docs/current/queries-with.html#QUERIES-WITH-RECURSIVE) to emulate it, achieving similar performance by walking the B-tree index and jumping between distinct values. SQLite doesn't implement loose index scan either, but this is unlikely to be a problem since SQLite is typically used in development with small datasets.
 
 Similarly to using prefixes, the same will happen if you have paused queues, because we need to get a list of all queues with a query like
 ```sql
@@ -373,6 +374,8 @@ The supervisor is in charge of managing these processes, and it responds to the
 
 When receiving a `QUIT` signal, if workers still have jobs in-flight, these will be returned to the queue when the processes are deregistered.
 
+On Windows, the `QUIT` signal can't be trapped, so the supervisor only responds to `TERM` and `INT` there.
+
 If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats. A stale supervised process isn't pruned while its supervisor has a fresh heartbeat, because the supervisor detects actual child exits directly. It becomes eligible for heartbeat-based pruning when its supervisor is also stale. Jobs claimed by a pruned process will be marked as failed with a `SolidQueue::Processes::ProcessPrunedError`. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this.
 
 In a similar way, if a worker is terminated in any other way not initiated by the above signals (e.g. a worker is sent a `KILL` signal), jobs in progress will be marked as failed so that they can be inspected, with a `SolidQueue::Processes::ProcessExitError`. Sometimes a job in particular is responsible for this, for example, if it has a memory leak and you have a mechanism to kill processes over a certain memory threshold, so this will help identifying this kind of situation.
@@ -409,6 +412,22 @@ There are several settings that control how Solid Queue works that you can set a
 - `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true — defaults to 1 day. When installing Solid Queue, [a recurring job](#recurring-tasks) is automatically configured to clear finished jobs every hour on the 12th minute in batches. You can edit the `recurring.yml` configuration to change this as you see fit.
 - `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes.
 
+### Validating the configuration
+
+You can validate the Solid Queue configuration ahead of time, without starting any process. This is handy in deploy scripts or CI to catch mistakes—a typo in `recurring.yml`, no processes configured, and so on—before they cause a supervisor to boot into a broken state:
+
+```bash
+# Using the bin/jobs binstub
+bin/jobs check
+
+# Or via rake
+bin/rails solid_queue:check
+```
+
+Both commands validate the configuration for the current Rails environment. On success they print `Solid Queue configuration is valid.` and exit `0`; otherwise they print the errors and exit non-zero. When the number of threads is larger than the [database connection pool](#database-configuration), they also print an advisory warning about it—the same one the supervisor logs on boot. They're tolerant of a missing database connection, so they can run on CI or deploy hosts without database credentials.
+
+`bin/jobs check` accepts the same options as `bin/jobs start` (e.g. `--config_file`, `--recurring_schedule_file`, `--skip-recurring`). The rake task honors the same environment variables Solid Queue already uses: `SOLID_QUEUE_CONFIG`, `SOLID_QUEUE_RECURRING_SCHEDULE`, and `SOLID_QUEUE_SKIP_RECURRING`. To validate a specific environment's configuration, set `RAILS_ENV`, for example `RAILS_ENV=production bin/jobs check`.
+
 
 ## Lifecycle hooks
 
@@ -472,7 +491,7 @@ Solid Queue extends Active Job with concurrency controls, that allows you to lim
 
 ```ruby
 class MyJob < ApplicationJob
-  limits_concurrency to: max_concurrent_executions, key: ->(arg1, arg2, **) { ... }, duration: max_interval_to_guarantee_concurrency_limit, group: concurrency_group, on_conflict: on_conflict_behaviour
+  limits_concurrency to: max_concurrent_executions, key: ->(arg1, arg2, *) { ... }, duration: max_interval_to_guarantee_concurrency_limit, group: concurrency_group, on_conflict: on_conflict_behaviour
 
   # ...
 ```
@@ -766,7 +785,9 @@ production:
 
 Tasks are specified as a hash/dictionary, where the key will be the task's key internally. Each task needs to either have a `class`, which will be the job class to enqueue, or a `command`, which will be eval'ed in the context of a job (`SolidQueue::RecurringJob`) that will be enqueued according to its schedule, in the `solid_queue_recurring` queue.
 
-Each task needs to have also a schedule, which is parsed using [Fugit](https://github.com/floraison/fugit), so it accepts anything [that Fugit accepts as a cron](https://github.com/floraison/fugit?tab=readme-ov-file#fugitcron). You can optionally supply the following for each task:
+Each task needs to have also a schedule, which is parsed using [Fugit](https://github.com/floraison/fugit), so it accepts anything [that Fugit accepts as a cron](https://github.com/floraison/fugit?tab=readme-ov-file#fugitcron). Schedules can include a time zone (e.g. `0 9 * * * America/New_York` or `every day at 9am America/New_York`). When a schedule doesn't specify one, it's interpreted in the application's configured time zone (`config.time_zone`) by default. You can change or disable this default with `config.solid_queue.time_zone`; setting it to `nil` falls back to the system's local time.
+
+You can optionally supply the following for each task:
 - `args`: the arguments to be passed to the job, as a single argument, a hash, or an array of arguments that can also include kwargs as the last element in the array.
 
 The job in the example configuration above will be enqueued every second as:

UPGRADING.md

@@ -1,3 +1,14 @@
+# Upgrading to version 1.5.x
+Ruby 3.1 is no longer supported, as it reached end-of-life in March 2025. Solid Queue now requires Ruby 3.2 or newer. If you're still on Ruby 3.1, Bundler will continue to resolve solid_queue 1.4.x for you, but you won't receive any new versions until you upgrade Ruby.
+
+Recurring schedules that don't specify a time zone are now interpreted in your application's configured time zone (`config.time_zone`) by default, instead of the system's local time. This only affects schedules without an explicit time zone (e.g. `every day at 9am`); schedules that already include one (e.g. `0 9 * * * America/New_York`) are unaffected.
+
+If your `config.time_zone` differs from the system time where your processes run, recurring jobs may fire at a different wall-clock time than before. To keep the previous behavior, set:
+
+```ruby
+config.solid_queue.time_zone = nil
+```
+
 # Upgrading to version 1.x
 The value returned for `enqueue_after_transaction_commit?` has changed to `true`, and it's no longer configurable. If you want to change this, you need to use Active Job's configuration options.
 

app/models/solid_queue/claimed_execution.rb

@@ -74,7 +74,7 @@ def perform
 
   def release
     SolidQueue.instrument(:release_claimed, job_id: job.id, process_id: process_id) do
-      with_claim do
+      unless_already_finalized do
         job.dispatch_bypassing_concurrency_limits
         destroy!
       end
@@ -86,9 +86,7 @@ def discard
   end
 
   def failed_with(error)
-    finalize_claim do
-      job.failed_with(error)
-    end
+    finalize { job.failed_with(error) }
   end
 
   private
@@ -100,28 +98,24 @@ def execute
     end
 
     def finished
-      finalize_claim do
-        job.finished!
-      end
+      finalize { job.finished! }
     end
 
-    def finalize_claim
-      permit_released = false
-
-      with_claim do
+    def finalize
+      finalized = unless_already_finalized do
         yield
         destroy!
-        # Return the permit inside the claim transaction so only the claim's
-        # owner can return it. Promoting the next blocked job is done after the
-        # transaction commits, so a failure there can't roll back the finished
-        # or failed state of a job that already ran.
-        permit_released = job.release_concurrency_permit
+        true
       end
 
-      job.promote_next_blocked_job if permit_released
+      # Unblock the next job outside the finalize transaction so a failure while
+      # releasing the concurrency lock or dispatching the next job can't roll back
+      # a job that already finished or failed. Only the actor that owned and
+      # finalized the claim gets here, so the lock is released exactly once.
+      job.unblock_next_blocked_job if finalized
     end
 
-    def with_claim
+    def unless_already_finalized
       transaction do
         return false unless self.class.unscoped.lock.find_by(id: id)
 

app/models/solid_queue/job/executable.rb

@@ -43,15 +43,16 @@ def dispatch_all_one_by_one(jobs)
           end
 
           def successfully_dispatched(jobs)
-            dispatched_and_ready(jobs) + dispatched_and_blocked(jobs)
+            jobs_by_id = jobs.index_by(&:id)
+            dispatched_and_ready(jobs_by_id) + dispatched_and_blocked(jobs_by_id)
           end
 
-          def dispatched_and_ready(jobs)
-            where(id: ReadyExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          def dispatched_and_ready(jobs_by_id)
+            ReadyExecution.where(job_id: jobs_by_id.keys).pluck(:job_id).map { |id| jobs_by_id[id] }
           end
 
-          def dispatched_and_blocked(jobs)
-            where(id: BlockedExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+          def dispatched_and_blocked(jobs_by_id)
+            BlockedExecution.where(job_id: jobs_by_id.keys).pluck(:job_id).map { |id| jobs_by_id[id] }
           end
       end
 

app/models/solid_queue/job/schedulable.rb

@@ -23,7 +23,8 @@ def schedule_all_at_once(jobs)
           end
 
           def successfully_scheduled(jobs)
-            where(id: ScheduledExecution.where(job_id: jobs.map(&:id)).pluck(:job_id))
+            jobs_by_id = jobs.index_by(&:id)
+            ScheduledExecution.where(job_id: jobs_by_id.keys).pluck(:job_id).map { |id| jobs_by_id[id] }
           end
       end
 

app/models/solid_queue/queue.rb

@@ -6,9 +6,7 @@ class Queue
 
     class << self
       def all
-        Job.select(:queue_name).distinct.collect do |job|
-          new(job.queue_name)
-        end
+        Job.distinct_values_of(:queue_name).map { |name| new(name) }
       end
 
       def find_by_name(name)

app/models/solid_queue/queue_selector.rb

@@ -43,7 +43,7 @@ def include_all_queues?
       end
 
       def all_queues
-        relation.distinct(:queue_name).pluck(:queue_name)
+        relation.distinct_values_of(:queue_name)
       end
 
       def exact_names
@@ -53,7 +53,7 @@ def exact_names
       def prefixed_names
         if prefixes.empty? then []
         else
-          relation.where(([ "queue_name LIKE ?" ] * prefixes.count).join(" OR "), *prefixes).distinct(:queue_name).pluck(:queue_name)
+          relation.where(([ "queue_name LIKE ?" ] * prefixes.count).join(" OR "), *prefixes).distinct_values_of(:queue_name)
         end
       end
 

app/models/solid_queue/record.rb

@@ -3,6 +3,9 @@
 module SolidQueue
   class Record < ActiveRecord::Base
     self.abstract_class = true
+    self.strict_loading_by_default = false
+
+    include DistinctValues
 
     connects_to(**SolidQueue.connects_to) if SolidQueue.connects_to
 

app/models/solid_queue/record/distinct_values.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Record
+    module DistinctValues
+      extend ActiveSupport::Concern
+
+      # PostgreSQL has no native loose index scan, so a plain DISTINCT on a leading
+      # index column degrades to a full index scan on large tables. We emulate one
+      # with a recursive CTE that walks the index jumping between distinct values.
+      class_methods do
+        def distinct_values_of(column)
+          if loose_index_scan_emulation_needed?
+            loose_distinct_via_recursive_cte(column)
+          else
+            distinct.pluck(column)
+          end
+        end
+
+        private
+          def loose_index_scan_emulation_needed?
+            connection.adapter_name == "PostgreSQL"
+          end
+
+          # Emulates a loose index scan, honoring the current scope (e.g. LIKE prefixes)
+          # by building the anchor and the recursive step as scoped relations, whose
+          # #to_sql inlines any bind parameters so they can be embedded in the raw CTE.
+          def loose_distinct_via_recursive_cte(column)
+            col = connection.quote_column_name(column)
+
+            connection.select_values(<<~SQL.squish)
+              WITH RECURSIVE t AS (
+                (#{next_distinct_value(col, "#{col} IS NOT NULL")})
+                UNION ALL
+                SELECT (#{next_distinct_value(col, "#{col} > t.#{col}")}) FROM t WHERE t.#{col} IS NOT NULL
+              )
+              SELECT #{col} FROM t WHERE #{col} IS NOT NULL
+            SQL
+          end
+
+          # Smallest value of `col` within the current scope that matches `condition`.
+          def next_distinct_value(col, condition)
+            all.where(Arel.sql(condition)).reorder(Arel.sql(col)).limit(1).select(Arel.sql(col)).to_sql
+          end
+      end
+    end
+  end
+end

app/models/solid_queue/recurring_task.rb

@@ -56,16 +56,17 @@ def create_or_update_all(tasks)
       end
     end
 
-    def delay_from_now
-      [ (next_time - Time.current).to_f, 0.1 ].max
+
+    def next_time_after(time)
+      parsed_schedule_with_time_zone.next_time(time).utc
     end
 
     def next_time
-      parsed_schedule.next_time.utc
+      parsed_schedule_with_time_zone.next_time.utc
     end
 
     def previous_time
-      parsed_schedule.previous_time.utc
+      parsed_schedule_with_time_zone.previous_time.utc
     end
 
     def last_enqueued_time
@@ -85,6 +86,7 @@ def enqueue(at:)
 
           perform_later.tap do |job|
             unless job.successfully_enqueued?
+              report_enqueue_error(job.enqueue_error, at: at)
               payload[:enqueue_error] = job.enqueue_error&.message
             end
           end
@@ -97,6 +99,7 @@ def enqueue(at:)
         payload[:skipped] = true
         false
       rescue Job::EnqueueError => error
+        report_enqueue_error(error, at: at)
         payload[:enqueue_error] = error.message
         false
       end
@@ -168,17 +171,40 @@ def arguments_with_kwargs
         end
       end
 
+      def parsed_schedule_with_time_zone
+        @parsed_schedule_with_time_zone ||= apply_default_time_zone_to(parsed_schedule)
+      end
 
       def parsed_schedule
         @parsed_schedule ||= Fugit.parse(schedule, multi: :fail)
       end
 
+      def apply_default_time_zone_to(schedule)
+        if schedule.respond_to?(:zone) && schedule.zone.nil? && default_time_zone.present?
+          Fugit.parse("#{schedule.to_cron_s} #{default_time_zone}", multi: :fail)
+        else
+          schedule
+        end
+      rescue ArgumentError
+        schedule
+      end
+
       def job_class
         @job_class ||= class_name.present? ? class_name.safe_constantize : self.class.default_job_class
       end
 
       def enqueue_options
         { queue: queue_name, priority: priority }.compact
       end
+
+      def default_time_zone
+        SolidQueue.time_zone
+      end
+
+      def report_enqueue_error(error, at:)
+        if error
+          Rails.error.report(error, handled: true, source: "application.solid_queue", context: { task: key, at: at })
+        end
+      end
   end
 end