diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 07b3475f7..30db51a0e 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -26,7 +26,6 @@ jobs: fail-fast: false matrix: ruby-version: - - 3.1 - 3.2 - 3.3 - 3.4 @@ -34,12 +33,6 @@ jobs: database: [ mysql, postgres, sqlite ] gemfile: [ rails_7_1, rails_7_2, rails_8_0, rails_8_1, rails_main ] exclude: - - ruby-version: "3.1" - gemfile: rails_8_0 - - ruby-version: "3.1" - gemfile: rails_8_1 - - ruby-version: "3.1" - gemfile: rails_main - ruby-version: "3.2" gemfile: rails_main services: diff --git a/Appraisals b/Appraisals index 9f3c8df48..303c1cc76 100644 --- a/Appraisals +++ b/Appraisals @@ -1,13 +1,10 @@ # frozen_string_literal: true appraise "rails-7-1" do - # rdoc 6.14 is not compatible with Ruby 3.1 - gem 'rdoc', '6.13' gem "railties", "~> 7.1.0" end appraise "rails-7-2" do - gem 'rdoc', '6.13' gem "railties", "~> 7.2.0" end diff --git a/Gemfile.lock b/Gemfile.lock index d2487e419..0f1783938 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,7 +1,7 @@ PATH remote: . specs: - solid_queue (1.4.0) + solid_queue (1.5.0) activejob (>= 7.1) activerecord (>= 7.1) concurrent-ruby (>= 1.3.7) diff --git a/README.md b/README.md index 589b40a1b..af571847e 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,7 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite, - [Threads, processes, and signals](#threads-processes-and-signals) - [Database configuration](#database-configuration) - [Other configuration settings](#other-configuration-settings) + - [Validating the configuration](#validating-the-configuration) - [Lifecycle hooks](#lifecycle-hooks) - [Errors when enqueuing](#errors-when-enqueuing) - [Concurrency controls](#concurrency-controls) @@ -45,7 +46,7 @@ Solid Queue is configured by default in new Rails 8 applications. If you're runn 1. `bundle add solid_queue` 2. `bin/rails solid_queue:install` -(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.1.6.) +(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.2.) This will configure Solid Queue as the production Active Job backend, create the configuration files `config/queue.yml` and `config/recurring.yml`, and create the `db/queue_schema.rb`. It'll also create a `bin/jobs` executable wrapper that you can use to start Solid Queue. @@ -339,7 +340,7 @@ FROM solid_queue_ready_executions WHERE queue_name LIKE 'beta%'; ``` -This type of `DISTINCT` query on a column that's the leftmost column in an index can be performed very fast in MySQL thanks to a technique called [Loose Index Scan](https://dev.mysql.com/doc/refman/8.0/en/group-by-optimization.html#loose-index-scan). PostgreSQL and SQLite, however, don't implement this technique, which means that if your `solid_queue_ready_executions` table is very big because your queues get very deep, this query will get slow. Normally your `solid_queue_ready_executions` table will be small, but it can happen. +This type of `DISTINCT` query on a column that's the leftmost column in an index can be performed very fast in MySQL thanks to a technique called [Loose Index Scan](https://dev.mysql.com/doc/refman/8.0/en/group-by-optimization.html#loose-index-scan). PostgreSQL doesn't implement this technique natively, so Solid Queue uses a [recursive CTE](https://www.postgresql.org/docs/current/queries-with.html#QUERIES-WITH-RECURSIVE) to emulate it, achieving similar performance by walking the B-tree index and jumping between distinct values. SQLite doesn't implement loose index scan either, but this is unlikely to be a problem since SQLite is typically used in development with small datasets. Similarly to using prefixes, the same will happen if you have paused queues, because we need to get a list of all queues with a query like ```sql @@ -373,6 +374,8 @@ The supervisor is in charge of managing these processes, and it responds to the When receiving a `QUIT` signal, if workers still have jobs in-flight, these will be returned to the queue when the processes are deregistered. +On Windows, the `QUIT` signal can't be trapped, so the supervisor only responds to `TERM` and `INT` there. + If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats. A stale supervised process isn't pruned while its supervisor has a fresh heartbeat, because the supervisor detects actual child exits directly. It becomes eligible for heartbeat-based pruning when its supervisor is also stale. Jobs claimed by a pruned process will be marked as failed with a `SolidQueue::Processes::ProcessPrunedError`. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this. In a similar way, if a worker is terminated in any other way not initiated by the above signals (e.g. a worker is sent a `KILL` signal), jobs in progress will be marked as failed so that they can be inspected, with a `SolidQueue::Processes::ProcessExitError`. Sometimes a job in particular is responsible for this, for example, if it has a memory leak and you have a mechanism to kill processes over a certain memory threshold, so this will help identifying this kind of situation. @@ -409,6 +412,22 @@ There are several settings that control how Solid Queue works that you can set a - `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true — defaults to 1 day. When installing Solid Queue, [a recurring job](#recurring-tasks) is automatically configured to clear finished jobs every hour on the 12th minute in batches. You can edit the `recurring.yml` configuration to change this as you see fit. - `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes. +### Validating the configuration + +You can validate the Solid Queue configuration ahead of time, without starting any process. This is handy in deploy scripts or CI to catch mistakes—a typo in `recurring.yml`, no processes configured, and so on—before they cause a supervisor to boot into a broken state: + +```bash +# Using the bin/jobs binstub +bin/jobs check + +# Or via rake +bin/rails solid_queue:check +``` + +Both commands validate the configuration for the current Rails environment. On success they print `Solid Queue configuration is valid.` and exit `0`; otherwise they print the errors and exit non-zero. When the number of threads is larger than the [database connection pool](#database-configuration), they also print an advisory warning about it—the same one the supervisor logs on boot. They're tolerant of a missing database connection, so they can run on CI or deploy hosts without database credentials. + +`bin/jobs check` accepts the same options as `bin/jobs start` (e.g. `--config_file`, `--recurring_schedule_file`, `--skip-recurring`). The rake task honors the same environment variables Solid Queue already uses: `SOLID_QUEUE_CONFIG`, `SOLID_QUEUE_RECURRING_SCHEDULE`, and `SOLID_QUEUE_SKIP_RECURRING`. To validate a specific environment's configuration, set `RAILS_ENV`, for example `RAILS_ENV=production bin/jobs check`. + ## Lifecycle hooks @@ -472,7 +491,7 @@ Solid Queue extends Active Job with concurrency controls, that allows you to lim ```ruby class MyJob < ApplicationJob - limits_concurrency to: max_concurrent_executions, key: ->(arg1, arg2, **) { ... }, duration: max_interval_to_guarantee_concurrency_limit, group: concurrency_group, on_conflict: on_conflict_behaviour + limits_concurrency to: max_concurrent_executions, key: ->(arg1, arg2, *) { ... }, duration: max_interval_to_guarantee_concurrency_limit, group: concurrency_group, on_conflict: on_conflict_behaviour # ... ``` @@ -766,7 +785,9 @@ production: Tasks are specified as a hash/dictionary, where the key will be the task's key internally. Each task needs to either have a `class`, which will be the job class to enqueue, or a `command`, which will be eval'ed in the context of a job (`SolidQueue::RecurringJob`) that will be enqueued according to its schedule, in the `solid_queue_recurring` queue. -Each task needs to have also a schedule, which is parsed using [Fugit](https://github.com/floraison/fugit), so it accepts anything [that Fugit accepts as a cron](https://github.com/floraison/fugit?tab=readme-ov-file#fugitcron). You can optionally supply the following for each task: +Each task needs to have also a schedule, which is parsed using [Fugit](https://github.com/floraison/fugit), so it accepts anything [that Fugit accepts as a cron](https://github.com/floraison/fugit?tab=readme-ov-file#fugitcron). Schedules can include a time zone (e.g. `0 9 * * * America/New_York` or `every day at 9am America/New_York`). When a schedule doesn't specify one, it's interpreted in the application's configured time zone (`config.time_zone`) by default. You can change or disable this default with `config.solid_queue.time_zone`; setting it to `nil` falls back to the system's local time. + +You can optionally supply the following for each task: - `args`: the arguments to be passed to the job, as a single argument, a hash, or an array of arguments that can also include kwargs as the last element in the array. The job in the example configuration above will be enqueued every second as: diff --git a/UPGRADING.md b/UPGRADING.md index 51ab06a80..544a6482e 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -1,3 +1,14 @@ +# Upgrading to version 1.5.x +Ruby 3.1 is no longer supported, as it reached end-of-life in March 2025. Solid Queue now requires Ruby 3.2 or newer. If you're still on Ruby 3.1, Bundler will continue to resolve solid_queue 1.4.x for you, but you won't receive any new versions until you upgrade Ruby. + +Recurring schedules that don't specify a time zone are now interpreted in your application's configured time zone (`config.time_zone`) by default, instead of the system's local time. This only affects schedules without an explicit time zone (e.g. `every day at 9am`); schedules that already include one (e.g. `0 9 * * * America/New_York`) are unaffected. + +If your `config.time_zone` differs from the system time where your processes run, recurring jobs may fire at a different wall-clock time than before. To keep the previous behavior, set: + +```ruby +config.solid_queue.time_zone = nil +``` + # Upgrading to version 1.x The value returned for `enqueue_after_transaction_commit?` has changed to `true`, and it's no longer configurable. If you want to change this, you need to use Active Job's configuration options. diff --git a/app/models/solid_queue/claimed_execution.rb b/app/models/solid_queue/claimed_execution.rb index f7ee781f9..265692692 100644 --- a/app/models/solid_queue/claimed_execution.rb +++ b/app/models/solid_queue/claimed_execution.rb @@ -74,7 +74,7 @@ def perform def release SolidQueue.instrument(:release_claimed, job_id: job.id, process_id: process_id) do - with_claim do + unless_already_finalized do job.dispatch_bypassing_concurrency_limits destroy! end @@ -86,9 +86,7 @@ def discard end def failed_with(error) - finalize_claim do - job.failed_with(error) - end + finalize { job.failed_with(error) } end private @@ -100,28 +98,24 @@ def execute end def finished - finalize_claim do - job.finished! - end + finalize { job.finished! } end - def finalize_claim - permit_released = false - - with_claim do + def finalize + finalized = unless_already_finalized do yield destroy! - # Return the permit inside the claim transaction so only the claim's - # owner can return it. Promoting the next blocked job is done after the - # transaction commits, so a failure there can't roll back the finished - # or failed state of a job that already ran. - permit_released = job.release_concurrency_permit + true end - job.promote_next_blocked_job if permit_released + # Unblock the next job outside the finalize transaction so a failure while + # releasing the concurrency lock or dispatching the next job can't roll back + # a job that already finished or failed. Only the actor that owned and + # finalized the claim gets here, so the lock is released exactly once. + job.unblock_next_blocked_job if finalized end - def with_claim + def unless_already_finalized transaction do return false unless self.class.unscoped.lock.find_by(id: id) diff --git a/app/models/solid_queue/job/executable.rb b/app/models/solid_queue/job/executable.rb index 32b070d9f..bd3625820 100644 --- a/app/models/solid_queue/job/executable.rb +++ b/app/models/solid_queue/job/executable.rb @@ -43,15 +43,16 @@ def dispatch_all_one_by_one(jobs) end def successfully_dispatched(jobs) - dispatched_and_ready(jobs) + dispatched_and_blocked(jobs) + jobs_by_id = jobs.index_by(&:id) + dispatched_and_ready(jobs_by_id) + dispatched_and_blocked(jobs_by_id) end - def dispatched_and_ready(jobs) - where(id: ReadyExecution.where(job_id: jobs.map(&:id)).pluck(:job_id)) + def dispatched_and_ready(jobs_by_id) + ReadyExecution.where(job_id: jobs_by_id.keys).pluck(:job_id).map { |id| jobs_by_id[id] } end - def dispatched_and_blocked(jobs) - where(id: BlockedExecution.where(job_id: jobs.map(&:id)).pluck(:job_id)) + def dispatched_and_blocked(jobs_by_id) + BlockedExecution.where(job_id: jobs_by_id.keys).pluck(:job_id).map { |id| jobs_by_id[id] } end end diff --git a/app/models/solid_queue/job/schedulable.rb b/app/models/solid_queue/job/schedulable.rb index 8ce2a4a93..04419aa61 100644 --- a/app/models/solid_queue/job/schedulable.rb +++ b/app/models/solid_queue/job/schedulable.rb @@ -23,7 +23,8 @@ def schedule_all_at_once(jobs) end def successfully_scheduled(jobs) - where(id: ScheduledExecution.where(job_id: jobs.map(&:id)).pluck(:job_id)) + jobs_by_id = jobs.index_by(&:id) + ScheduledExecution.where(job_id: jobs_by_id.keys).pluck(:job_id).map { |id| jobs_by_id[id] } end end diff --git a/app/models/solid_queue/queue.rb b/app/models/solid_queue/queue.rb index 7968d3957..60074a878 100644 --- a/app/models/solid_queue/queue.rb +++ b/app/models/solid_queue/queue.rb @@ -6,9 +6,7 @@ class Queue class << self def all - Job.select(:queue_name).distinct.collect do |job| - new(job.queue_name) - end + Job.distinct_values_of(:queue_name).map { |name| new(name) } end def find_by_name(name) diff --git a/app/models/solid_queue/queue_selector.rb b/app/models/solid_queue/queue_selector.rb index 24f6a6ad2..d0d61dd14 100644 --- a/app/models/solid_queue/queue_selector.rb +++ b/app/models/solid_queue/queue_selector.rb @@ -43,7 +43,7 @@ def include_all_queues? end def all_queues - relation.distinct(:queue_name).pluck(:queue_name) + relation.distinct_values_of(:queue_name) end def exact_names @@ -53,7 +53,7 @@ def exact_names def prefixed_names if prefixes.empty? then [] else - relation.where(([ "queue_name LIKE ?" ] * prefixes.count).join(" OR "), *prefixes).distinct(:queue_name).pluck(:queue_name) + relation.where(([ "queue_name LIKE ?" ] * prefixes.count).join(" OR "), *prefixes).distinct_values_of(:queue_name) end end diff --git a/app/models/solid_queue/record.rb b/app/models/solid_queue/record.rb index 8c7000bfe..8617dc85b 100644 --- a/app/models/solid_queue/record.rb +++ b/app/models/solid_queue/record.rb @@ -3,6 +3,9 @@ module SolidQueue class Record < ActiveRecord::Base self.abstract_class = true + self.strict_loading_by_default = false + + include DistinctValues connects_to(**SolidQueue.connects_to) if SolidQueue.connects_to diff --git a/app/models/solid_queue/record/distinct_values.rb b/app/models/solid_queue/record/distinct_values.rb new file mode 100644 index 000000000..59b2545c3 --- /dev/null +++ b/app/models/solid_queue/record/distinct_values.rb @@ -0,0 +1,48 @@ +# frozen_string_literal: true + +module SolidQueue + class Record + module DistinctValues + extend ActiveSupport::Concern + + # PostgreSQL has no native loose index scan, so a plain DISTINCT on a leading + # index column degrades to a full index scan on large tables. We emulate one + # with a recursive CTE that walks the index jumping between distinct values. + class_methods do + def distinct_values_of(column) + if loose_index_scan_emulation_needed? + loose_distinct_via_recursive_cte(column) + else + distinct.pluck(column) + end + end + + private + def loose_index_scan_emulation_needed? + connection.adapter_name == "PostgreSQL" + end + + # Emulates a loose index scan, honoring the current scope (e.g. LIKE prefixes) + # by building the anchor and the recursive step as scoped relations, whose + # #to_sql inlines any bind parameters so they can be embedded in the raw CTE. + def loose_distinct_via_recursive_cte(column) + col = connection.quote_column_name(column) + + connection.select_values(<<~SQL.squish) + WITH RECURSIVE t AS ( + (#{next_distinct_value(col, "#{col} IS NOT NULL")}) + UNION ALL + SELECT (#{next_distinct_value(col, "#{col} > t.#{col}")}) FROM t WHERE t.#{col} IS NOT NULL + ) + SELECT #{col} FROM t WHERE #{col} IS NOT NULL + SQL + end + + # Smallest value of `col` within the current scope that matches `condition`. + def next_distinct_value(col, condition) + all.where(Arel.sql(condition)).reorder(Arel.sql(col)).limit(1).select(Arel.sql(col)).to_sql + end + end + end + end +end diff --git a/app/models/solid_queue/recurring_task.rb b/app/models/solid_queue/recurring_task.rb index 9bb634e6f..40a5531e3 100644 --- a/app/models/solid_queue/recurring_task.rb +++ b/app/models/solid_queue/recurring_task.rb @@ -56,16 +56,17 @@ def create_or_update_all(tasks) end end - def delay_from_now - [ (next_time - Time.current).to_f, 0.1 ].max + + def next_time_after(time) + parsed_schedule_with_time_zone.next_time(time).utc end def next_time - parsed_schedule.next_time.utc + parsed_schedule_with_time_zone.next_time.utc end def previous_time - parsed_schedule.previous_time.utc + parsed_schedule_with_time_zone.previous_time.utc end def last_enqueued_time @@ -85,6 +86,7 @@ def enqueue(at:) perform_later.tap do |job| unless job.successfully_enqueued? + report_enqueue_error(job.enqueue_error, at: at) payload[:enqueue_error] = job.enqueue_error&.message end end @@ -97,6 +99,7 @@ def enqueue(at:) payload[:skipped] = true false rescue Job::EnqueueError => error + report_enqueue_error(error, at: at) payload[:enqueue_error] = error.message false end @@ -168,11 +171,24 @@ def arguments_with_kwargs end end + def parsed_schedule_with_time_zone + @parsed_schedule_with_time_zone ||= apply_default_time_zone_to(parsed_schedule) + end def parsed_schedule @parsed_schedule ||= Fugit.parse(schedule, multi: :fail) end + def apply_default_time_zone_to(schedule) + if schedule.respond_to?(:zone) && schedule.zone.nil? && default_time_zone.present? + Fugit.parse("#{schedule.to_cron_s} #{default_time_zone}", multi: :fail) + else + schedule + end + rescue ArgumentError + schedule + end + def job_class @job_class ||= class_name.present? ? class_name.safe_constantize : self.class.default_job_class end @@ -180,5 +196,15 @@ def job_class def enqueue_options { queue: queue_name, priority: priority }.compact end + + def default_time_zone + SolidQueue.time_zone + end + + def report_enqueue_error(error, at:) + if error + Rails.error.report(error, handled: true, source: "application.solid_queue", context: { task: key, at: at }) + end + end end end diff --git a/app/models/solid_queue/semaphore.rb b/app/models/solid_queue/semaphore.rb index d8caa64ea..7d111ac55 100644 --- a/app/models/solid_queue/semaphore.rb +++ b/app/models/solid_queue/semaphore.rb @@ -32,7 +32,13 @@ def create_unique_by(attributes) class Proxy def self.signal_all(jobs) - Semaphore.where(key: jobs.map(&:concurrency_key)).update_all("value = value + 1") + # Guard against incrementing a semaphore's value beyond its limit. Jobs can + # have different limits, so group them and cap each group with `value < limit`. + jobs.group_by { |job| job.concurrency_limit || 1 }.each do |limit, grouped_jobs| + Semaphore.where(key: grouped_jobs.map(&:concurrency_key)) + .where(value: ...limit) + .update_all("value = value + 1") + end end def initialize(job) diff --git a/gemfiles/rails_7_1.gemfile b/gemfiles/rails_7_1.gemfile index fcb9a6541..036d98368 100644 --- a/gemfiles/rails_7_1.gemfile +++ b/gemfiles/rails_7_1.gemfile @@ -2,7 +2,6 @@ source "https://rubygems.org" -gem "rdoc", "6.13" gem "railties", "~> 7.1.0" gemspec path: "../" diff --git a/gemfiles/rails_7_2.gemfile b/gemfiles/rails_7_2.gemfile index bfd049922..23b503f0c 100644 --- a/gemfiles/rails_7_2.gemfile +++ b/gemfiles/rails_7_2.gemfile @@ -2,7 +2,6 @@ source "https://rubygems.org" -gem "rdoc", "6.13" gem "railties", "~> 7.2.0" gemspec path: "../" diff --git a/lib/active_job/queue_adapters/solid_queue_adapter.rb b/lib/active_job/queue_adapters/solid_queue_adapter.rb index fe5560427..f7e23a1c9 100644 --- a/lib/active_job/queue_adapters/solid_queue_adapter.rb +++ b/lib/active_job/queue_adapters/solid_queue_adapter.rb @@ -8,9 +8,17 @@ module QueueAdapters # # Rails.application.config.active_job.queue_adapter = :solid_queue class SolidQueueAdapter < (Rails::VERSION::MAJOR == 7 && Rails::VERSION::MINOR == 1 ? Object : AbstractAdapter) - class_attribute :stopping, default: false, instance_writer: false + class_attribute :stopping, default: false, instance_writer: false, instance_predicate: false SolidQueue.on_worker_stop { self.stopping = true } + # Accept an optional job argument for compatibility with Rails main, which + # began passing the running job to +queue_adapter.stopping?+ so adapters can + # decide whether to checkpoint based on it. We rely solely on the worker + # shutdown flag, so the argument is ignored. + def stopping?(_job = nil) + self.class.stopping + end + def enqueue_after_transaction_commit? true end diff --git a/lib/generators/solid_queue/install/templates/config/queue.yml b/lib/generators/solid_queue/install/templates/config/queue.yml index 15691e9d2..45866822b 100644 --- a/lib/generators/solid_queue/install/templates/config/queue.yml +++ b/lib/generators/solid_queue/install/templates/config/queue.yml @@ -6,7 +6,7 @@ default: &default - queues: "*" threads: 3 processes: <%%= ENV.fetch("JOB_CONCURRENCY", 1) %> - polling_interval: 0.1 + polling_interval: 1 development: <<: *default diff --git a/lib/solid_queue.rb b/lib/solid_queue.rb index b40da92a3..82b2d457f 100644 --- a/lib/solid_queue.rb +++ b/lib/solid_queue.rb @@ -42,6 +42,15 @@ module SolidQueue mattr_accessor :clear_finished_jobs_after, default: 1.day mattr_accessor :default_concurrency_control_period, default: 3.minutes + mattr_reader :time_zone + + def time_zone=(zone) + @@time_zone = if zone + resolved = zone.respond_to?(:tzinfo) ? zone : ActiveSupport::TimeZone[zone] + resolved&.tzinfo&.name || zone.to_s + end + end + delegate :on_start, :on_stop, :on_exit, to: Supervisor def schedule_recurring_task(key, **options) diff --git a/lib/solid_queue/cli.rb b/lib/solid_queue/cli.rb index 2f3f3e13b..11be9d12a 100644 --- a/lib/solid_queue/cli.rb +++ b/lib/solid_queue/cli.rb @@ -30,5 +30,11 @@ def self.exit_on_failure? def start SolidQueue::Supervisor.start(**options.symbolize_keys) end + + desc :check, "Validates the Solid Queue configuration for the current Rails env without starting anything. Exits non-zero on errors." + def check + configuration = SolidQueue::Configuration.new(**options.symbolize_keys) + exit 1 unless configuration.check + end end end diff --git a/lib/solid_queue/configuration.rb b/lib/solid_queue/configuration.rb index e63a000ca..d04a0aaab 100644 --- a/lib/solid_queue/configuration.rb +++ b/lib/solid_queue/configuration.rb @@ -3,10 +3,12 @@ module SolidQueue class Configuration include ActiveModel::Model + include ActiveModel::Validations::Callbacks - validate :ensure_configured_processes - validate :ensure_valid_recurring_tasks - validate :ensure_correctly_sized_thread_pool + validate :ensure_configured_processes, :ensure_valid_recurring_tasks + validate :warn_about_incorrectly_sized_thread_pool, :warn_about_missing_config_files + + before_validation { warnings.clear } class Process < Struct.new(:kind, :attributes) def instantiate @@ -47,26 +49,32 @@ def configured_processes end end - def error_messages - if configured_processes.none? - "No workers or processed configured. Exiting..." - else - error_messages = invalid_tasks.map do |task| - all_messages = task.errors.full_messages.map { |msg| "\t#{msg}" }.join("\n") - "#{task.key}:\n#{all_messages}" - end - .join("\n") + def mode + options[:mode].to_s.inquiry + end - "Invalid processes configured:\n#{error_messages}" - end + def standalone? + mode.fork? || options[:standalone] end - def mode - @options[:mode].to_s.inquiry + def warnings + @warnings ||= ActiveModel::Errors.new(self) end - def standalone? - mode.fork? || @options[:standalone] + def check + if valid? + warnings.full_messages.each { |warning| $stderr.puts warning } + $stdout.puts "Solid Queue configuration is valid." + + true + else + $stderr.puts "Solid Queue configuration is invalid:" + (warnings.full_messages + errors.full_messages).each do |message| + message.each_line { |line| $stderr.puts " #{line.chomp}" } + end + + false + end end private @@ -88,11 +96,26 @@ def ensure_valid_recurring_tasks end end - def ensure_correctly_sized_thread_pool - if (db_pool_size = SolidQueue::Record.connection_pool&.size) && db_pool_size < estimated_number_of_threads - errors.add(:base, "Solid Queue is configured to use #{estimated_number_of_threads} threads but the " + + def warn_about_incorrectly_sized_thread_pool + db_pool_size = SolidQueue::Record.connection_pool&.size + + if db_pool_size && db_pool_size < estimated_number_of_threads + warnings.add(:base, "Warning: Solid Queue is configured to use #{estimated_number_of_threads} threads but the " \ "database connection pool is #{db_pool_size}. Increase it in `config/database.yml`") end + rescue ActiveRecord::ActiveRecordError + # No usable database connection. Skip the pool-size warning in that case. + end + + def warn_about_missing_config_files + files = [ options[:config_file] ] + files << options[:recurring_schedule_file] unless skip_recurring_tasks? + + files.compact.each do |file| + unless Pathname.new(file).exist? + warnings.add(:base, "Warning: provided configuration file '#{file}' does not exist. Falling back to default configuration.") + end + end end def default_options @@ -221,7 +244,6 @@ def load_config_from_file(file) if file.exist? ActiveSupport::ConfigurationFile.parse(file).deep_symbolize_keys else - puts "[solid_queue] WARNING: Provided configuration file '#{file}' does not exist. Falling back to default configuration." {} end end diff --git a/lib/solid_queue/engine.rb b/lib/solid_queue/engine.rb index 8daffe0e1..dbe370ff3 100644 --- a/lib/solid_queue/engine.rb +++ b/lib/solid_queue/engine.rb @@ -16,6 +16,12 @@ class Engine < ::Rails::Engine end end + initializer "solid_queue.time_zone" do |app| + unless config.solid_queue.key?(:time_zone) + SolidQueue.time_zone = app.config.time_zone + end + end + initializer "solid_queue.app_executor", before: :run_prepare_callbacks do |app| config.solid_queue.app_executor ||= app.executor config.solid_queue.on_thread_error ||= ->(exception) { Rails.error.report(exception, handled: false) } diff --git a/lib/solid_queue/processes/runnable.rb b/lib/solid_queue/processes/runnable.rb index c6e002e4f..7738391cf 100644 --- a/lib/solid_queue/processes/runnable.rb +++ b/lib/solid_queue/processes/runnable.rb @@ -4,7 +4,9 @@ module SolidQueue::Processes module Runnable include Supervised - attr_writer :mode + def mode=(value) + @mode = (value || DEFAULT_MODE).to_s.inquiry + end def start run_in_mode do @@ -33,7 +35,7 @@ def alive? DEFAULT_MODE = :async def mode - (@mode || DEFAULT_MODE).to_s.inquiry + @mode ||= DEFAULT_MODE.to_s.inquiry end def run_in_mode(&block) diff --git a/lib/solid_queue/scheduler/recurring_schedule.rb b/lib/solid_queue/scheduler/recurring_schedule.rb index a1e2409e0..526a8edcd 100644 --- a/lib/solid_queue/scheduler/recurring_schedule.rb +++ b/lib/solid_queue/scheduler/recurring_schedule.rb @@ -33,8 +33,8 @@ def schedule_tasks end end - def schedule_task(task) - scheduled_tasks[task.key] = schedule(task) + def schedule_task(task, run_at: task.next_time) + scheduled_tasks[task.key] = schedule(task, run_at: run_at) end def unschedule_tasks @@ -99,9 +99,11 @@ def load_dynamic_tasks dynamic_tasks_enabled? ? RecurringTask.dynamic.to_a : [] end - def schedule(task) - scheduled_task = Concurrent::ScheduledTask.new(task.delay_from_now, args: [ self, task, task.next_time ]) do |thread_schedule, thread_task, thread_task_run_at| - thread_schedule.schedule_task(thread_task) + def schedule(task, run_at: task.next_time) + delay = [ (run_at - Time.current).to_f, 0.1 ].max + + scheduled_task = Concurrent::ScheduledTask.new(delay, args: [ self, task, run_at ]) do |thread_schedule, thread_task, thread_task_run_at| + thread_schedule.schedule_task(thread_task, run_at: thread_task.next_time_after(thread_task_run_at)) wrap_in_app_executor do thread_task.enqueue(at: thread_task_run_at) diff --git a/lib/solid_queue/supervisor.rb b/lib/solid_queue/supervisor.rb index ae17ec95c..17b90c544 100644 --- a/lib/solid_queue/supervisor.rb +++ b/lib/solid_queue/supervisor.rb @@ -13,6 +13,8 @@ def start(**options) configuration = Configuration.new(**options) if configuration.valid? + configuration.warnings.full_messages.each { |warning| SolidQueue.logger.warn(warning) } + klass = configuration.mode.fork? ? ForkSupervisor : AsyncSupervisor klass.new(configuration).tap(&:start) else diff --git a/lib/solid_queue/supervisor/signals.rb b/lib/solid_queue/supervisor/signals.rb index 7bee107d5..c7731eb73 100644 --- a/lib/solid_queue/supervisor/signals.rb +++ b/lib/solid_queue/supervisor/signals.rb @@ -11,7 +11,7 @@ module Signals end private - SIGNALS = %i[ QUIT INT TERM ] + SIGNALS = Gem.win_platform? ? %i[ INT TERM ] : %i[ QUIT INT TERM ] def register_signal_handlers SIGNALS.each do |signal| diff --git a/lib/solid_queue/tasks.rb b/lib/solid_queue/tasks.rb index 91cd778bf..704225620 100644 --- a/lib/solid_queue/tasks.rb +++ b/lib/solid_queue/tasks.rb @@ -8,4 +8,10 @@ task start: :environment do SolidQueue::Supervisor.start end + + desc "validate the Solid Queue configuration for the current Rails env without starting any process" + task check: :environment do + configuration = SolidQueue::Configuration.new + exit 1 unless configuration.check + end end diff --git a/lib/solid_queue/version.rb b/lib/solid_queue/version.rb index 8514496c3..da6b7c906 100644 --- a/lib/solid_queue/version.rb +++ b/lib/solid_queue/version.rb @@ -1,3 +1,3 @@ module SolidQueue - VERSION = "1.4.0" + VERSION = "1.5.0" end diff --git a/solid_queue.gemspec b/solid_queue.gemspec index c2c11d118..f7bf8e37b 100644 --- a/solid_queue.gemspec +++ b/solid_queue.gemspec @@ -22,7 +22,7 @@ Gem::Specification.new do |spec| Dir["{app,config,db,lib}/**/*", "MIT-LICENSE", "README.md", "UPGRADING.md"] end - spec.required_ruby_version = '>= 3.1' + spec.required_ruby_version = '>= 3.2' rails_version = ">= 7.1" spec.add_dependency "activerecord", rails_version diff --git a/test/dummy/config/application.rb b/test/dummy/config/application.rb index 8fa76e8c1..c242feb3b 100644 --- a/test/dummy/config/application.rb +++ b/test/dummy/config/application.rb @@ -28,5 +28,7 @@ class Application < Rails::Application # config.eager_load_paths << Rails.root.join("extras") config.active_job.queue_adapter = :solid_queue + + config.active_record.strict_loading_by_default = true end end diff --git a/test/integration/async_processes_lifecycle_test.rb b/test/integration/async_processes_lifecycle_test.rb index fd284210e..6f269e220 100644 --- a/test/integration/async_processes_lifecycle_test.rb +++ b/test/integration/async_processes_lifecycle_test.rb @@ -22,7 +22,7 @@ class AsyncProcessesLifecycleTest < ActiveSupport::TestCase wait_for_jobs_to_finish_for(2.seconds) - assert_equal 12, JobResult.count + assert_equal 12, skip_active_record_query_cache { JobResult.count } 6.times { |i| assert_completed_job_results("job_#{i}", :background) } 6.times { |i| assert_completed_job_results("job_#{i}", :default) } @@ -58,7 +58,7 @@ class AsyncProcessesLifecycleTest < ActiveSupport::TestCase signal_process(@pid, :TERM, wait: 0.1.second) end - sleep(1.second) + wait_while_with_timeout(SolidQueue.shutdown_timeout + 1.second) { process_exists?(@pid) } assert_clean_termination end @@ -212,15 +212,17 @@ def enqueue_store_result_job(value, queue_name = :background, **options) end def assert_completed_job_results(value, queue_name = :background, count = 1) - skip_active_record_query_cache do - assert_equal count, JobResult.where(queue_name: queue_name, status: "completed", value: value).count - end + actual = skip_active_record_query_cache { + JobResult.where(queue_name: queue_name, status: "completed", value: value).count + } + assert_equal count, actual end def assert_started_job_result(value, queue_name = :background, count = 1) - skip_active_record_query_cache do - assert_equal count, JobResult.where(queue_name: queue_name, status: "started", value: value).count - end + actual = skip_active_record_query_cache { + JobResult.where(queue_name: queue_name, status: "started", value: value).count + } + assert_equal count, actual end def assert_job_status(active_job, status) diff --git a/test/integration/concurrency_controls_test.rb b/test/integration/concurrency_controls_test.rb index f5cd1d366..51d73dd9f 100644 --- a/test/integration/concurrency_controls_test.rb +++ b/test/integration/concurrency_controls_test.rb @@ -6,18 +6,27 @@ class ConcurrencyControlsTest < ActiveSupport::TestCase self.use_transactional_tests = false setup do - @result = JobResult.create!(queue_name: "default", status: "") + # Previous tests may leave forked workers briefly alive; those can still write to + # JobResult rows whose primary keys get reused by create! below (e.g. overwriting + # status with StoreResultJob's default "completed"). + wait_for_registered_processes(0, timeout: 5.seconds) + destroy_records default_worker = { queues: "default", polling_interval: 0.1, processes: 3, threads: 2 } dispatcher = { polling_interval: 0.1, batch_size: 200, concurrency_maintenance_interval: 1 } @pid = run_supervisor_as_fork(workers: [ default_worker ], dispatchers: [ dispatcher ]) + wait_for_registered_processes(5, timeout: 3.seconds) # 3 workers + dispatcher + supervisor - wait_for_registered_processes(5, timeout: 0.5.second) # 3 workers working the default queue + dispatcher + supervisor + @result = JobResult.create!(queue_name: "default", status: "") end teardown do - terminate_process(@pid) if process_exists?(@pid) + if @pid && process_exists?(@pid) + terminate_process(@pid) + end + wait_for_registered_processes(0, timeout: 5.seconds) + destroy_records end test "run several conflicting jobs over the same record without overlapping" do @@ -36,51 +45,53 @@ class ConcurrencyControlsTest < ActiveSupport::TestCase end test "schedule several conflicting jobs over the same record sequentially" do - # Writes to @result at 0.4s - UpdateResultJob.set(wait: 0.2.seconds).perform_later(@result, name: "000", pause: 0.2.seconds) - - ("A".."F").each_with_index do |name, i| - # "A" is enqueued at 0.2s and writes to @result at 0.6s, the write at 0.4s gets overwritten - NonOverlappingUpdateResultJob.set(wait: (0.2 + i * 0.1).seconds).perform_later(@result, name: name, pause: 0.4.seconds) + # "000" isn't concurrency-limited, so it runs alongside A. Both read @result + # while it's still empty; "000" writes "s000c000" partway through A's run, but + # A pauses much longer and saves last of the two, overwriting it. A is + # scheduled well ahead of B–K so it reliably holds the semaphore first, and + # the rest of the chain builds only on A's clean write — so "000" never + # survives in the final result. + UpdateResultJob.set(wait: 0.1.seconds).perform_later(@result, name: "000", pause: 1.second) + + NonOverlappingUpdateResultJob.set(wait: 0.1.seconds).perform_later(@result, name: "A", pause: 2.5.seconds) + + ("B".."F").each_with_index do |name, i| + NonOverlappingUpdateResultJob.set(wait: (1 + i * 0.1).seconds).perform_later(@result, name: name, pause: 0.1.seconds) end ("G".."K").each_with_index do |name, i| - NonOverlappingUpdateResultJob.set(wait: (1 + i * 0.1).seconds).perform_later(@result, name: name) + NonOverlappingUpdateResultJob.set(wait: (1.5 + i * 0.1).seconds).perform_later(@result, name: name) end - wait_for_jobs_to_finish_for(5.seconds) + wait_for_jobs_to_finish_for(15.seconds) assert_no_unfinished_jobs assert_stored_sequence @result, ("A".."K").to_a end test "run several jobs over the same record limiting concurrency" do - incr = 0 - # C is the last one to update the record - # A: 0 to 0.5 - # B: 0 to 1.0 - # C: 0 to 1.5 + # ThrottledUpdateResultJob has a concurrency limit of 3, so A, B and C run + # together — all reading @result while it's still empty — and D–H block. A + # and B finish quickly, freeing slots that drain D–H; C reads the empty + # status and pauses far longer than everyone else, so it saves last and its + # write (built on the empty status) overwrites all the others, leaving "C". assert_no_difference -> { SolidQueue::BlockedExecution.count } do - ("A".."C").each do |name| - ThrottledUpdateResultJob.perform_later(@result, name: name, pause: (0.5 + incr).seconds) - incr += 0.5 - end + ThrottledUpdateResultJob.perform_later(@result, name: "A", pause: 0.5.seconds) + ThrottledUpdateResultJob.perform_later(@result, name: "B", pause: 0.5.seconds) + ThrottledUpdateResultJob.perform_later(@result, name: "C", pause: 3.seconds) end - sleep(0.01) # To ensure these aren't picked up before ABC - # D to H: 0.51 to 0.76 (starting after A finishes, and in order, 5 * 0.05 = 0.25) - # These would finish all before B and C + wait_for(timeout: 2.seconds) { SolidQueue::ClaimedExecution.count >= 3 } + assert_difference -> { SolidQueue::BlockedExecution.count }, +5 do ("D".."H").each do |name| - ThrottledUpdateResultJob.perform_later(@result, name: name, pause: 0.05.seconds) + ThrottledUpdateResultJob.perform_later(@result, name: name, pause: 0.01.seconds) end end - wait_for_jobs_to_finish_for(5.seconds) + wait_for_jobs_to_finish_for(15.seconds) assert_no_unfinished_jobs - # C would have started in the beginning, seeing the status empty, and would finish after - # all other jobs, so it'll do the last update with only itself assert_stored_sequence(@result, [ "C" ]) end @@ -94,7 +105,7 @@ class ConcurrencyControlsTest < ActiveSupport::TestCase NonOverlappingUpdateResultJob.perform_later(@result, name: name) end - wait_for_jobs_to_finish_for(5.seconds) + wait_for_jobs_to_finish_for(15.seconds) assert_equal 3, SolidQueue::FailedExecution.count assert_stored_sequence @result, [ "B", "D", "F" ] + ("G".."K").to_a @@ -166,12 +177,11 @@ class ConcurrencyControlsTest < ActiveSupport::TestCase NonOverlappingUpdateResultJob.perform_later(@result, name: "I'll be released to ready", pause: SolidQueue.shutdown_timeout + 10.seconds) job = SolidQueue::Job.last - sleep(0.2) - assert job.claimed? + wait_for(timeout: 2.seconds) { job.reload.claimed? } - # This won't leave time to the job to finish - signal_process(@pid, :TERM, wait: 0.1.second) - sleep(SolidQueue.shutdown_timeout + 0.6.seconds) + # This won't leave time to the job to finish, so the worker should + # release it back to ready during shutdown. + terminate_process(@pid) assert_not job.reload.finished? assert job.reload.ready? @@ -196,8 +206,8 @@ class ConcurrencyControlsTest < ActiveSupport::TestCase end test "discard jobs when concurrency limit is reached with on_conflict: :discard" do - job1 = DiscardableUpdateResultJob.perform_later(@result, name: "1", pause: 3) - sleep(0.1) + job1 = DiscardableUpdateResultJob.perform_later(@result, name: "1", pause: 1.second) + wait_for(timeout: 2.seconds) { SolidQueue::Job.find_by(active_job_id: job1.job_id)&.claimed? } # should be discarded due to concurrency limit job2 = DiscardableUpdateResultJob.perform_later(@result, name: "2") @@ -278,9 +288,8 @@ class ConcurrencyControlsTest < ActiveSupport::TestCase private def assert_stored_sequence(result, sequence) expected = sequence.sort.map { |name| "s#{name}c#{name}" }.join - skip_active_record_query_cache do - assert_equal expected, result.reload.status.split(" + ").sort.join - end + actual = skip_active_record_query_cache { result.reload.status.split(" + ").sort.join } + assert_equal expected, actual end def wait_for_semaphores_to_be_released_for(timeout) diff --git a/test/integration/forked_processes_lifecycle_test.rb b/test/integration/forked_processes_lifecycle_test.rb index f104d019b..8a2387cfb 100644 --- a/test/integration/forked_processes_lifecycle_test.rb +++ b/test/integration/forked_processes_lifecycle_test.rb @@ -22,7 +22,7 @@ class ForkedProcessesLifecycleTest < ActiveSupport::TestCase wait_for_jobs_to_finish_for(2.seconds) - assert_equal 12, JobResult.count + assert_equal 12, skip_active_record_query_cache { JobResult.count } 6.times { |i| assert_completed_job_results("job_#{i}", :background) } 6.times { |i| assert_completed_job_results("job_#{i}", :default) } @@ -124,7 +124,12 @@ class ForkedProcessesLifecycleTest < ActiveSupport::TestCase no_pause = enqueue_store_result_job("no pause") pause = enqueue_store_result_job("pause", pause: SolidQueue.shutdown_timeout + 10.seconds) - wait_while_with_timeout(1.second) { SolidQueue::ReadyExecution.count > 1 } + wait_while_with_timeout(5.seconds) { + SolidQueue::ReadyExecution.joins(:job).exists?(solid_queue_jobs: { active_job_id: pause.job_id }) + } + wait_while_with_timeout(5.seconds) { + !JobResult.exists?(status: "started", value: "pause") + } signal_process(@pid, :TERM, wait: 0.5.second) wait_for_jobs_to_finish_for(2.seconds, except: pause) @@ -167,19 +172,18 @@ class ForkedProcessesLifecycleTest < ActiveSupport::TestCase end test "process a job that exits" do - 2.times do - enqueue_store_result_job("no exit", :background) - enqueue_store_result_job("no exit", :default) - end + # Enqueue all four "no exit" background jobs ahead of the exiting job. Jobs are + # claimed in enqueue order, so these are always claimed no later than exit_job + # and — being pauseless — finish before the worker exits, completing normally. + 4.times { enqueue_store_result_job("no exit", :background) } + 2.times { enqueue_store_result_job("no exit", :default) } enqueue_store_result_job("paused no exit", :default, pause: 0.5.second) + # the worker for :background queue will exit abnormally exit_job = enqueue_store_result_job("exit", :background, exit_value: 1, pause: 0.5.second) - # this will run *after* exit_job (pause: 1.second) - should also be marked as failed + # claimed alongside exit_job and still paused when it exits, so it's failed too pause_job = enqueue_store_result_job("exit", :background, pause: 1.second) - # this will run *before* exit_job (no pause) - should complete normally - 2.times { enqueue_store_result_job("no exit", :background) } - wait_for_jobs_to_finish_for(3.seconds, except: [ exit_job, pause_job ]) assert_completed_job_results("no exit", :default, 2) @@ -236,6 +240,10 @@ class ForkedProcessesLifecycleTest < ActiveSupport::TestCase enqueue_store_result_job("pause", :default, pause: 0.5.seconds) wait_for_jobs_to_finish_for(1.second, except: [ killed_pause ]) + # Ensure the long job has written its "started" row before we SIGKILL the worker. + wait_while_with_timeout(2.seconds) do + JobResult.where(status: "started", value: "killed_pause").none? + end worker = find_processes_registered_as("Worker").detect { |process| process.metadata["queues"].include? "background" } signal_process(worker.pid, :KILL, wait: 0.5.seconds) @@ -307,15 +315,17 @@ def enqueue_store_result_job(value, queue_name = :background, **options) end def assert_completed_job_results(value, queue_name = :background, count = 1) - skip_active_record_query_cache do - assert_equal count, JobResult.where(queue_name: queue_name, status: "completed", value: value).count - end + actual = skip_active_record_query_cache { + JobResult.where(queue_name: queue_name, status: "completed", value: value).count + } + assert_equal count, actual end def assert_started_job_result(value, queue_name = :background, count = 1) - skip_active_record_query_cache do - assert_equal count, JobResult.where(queue_name: queue_name, status: "started", value: value).count - end + actual = skip_active_record_query_cache { + JobResult.where(queue_name: queue_name, status: "started", value: value).count + } + assert_equal count, actual end def assert_job_status(active_job, status) diff --git a/test/integration/jobs_lifecycle_test.rb b/test/integration/jobs_lifecycle_test.rb index 8444c3759..4b23de296 100644 --- a/test/integration/jobs_lifecycle_test.rb +++ b/test/integration/jobs_lifecycle_test.rb @@ -40,11 +40,12 @@ class JobsLifecycleTest < ActiveSupport::TestCase @dispatcher.start @worker.start - wait_for_jobs_to_finish_for(3.seconds) + wait_while_with_timeout(3.seconds) { SolidQueue::FailedExecution.count < 2 } message = "raised ExpectedTestError for the 1st time" assert_equal [ "A: #{message}", "B: #{message}" ], JobBuffer.values.sort + assert_equal 2, SolidQueue::FailedExecution.count assert_empty SolidQueue::Job.finished end diff --git a/test/integration/puma/plugin_testing.rb b/test/integration/puma/plugin_testing.rb index 14165c9b8..b5a5c35af 100644 --- a/test/integration/puma/plugin_testing.rb +++ b/test/integration/puma/plugin_testing.rb @@ -1,6 +1,7 @@ # frozen_string_literal: true require "test_helper" +require "socket" module PluginTesting extend ActiveSupport::Concern @@ -12,10 +13,12 @@ module PluginTesting setup do FileUtils.mkdir_p Rails.root.join("tmp", "pids") + port = find_available_port + Dir.chdir("test/dummy") do cmd = %W[ bundle exec puma - -b tcp://127.0.0.1:9222 + -b tcp://127.0.0.1:#{port} -C config/puma_#{solid_queue_mode}.rb -s config.ru @@ -57,4 +60,11 @@ module PluginTesting def solid_queue_mode raise NotImplementedError end + + def find_available_port + server = TCPServer.new("127.0.0.1", 0) + server.addr[1] + ensure + server&.close + end end diff --git a/test/integration/recurring_tasks_test.rb b/test/integration/recurring_tasks_test.rb index f2fc7145f..c535ff058 100644 --- a/test/integration/recurring_tasks_test.rb +++ b/test/integration/recurring_tasks_test.rb @@ -14,8 +14,10 @@ class RecurringTasksTest < ActiveSupport::TestCase end test "enqueue and process periodic tasks" do - wait_for_jobs_to_be_enqueued(2, timeout: 2.5.seconds) - wait_for_jobs_to_finish_for(2.5.seconds) + wait_for_jobs_to_be_enqueued(2, timeout: 5.seconds) + wait_while_with_timeout(5.seconds) do + JobResult.where(status: "custom_status", value: "42").count < 2 + end skip_active_record_query_cache do assert SolidQueue::Job.count >= 2 @@ -23,13 +25,12 @@ class RecurringTasksTest < ActiveSupport::TestCase assert_equal "periodic_store_result", job.recurring_execution.task_key assert_equal "StoreResultJob", job.class_name end - - assert JobResult.count >= 2 - JobResult.all.each do |result| - assert_equal "custom_status", result.status - assert_equal "42", result.value - end end + + # Scope to this recurring task's results. Other tests may leave JobResult + # rows (e.g. status "completed") that would fail a JobResult.all assertion. + results = skip_active_record_query_cache { JobResult.where(status: "custom_status", value: "42").to_a } + assert_operator results.size, :>=, 2 end test "persist and delete configured tasks" do diff --git a/test/models/solid_queue/claimed_execution_concurrency_test.rb b/test/models/solid_queue/claimed_execution_concurrency_test.rb index b48b68376..1847d49f3 100644 --- a/test/models/solid_queue/claimed_execution_concurrency_test.rb +++ b/test/models/solid_queue/claimed_execution_concurrency_test.rb @@ -22,9 +22,8 @@ class SolidQueue::ClaimedExecutionConcurrencyTest < ActiveSupport::TestCase in_transaction: ->(locked_claim) do first_job.failed_with(SolidQueue::Processes::ProcessPrunedError.new(1.day.ago)) locked_claim.destroy! - first_job.release_concurrency_permit end, - after_commit: -> { first_job.promote_next_blocked_job }) do + after_commit: -> { first_job.unblock_next_blocked_job }) do claimed_execution.perform end @@ -46,9 +45,8 @@ class SolidQueue::ClaimedExecutionConcurrencyTest < ActiveSupport::TestCase in_transaction: ->(locked_claim) do first_job.failed_with(SolidQueue::Processes::ProcessPrunedError.new(1.day.ago)) locked_claim.destroy! - first_job.release_concurrency_permit end, - after_commit: -> { first_job.promote_next_blocked_job }) do + after_commit: -> { first_job.unblock_next_blocked_job }) do assert_raises(RuntimeError) { claimed_execution.perform } end @@ -70,9 +68,8 @@ class SolidQueue::ClaimedExecutionConcurrencyTest < ActiveSupport::TestCase in_transaction: ->(locked_claim) do first_job.failed_with(SolidQueue::Processes::ProcessPrunedError.new(1.day.ago)) locked_claim.destroy! - first_job.release_concurrency_permit end, - after_commit: -> { first_job.promote_next_blocked_job }) do + after_commit: -> { first_job.unblock_next_blocked_job }) do claimed_execution.release end @@ -94,9 +91,8 @@ class SolidQueue::ClaimedExecutionConcurrencyTest < ActiveSupport::TestCase in_transaction: ->(locked_claim) do first_job.finished! locked_claim.destroy! - first_job.release_concurrency_permit end, - after_commit: -> { first_job.promote_next_blocked_job }) do + after_commit: -> { first_job.unblock_next_blocked_job }) do @process.prune end @@ -135,9 +131,10 @@ class SolidQueue::ClaimedExecutionConcurrencyTest < ActiveSupport::TestCase holding_blocked_row.wait(5) - # finalize returns the permit and commits before it tries to promote, so it - # never holds the semaphore lock while waiting on the blocked row. Under the - # old in-transaction promotion this reverse lock order could deadlock. + # finalize releases the concurrency lock and commits before it tries to + # release the next blocked job, so it never holds the semaphore lock while + # waiting on the blocked row. Releasing the blocked job inside the claim + # transaction would reverse the lock order and could deadlock. assert_nothing_raised do Timeout.timeout(10) do finisher = Thread.new do diff --git a/test/models/solid_queue/claimed_execution_test.rb b/test/models/solid_queue/claimed_execution_test.rb index 148eef42d..72a988f69 100644 --- a/test/models/solid_queue/claimed_execution_test.rb +++ b/test/models/solid_queue/claimed_execution_test.rb @@ -17,7 +17,7 @@ class SolidQueue::ClaimedExecutionTest < ActiveSupport::TestCase assert job.reload.finished? end - test "stale performer cannot return a concurrency permit after its claim is pruned" do + test "stale performer cannot release a concurrency lock after its claim is pruned" do job_result = JobResult.create!(queue_name: "default", status: "") first_active_job = NonOverlappingUpdateResultJob.perform_later(job_result, name: "A") NonOverlappingUpdateResultJob.perform_later(job_result, name: "B") diff --git a/test/models/solid_queue/job_test.rb b/test/models/solid_queue/job_test.rb index 7c3bd6c06..e11911408 100644 --- a/test/models/solid_queue/job_test.rb +++ b/test/models/solid_queue/job_test.rb @@ -352,7 +352,7 @@ class DiscardableNonOverlappingGroupedJob2 < NonOverlappingJob job = SolidQueue::Job.last worker = SolidQueue::Worker.new(queues: "background").tap(&:start) - sleep(0.2) + wait_while_with_timeout(2.seconds) { !job.reload.claimed? } assert_no_difference -> { SolidQueue::Job.count }, -> { SolidQueue::ClaimedExecution.count } do assert_raises SolidQueue::Execution::UndiscardableError do diff --git a/test/models/solid_queue/ready_execution_test.rb b/test/models/solid_queue/ready_execution_test.rb index dd9269ca1..a5f40659a 100644 --- a/test/models/solid_queue/ready_execution_test.rb +++ b/test/models/solid_queue/ready_execution_test.rb @@ -168,6 +168,40 @@ class SolidQueue::ReadyExecutionTest < ActiveSupport::TestCase claimed_jobs.map(&:queue_name) end + test "distinct_values_of returns all distinct queue names" do + AddToBufferJob.perform_later("hey") # goes to background queue + + names = SolidQueue::ReadyExecution.distinct_values_of(:queue_name) + assert_includes names, "backend" + assert_includes names, "background" + assert_equal 2, names.size + end + + test "distinct_values_of honors a chained where" do + AddToBufferJob.perform_later("hey") # background queue + + names = SolidQueue::ReadyExecution.where("queue_name LIKE ?", "back%").distinct_values_of(:queue_name) + assert_includes names, "backend" + assert_includes names, "background" + assert_equal 2, names.size + + names = SolidQueue::ReadyExecution.where("queue_name LIKE ?", "backe%").distinct_values_of(:queue_name) + assert_equal [ "backend" ], names + end + + test "distinct_values_of returns empty array for no matches" do + names = SolidQueue::ReadyExecution.where("queue_name LIKE ?", "nonexistent%").distinct_values_of(:queue_name) + assert_equal [], names + end + + test "distinct_values_of returns empty array on empty table" do + SolidQueue::ReadyExecution.delete_all + SolidQueue::Job.delete_all + + names = SolidQueue::ReadyExecution.distinct_values_of(:queue_name) + assert_equal [], names + end + test "discard all" do 3.times { |i| AddToBufferJob.perform_later(i) } diff --git a/test/models/solid_queue/recurring_task_test.rb b/test/models/solid_queue/recurring_task_test.rb index dba9d6b91..518fc7ed8 100644 --- a/test/models/solid_queue/recurring_task_test.rb +++ b/test/models/solid_queue/recurring_task_test.rb @@ -203,6 +203,18 @@ def perform assert_equal 4, job.priority end + test "next_time_after returns the next occurrence after the given time" do + task = recurring_task_with(class_name: "JobWithoutArguments", schedule: "every minute") + + # next_time_after a time exactly on the minute boundary should return the following minute + time = Time.utc(2026, 3, 12, 1, 28, 0) + assert_equal Time.utc(2026, 3, 12, 1, 29, 0), task.next_time_after(time) + + # next_time_after a time just before the boundary should return that boundary + time = Time.utc(2026, 3, 12, 1, 27, 59) + assert_equal Time.utc(2026, 3, 12, 1, 28, 0), task.next_time_after(time) + end + test "task configured with a command" do task = recurring_task_with(command: "JobBuffer.add('from_a_command')") enqueue_and_assert_performed_with_result(task, "from_a_command") @@ -260,7 +272,90 @@ def perform end end + test "schedule without a time zone uses the configured default time zone" do + with_time_zone "America/New_York" do + task = recurring_task_with(class_name: "JobWithoutArguments", schedule: "every day at 9am") + assert_equal Fugit.parse("0 9 * * * America/New_York").next_time.utc, task.next_time + end + end + + test "schedule with an explicit time zone ignores the configured default time zone" do + with_time_zone "America/New_York" do + task = recurring_task_with(class_name: "JobWithoutArguments", schedule: "0 9 * * * Europe/Madrid") + assert_equal Fugit.parse("0 9 * * * Europe/Madrid").next_time.utc, task.next_time + end + end + + test "schedule falls back to local time when no default time zone is configured" do + with_time_zone nil do + task = recurring_task_with(class_name: "JobWithoutArguments", schedule: "every day at 9am") + assert_equal Fugit.parse("0 9 * * *").next_time.utc, task.next_time + end + end + + test "configured default time zone accepts Rails time zone names" do + with_time_zone "Eastern Time (US & Canada)" do + task = recurring_task_with(class_name: "JobWithoutArguments", schedule: "every day at 9am") + assert_equal Fugit.parse("0 9 * * * America/New_York").next_time.utc, task.next_time + end + end + + test "default time zone doesn't change the displayed schedule" do + with_time_zone "America/New_York" do + task = recurring_task_with(class_name: "JobWithoutArguments", schedule: "every day at 9am") + assert task.to_s.ends_with? "[ 0 9 * * * ]" + end + end + + test "reports Job::EnqueueError to Rails.error when enqueuing via Solid Queue" do + SolidQueue::Job.stubs(:create!).raises(ActiveRecord::Deadlocked) + subscriber = ErrorBuffer.new + at = Time.now + + with_error_subscriber(subscriber) do + task = recurring_task_with(class_name: "JobWithoutArguments") + task.enqueue(at: at) + end + + assert_equal 1, subscriber.errors.count + error, options = subscriber.errors.first + assert_kind_of SolidQueue::Job::EnqueueError, error + assert_match "ActiveRecord::Deadlocked", error.message + assert_equal true, options[:handled] + assert_equal "application.solid_queue", options[:source] + assert_equal "task-id", options[:context][:task] + assert_equal at, options[:context][:at] + end + + test "reports enqueue error to Rails.error when using another adapter" do + ActiveJob::QueueAdapters::AsyncAdapter.any_instance.stubs(:enqueue).raises(ActiveJob::EnqueueError.new("All is broken")) + subscriber = ErrorBuffer.new + at = Time.now + + with_error_subscriber(subscriber) do + task = recurring_task_with(class_name: "JobUsingAsyncAdapter") + task.enqueue(at: at) + end + + assert_equal 1, subscriber.errors.count + error, options = subscriber.errors.first + assert_kind_of ActiveJob::EnqueueError, error + assert_equal "All is broken", error.message + assert_equal true, options[:handled] + assert_equal "application.solid_queue", options[:source] + assert_equal "task-id", options[:context][:task] + assert_equal at, options[:context][:at] + end + private + def with_time_zone(zone) + previous = SolidQueue.time_zone + SolidQueue.time_zone = zone + yield + ensure + SolidQueue.time_zone = previous + end + def enqueue_and_assert_performed_with_result(task, result) assert_difference [ -> { SolidQueue::Job.count }, -> { SolidQueue::ReadyExecution.count } ], +1 do task.enqueue(at: Time.now) @@ -289,4 +384,11 @@ def run_all_jobs_inline worker.start end end + + def with_error_subscriber(subscriber) + Rails.error.subscribe(subscriber) + yield + ensure + Rails.error.unsubscribe(subscriber) if Rails.error.respond_to?(:unsubscribe) + end end diff --git a/test/models/solid_queue/semaphore_test.rb b/test/models/solid_queue/semaphore_test.rb index 432b97afd..1aa9f39a8 100644 --- a/test/models/solid_queue/semaphore_test.rb +++ b/test/models/solid_queue/semaphore_test.rb @@ -108,6 +108,40 @@ class SolidQueue::SemaphoreTest < ActiveSupport::TestCase assert_equal 0, SolidQueue::BlockedExecution.where(concurrency_key: concurrency_key).count end + test "signal_all does not increment semaphore value beyond limit" do + # Create a job with concurrency limit of 1 + NonOverlappingUpdateResultJob.perform_later(@result) + job = SolidQueue::Job.last + concurrency_key = job.concurrency_key + + # Semaphore should exist with value=0 (one slot taken) + semaphore = SolidQueue::Semaphore.find_by(key: concurrency_key) + assert_equal 0, semaphore.value + + # Manually set semaphore to its limit (simulating the slot was already returned) + semaphore.update!(value: 1) + + # signal_all should NOT increment beyond the limit + SolidQueue::Semaphore.signal_all([ job ]) + + assert_equal 1, semaphore.reload.value, "signal_all should not increment semaphore beyond its limit" + end + + test "signal_all increments semaphore when below limit" do + # Create a job with concurrency limit of 1 + NonOverlappingUpdateResultJob.perform_later(@result) + job = SolidQueue::Job.last + concurrency_key = job.concurrency_key + + # Semaphore should exist with value=0 + assert_equal 0, SolidQueue::Semaphore.find_by(key: concurrency_key).value + + # signal_all should increment from 0 to 1 + SolidQueue::Semaphore.signal_all([ job ]) + + assert_equal 1, SolidQueue::Semaphore.find_by(key: concurrency_key).value + end + private def skip_on_sqlite skip "Row-level locking not supported on SQLite" if SolidQueue::Record.connection.adapter_name.downcase.include?("sqlite") diff --git a/test/solid_queue_test.rb b/test/solid_queue_test.rb index d6d61b576..825d1c1ea 100644 --- a/test/solid_queue_test.rb +++ b/test/solid_queue_test.rb @@ -4,4 +4,8 @@ class SolidQueueTest < ActiveSupport::TestCase test "it has a version number" do assert SolidQueue::VERSION end + + test "time_zone defaults to the application's configured time zone, normalized to its IANA name" do + assert_equal "Etc/UTC", SolidQueue.time_zone + end end diff --git a/test/test_helper.rb b/test/test_helper.rb index db5bd5c34..99d00ba27 100644 --- a/test/test_helper.rb +++ b/test/test_helper.rb @@ -102,7 +102,12 @@ def wait_while_with_timeout!(timeout, &block) # by the cached queries might have been updated, created or deleted in the forked # processes. def skip_active_record_query_cache(&block) - SolidQueue::Record.uncached(&block) + # Forked processes write to both the queue tables and the app's JobResult + # rows, which live in separate databases/connections, so bypass the query + # cache on both — SolidQueue::Record alone leaves JobResult reads stale. + SolidQueue::Record.uncached do + JobResult.uncached(&block) + end end # Silences specified exceptions during the execution of a block diff --git a/test/unit/async_supervisor_test.rb b/test/unit/async_supervisor_test.rb index 6a0f3553b..962c4de7e 100644 --- a/test/unit/async_supervisor_test.rb +++ b/test/unit/async_supervisor_test.rb @@ -52,7 +52,9 @@ class AsyncSupervisorTest < ActiveSupport::TestCase wait_for_registered_processes(2, timeout: 3.seconds) # supervisor + 1 worker assert_registered_processes(kind: "Supervisor(async)") - wait_while_with_timeout(1.second) { SolidQueue::ClaimedExecution.count > 0 } + wait_while_with_timeout(5.seconds) { + SolidQueue::ClaimedExecution.count > 0 || SolidQueue::FailedExecution.count < 3 + } skip_active_record_query_cache do assert_equal 0, SolidQueue::ClaimedExecution.count @@ -74,7 +76,9 @@ class AsyncSupervisorTest < ActiveSupport::TestCase wait_for_registered_processes(2, timeout: 3.seconds) # supervisor + 1 worker assert_registered_processes(kind: "Supervisor(async)") - wait_while_with_timeout(1.second) { SolidQueue::ClaimedExecution.count > 0 } + wait_while_with_timeout(5.seconds) { + SolidQueue::ClaimedExecution.count > 0 || SolidQueue::FailedExecution.count < 3 + } terminate_process(pid) @@ -84,11 +88,42 @@ class AsyncSupervisorTest < ActiveSupport::TestCase end end + test "warns on boot when the thread pool is larger than the database connection pool" do + log = StringIO.new + with_solid_queue_logger(ActiveSupport::Logger.new(log)) do + supervisor = run_supervisor_as_thread(workers: [ { queues: "background", threads: 50, polling_interval: 10 } ], dispatchers: []) + wait_for_registered_processes(2, timeout: 3.seconds) # supervisor + 1 worker + ensure + supervisor.stop + end + + assert_match /Solid Queue is configured to use \d+ threads but the database connection pool is \d+\. Increase it in `config\/database.yml`/, log.string + end + + test "does not warn on boot when the database connection pool is large enough" do + log = StringIO.new + with_solid_queue_logger(ActiveSupport::Logger.new(log)) do + supervisor = run_supervisor_as_thread(workers: [ { queues: "background", threads: 1, polling_interval: 10 } ], dispatchers: []) + wait_for_registered_processes(2, timeout: 3.seconds) # supervisor + 1 worker + ensure + supervisor.stop + end + + assert_no_match /the database connection pool is/, log.string + end + private def run_supervisor_as_thread(**options) SolidQueue::Supervisor.start(mode: :async, standalone: false, **options.with_defaults(skip_recurring: true)) end + def with_solid_queue_logger(logger) + old_logger, SolidQueue.logger = SolidQueue.logger, logger + yield + ensure + SolidQueue.logger = old_logger + end + def simulate_orphaned_executions(count) count.times { |i| StoreResultJob.set(queue: :new_queue).perform_later(i) } process = SolidQueue::Process.register(kind: "Worker", pid: 42, name: "worker-123") diff --git a/test/unit/cli_test.rb b/test/unit/cli_test.rb index 9ee0e233a..f3d5415a3 100644 --- a/test/unit/cli_test.rb +++ b/test/unit/cli_test.rb @@ -36,6 +36,27 @@ class CliTest < ActiveSupport::TestCase end end + test "check exits 0 and prints OK message for a valid configuration" do + out, err = capture_io do + assert_nothing_raised { SolidQueue::Cli.start([ "check", "--skip-recurring" ]) } + end + + assert_match "Solid Queue configuration is valid.", out + assert_empty err + end + + test "check exits 1 and prints invalid recurring task errors" do + out, err, exit_status = capture_check_run( + "--recurring_schedule_file", config_file_path(:recurring_with_invalid).to_s + ) + + assert_equal 1, exit_status + assert_match "Solid Queue configuration is invalid", err + assert_match "periodic_invalid_class", err + assert_match "periodic_incorrect_schedule", err + assert_empty out + end + private def configuration_from_cli(**cli_options) cli = SolidQueue::Cli.new([], cli_options) @@ -43,4 +64,18 @@ def configuration_from_cli(**cli_options) SolidQueue::Configuration.new(**options) end + + # capture_io re-raises SystemExit before returning the captured strings, so we + # swallow it inside the block and return its status alongside the captured IO. + def capture_check_run(*args) + status = nil + out, err = capture_io do + begin + SolidQueue::Cli.start([ "check", *args ]) + rescue SystemExit => e + status = e.status + end + end + [ out, err, status ] + end end diff --git a/test/unit/configuration_test.rb b/test/unit/configuration_test.rb index 34f69658b..7bb7f70a5 100644 --- a/test/unit/configuration_test.rb +++ b/test/unit/configuration_test.rb @@ -27,10 +27,11 @@ class ConfigurationTest < ActiveSupport::TestCase end test "warns if provided configuration file does not exist" do - assert_output "[solid_queue] WARNING: Provided configuration file '/path/to/nowhere.yml' does not exist. Falling back to default configuration.\n" do - configuration = SolidQueue::Configuration.new(config_file: Pathname.new("/path/to/nowhere.yml")) - assert configuration.valid? - end + configuration = SolidQueue::Configuration.new(config_file: Pathname.new("/path/to/nowhere.yml")) + + assert configuration.valid? + assert_includes configuration.warnings.full_messages, + "Warning: provided configuration file '/path/to/nowhere.yml' does not exist. Falling back to default configuration." end test "read configuration from default file" do @@ -145,31 +146,79 @@ class ConfigurationTest < ActiveSupport::TestCase assert error.include?("periodic_invalid_class: Class name doesn't correspond to an existing class") assert error.include?("periodic_incorrect_schedule: Schedule is not a supported recurring schedule") - assert_output(/Provided configuration file '[^']+' does not exist\./) do - assert SolidQueue::Configuration.new(recurring_schedule_file: config_file_path(:empty_recurring)).valid? - end + configuration = SolidQueue::Configuration.new(recurring_schedule_file: config_file_path(:empty_recurring)) + assert configuration.valid? + assert_match(/provided configuration file '[^']+' does not exist\./, configuration.warnings.full_messages.join) + assert SolidQueue::Configuration.new(skip_recurring: true).valid? configuration = SolidQueue::Configuration.new(recurring_schedule_file: config_file_path(:recurring_with_production_only)) assert configuration.valid? assert_processes configuration, :scheduler, 0 - assert_output(/Provided configuration file '[^']+' does not exist\./) do - configuration = SolidQueue::Configuration.new(recurring_schedule_file: config_file_path(:recurring_with_empty)) - assert configuration.valid? - assert_processes configuration, :scheduler, 0 - end + configuration = SolidQueue::Configuration.new(recurring_schedule_file: config_file_path(:recurring_with_empty)) + assert configuration.valid? + assert_processes configuration, :scheduler, 0 + assert_match(/provided configuration file '[^']+' does not exist\./, configuration.warnings.full_messages.join) # No processes configuration = SolidQueue::Configuration.new(skip_recurring: true, dispatchers: [], workers: []) assert_not configuration.valid? assert_equal [ "No processes configured" ], configuration.errors.full_messages - # Not enough DB connections + # Not enough DB connections: still valid so boot is not blocked configuration = SolidQueue::Configuration.new(workers: [ { queues: "background", threads: 50, polling_interval: 10 } ]) - assert_not configuration.valid? - assert_match /Solid Queue is configured to use \d+ threads but the database connection pool is \d+. Increase it in `config\/database.yml`/, - configuration.errors.full_messages.first + assert configuration.valid? + end + + test "reports an undersized thread pool as a warning rather than an error" do + configuration = SolidQueue::Configuration.new(workers: [ { queues: "background", threads: 50, polling_interval: 10 } ], skip_recurring: true) + + assert configuration.valid? + assert_equal 1, configuration.warnings.size + assert_match /Solid Queue is configured to use \d+ threads but the database connection pool is \d+\. Increase it in `config\/database.yml`/, configuration.warnings.full_messages.first + end + + test "has no warnings when the database connection pool is large enough" do + configuration = SolidQueue::Configuration.new(workers: [ { queues: "background", threads: 1, polling_interval: 10 } ], skip_recurring: true) + + assert configuration.valid? + assert_empty configuration.warnings + end + + test "does not duplicate warnings when validated more than once" do + configuration = SolidQueue::Configuration.new(workers: [ { queues: "background", threads: 50, polling_interval: 10 } ], skip_recurring: true) + + 3.times { configuration.valid? } + + assert_equal 1, configuration.warnings.size + end + + test "check prints a success message and returns true for a valid configuration" do + out, err = capture_io do + assert SolidQueue::Configuration.new(skip_recurring: true).check + end + + assert_match "Solid Queue configuration is valid.", out + assert_empty err + end + + test "check prints warnings to stderr on the valid path" do + out, err = capture_io do + assert SolidQueue::Configuration.new(workers: [ { queues: "background", threads: 50, polling_interval: 10 } ], skip_recurring: true).check + end + + assert_match "Solid Queue configuration is valid.", out + assert_match /Solid Queue is configured to use \d+ threads but the database connection pool is \d+/, err + end + + test "check prints errors to stderr and returns false for an invalid configuration" do + out, err = capture_io do + assert_not SolidQueue::Configuration.new(recurring_schedule_file: config_file_path(:recurring_with_invalid)).check + end + + assert_match "Solid Queue configuration is invalid:", err + assert_match "periodic_invalid_class", err end private diff --git a/test/unit/rake_tasks_test.rb b/test/unit/rake_tasks_test.rb new file mode 100644 index 000000000..717bd9807 --- /dev/null +++ b/test/unit/rake_tasks_test.rb @@ -0,0 +1,50 @@ +# frozen_string_literal: true + +require "test_helper" +require "rake" + +class RakeTasksTest < ActiveSupport::TestCase + setup do + @previous_rake_application = Rake.application + @rake = Rake::Application.new + Rake.application = @rake + Rake::Task.define_task(:environment) + load File.expand_path("../../lib/solid_queue/tasks.rb", __dir__) + end + + teardown do + Rake.application = @previous_rake_application + end + + test "solid_queue:check exits 0 and prints OK message for a valid configuration" do + SolidQueue::Configuration.any_instance.stubs(:skip_recurring_tasks?).returns(true) + + out, err = capture_io do + assert_nothing_raised { @rake["solid_queue:check"].invoke } + end + + assert_match "Solid Queue configuration is valid.", out + assert_empty err + end + + test "solid_queue:check exits 1 and prints errors for an invalid configuration" do + SolidQueue::Configuration.any_instance.stubs(:invalid_tasks).returns( + [ stub(key: "broken", errors: stub(full_messages: [ "is invalid" ])) ] + ) + SolidQueue::Configuration.any_instance.stubs(:skip_recurring_tasks?).returns(false) + + status = nil + out, err = capture_io do + begin + @rake["solid_queue:check"].invoke + rescue SystemExit => e + status = e.status + end + end + + assert_equal 1, status + assert_empty out + assert_match "Solid Queue configuration is invalid:", err + assert_match "broken", err + end +end