Solid Queue Documentation

The Supervisor Mode determines how the supervised actors (workers, dispatchers, etc.) are run relative to the supervisor process.

Fork Mode (Default & Recommended)

The supervisor forks a separate process for each supervised actor. This provides the best isolation and performance but uses more memory.

Async Mode

All actors run in the same process as the supervisor using different threads. This is useful if forking is not supported or memory is extremely constrained.

How to use Async mode:

• Run via CLI: bin/jobs --mode async
• Or set the environment variable: SOLID_QUEUE_SUPERVISOR_MODE=async

Note: In async mode, the processes configuration option is ignored. This mode is separate from the worker's concurrency model (which uses threads or fibers).

For jobs scheduled in the future (using Active Job's wait or wait_until), concurrency limits are enforced when the jobs are due, not when they are initially scheduled.

Example flow:

1. Multiple jobs are scheduled with wait: 10.minutes.
2. At the 10-minute mark, they are enqueued.
3. If the concurrency limit is 1, the first job runs and the second is blocked.
4. Once the first job finishes, the second is unblocked and becomes ready.

Warning: Mixing concurrency controls with future scheduling can impact performance because jobs must be enqueued one-by-one rather than in batches to ensure limits are respected.

Solid Queue uses several specialized actors to manage job lifecycles:

• Workers: Pick up ready jobs from queues and process them (using the solid_queue_ready_executions table).
• Dispatchers: Select scheduled jobs that are due and move them from solid_queue_scheduled_executions to solid_queue_ready_executions. They also handle concurrency control maintenance.
• Scheduler: Manages recurring tasks by enqueuing them when they are due.
• Supervisor: The orchestrator that runs workers, dispatchers, and schedulers according to your configuration, manages their heartbeats, and handles starting/stopping them.

Solid Queue uses two mechanisms for ordering jobs:

1. Queue Order: If a worker is configured with a list of queues (e.g., [real_time, background]), it polls them in that specific order. No jobs will be taken from background unless real_time is empty.
2. Job Priority: Active Job supports integer priorities. In Solid Queue, smaller values have higher priority. Within a single queue, jobs are picked by priority.

Warning: Queue order takes precedence over job priority. If real_time has low-priority jobs and background has high-priority jobs, the worker will still pick the real_time jobs first. It is recommended to use either queue order or priorities, but not both mixed together.

Fiber workers execute jobs as fibers on a single reactor thread. This is best for cooperative, I/O-bound jobs.

Requirements & Best Practices:

• Isolation: You MUST set config.active_support.isolation_level = :fiber in your Rails application. Solid Queue will refuse to boot fiber workers if isolation remains thread-scoped.
• Database Connections: On Rails 7.2+, you can often use a smaller pool (e.g., 3-5 connections per worker process) because Active Record can release connections during non-blocking waits. On Rails 7.1, size the pool more conservatively.
• Avoid Blocking: CPU-heavy or blocking code will block the entire fiber reactor thread.

Solid Queue does not implement an automatic retry mechanism; it relies on Active Job for retries. When a job fails, a record is created in the solid_queue_failed_executions table. These failed executions persist until they are manually discarded or re-enqueued via the Rails console.

To manage failed jobs:

1. Find the execution using SolidQueue::FailedExecution.find(...).
2. Inspect the error using .error.
3. Use .retry to re-enqueue the job as if it were new.
4. Use .discard to delete the job from the system.

failed_execution = SolidQueue::FailedExecution.find(...) # Find the failed execution related to your job
failed_execution.error # inspect the error

failed_execution.retry # This will re-enqueue the job as if it was enqueued for the first time
failed_execution.discard # This will delete the job from the system

To ensure optimal performance and use covering indexes, follow these best practices:

1. Use exact queue names instead of wildcards (e.g., queues: [background, backend] instead of queues: back*).
2. Avoid paused queues. Pausing queues requires a DISTINCT query that can be slower on some databases.

Wildcard prefixes (like beta*) require an extra step to fetch the list of matching queues before polling, which can impact performance depending on the database engine.

If you prefer not to use a separate database for the queue, you can use a single database for both the app and the queue:

1. Copy the contents of db/queue_schema.rb into a standard Rails migration and delete db/queue_schema.rb.
2. Remove config.solid_queue.connects_to from your production configuration.
3. Run your migrations.

In this mode, database.yml only needs a primary entry.