GoodJob Documentation

To integrate GoodJob into your Rails application, follow these steps:

1. Add the gem to your Gemfile and install:

   bundle add good_job

2. Run the installation generator to create the necessary database migrations:

   bin/rails g good_job:install

   Note: If using Rails multiple databases with migrations_paths, use the --database flag, e.g., bin/rails g good_job:install --database animals.
3. Run the migrations:

   bin/rails db:migrate

4. Configure Active Job to use the GoodJob adapter in config/application.rb or your environment-specific config files:

   config.active_job.queue_adapter = :good_job

To develop the GoodJob gem itself, clone the repository and run the provided setup script to prepare the environment.

# Clone the repository locally
git clone git@github.com:bensheldon/good_job.git

# Set up the gem development environment
bin/setup

In production, GoodJob typically runs in external mode, meaning jobs are executed by a separate process from your web server. Use the GoodJob CLI to start the worker process:

bundle exec good_job start

For example, in a Heroku Procfile:

web: rails server
worker: bundle exec good_job start

Alternatively, you can run jobs within the web server process (useful for low-workloads) by setting the GOOD_JOB_EXECUTION_MODE environment variable to async:

GOOD_JOB_EXECUTION_MODE=async rails server

GoodJob uses an "optimistic retry with an incremental backoff" strategy for perform_limit:

1. Optimistic: Assumes collisions are atypical. For high-collision scenarios, manage concurrency via the number of GoodJob threads/processes (e.g., good_job --queues "serial:1;-serial:5").
2. Retry with Backoff: When perform_limit is exceeded, a GoodJob::ActiveJobExtensions::Concurrency::ConcurrencyExceededError is raised. This is caught by a retry_on handler which re-schedules the job with incremental backoff.
3. Ordering: First-in-first-out (FIFO) order is not preserved when a job is retried via backoff.

To upgrade GoodJob database tables during minor version updates, follow these steps:

1. Generate migration files: bin/rails g good_job:update (If using Rails multiple databases, use --database <name>)
2. Run the migration locally: bin/rails db:migrate
3. Commit the migration files and db/schema.rb.
4. Deploy and run migrations against production, then restart server/worker processes.

bin/rails g good_job:update
# or
bin/rails g good_job:update --database animals

bin/rails db:migrate

In development, GoodJob defaults to async mode, executing jobs in a background thread pool within rails server.

To prevent issues with Rails deferred autoloading (where jobs enqueued via rails console might not run until a web page is loaded), add this to an initializer to force early initialization:

# config/initializers/good_job.rb
Rails.configuration.after_initialize do
  ActiveJob::Base && ActiveRecord::Base
end

If you use PgBouncer in connection mode, GoodJob is compatible. However, if you use PgBouncer in transaction mode, you can work around it by providing a direct (non-proxied) connection to GoodJob using Rails multiple databases support.

1. Define a primary_direct connection in database.yml.
2. Create an abstract ApplicationDirectRecord < ActiveRecord::Base that connects_to database: :primary_direct.
3. Set GoodJob.active_record_parent_class = "ApplicationDirectRecord" in an initializer.

# config/initializers/good_job.rb
GoodJob.active_record_parent_class = "ApplicationDirectRecord"

To migrate without losing jobs:

1. Set ActiveJob::Base.queue_adapter = :good_job (or set it on specific job classes) to enqueue new jobs into GoodJob.
2. Keep running executors for both the old backend and GoodJob simultaneously.
3. Once the old backend's queue is empty, remove the old configuration and executors.

# jobs/specific_job.rb
class SpecificJob < ApplicationJob
  self.queue_adapter = :good_job
  # ...
end

To achieve predictable performance and avoid 'head-of-line blocking' (where a long-running job prevents short jobs from starting), GoodJob recommends organizing jobs into isolated thread pools based on their total latency target (queuing latency + execution latency) rather than functional names (like mailers or sms).

Best Practices

• Avoid functional names: Instead of mailers, use names like latency_30s or latency_5m.
• Group by latency: Categorize jobs into 'Mice' (fast/small) and 'Elephants' (slow/big) to ensure small tasks aren't stuck behind large ones.
• Use isolated thread pools: Create dedicated pools for different latency tiers so that an 'Elephant' in one pool cannot block a 'Mouse' in another.
• Scale based on queue latency: If jobs are missing their latency targets due to queuing delays, increase capacity (processes or threads).

If you use PgBouncer in transaction mode, you must use the :skiplocked or :hybrid lock strategy. :skiplocked uses SELECT FOR UPDATE SKIP LOCKED instead of advisory locks.

When using :skiplocked with PgBouncer transaction mode, you must also disable LISTEN/NOTIFY and the advisory lock heartbeat, and rely on polling instead.

# config/initializers/good_job.rb
GoodJob.configure do |config|
  config.lock_strategy = :skiplocked
  config.enable_listen_notify = false
  config.advisory_lock_heartbeat = false
  config.poll_interval = 5 # seconds
end