Rihanna Documentation

repository·master·Indexed 19 days ago

https://github.com/samsondav/rihanna

A fast, Postgres-backed distributed job queue for Elixir that uses advisory locks for high performance. It provides ACID guarantees and supports passing arbitrary Elixir/Erlang terms as job arguments. Rihanna supports both module-function-arguments (MFA) tuples and the Rihanna.Job behaviour for job execution, along with features for scheduling deferred jobs, custom retry strategies, and integration with Ecto.

Tokens
6.2K
Snippets
32
Records
38
Agent score
63%

What's inside Rihanna

  1. Install Rihanna without Ecto

    master

    To use Rihanna without Ecto:

    1. Add dependency: Add {:rihanna, ">= 0.0.0"} to your mix.exs.
    2. Migrate database: Manually create the jobs table (refer to Rihanna.Migration documentation).
    3. Boot the Supervisor: Add Rihanna.Supervisor to your supervision tree with a postgrex configuration block.
    # In your application.ex (Elixir 1.6+)
    children = [
      {Rihanna.Supervisor, [name: Rihanna.Supervisor, postgrex: My.Repo.config()]}
    ]
  2. Upgrade to Rihanna v2 using Direct SQL (Zero Downtime)

    master

    If you need to avoid downtime, you can perform a manual SQL upgrade. Warning: This requires high SQL proficiency. This method assumes your table is named rihanna_jobs (replace if different).

    1. Manual Schema Update

    While the existing code is still running, add the required column and the new concurrent index:

    ALTER TABLE rihanna_jobs ADD COLUMN priority integer;
    CREATE INDEX CONCURRENTLY rihanna_jobs_locking_index ON rihanna_jobs (priority ASC, due_at ASC NULLS FIRST, enqueued_at ASC, id ASC);

    2. Code and Migration Setup

    1. Upgrade your application code to use Rihanna v2.
    2. Create a migration using mix ecto.gen.migration and include use Rihanna.Migration.Upgrade as shown in the downtime guide.
    3. Manually mark the migration as complete by inserting its version into the schema_migrations table so it does not attempt to run on deployment:
      INSERT INTO schema_migrations VALUES (YOUR_MIGRATION_VERSION_HERE, now());
    4. Deploy/release your code.

    3. Finalize Column Constraints

    Once the new code is running, perform the following SQL commands to set defaults and constraints:

    1. Set default value:
      ALTER TABLE rihanna_jobs ALTER COLUMN priority SET DEFAULT 50;
    2. Backfill existing NULL values:
      UPDATE rihanna_jobs SET priority = 50 WHERE priority IS NULL;
    3. Apply NOT NULL constraint:
      ALTER TABLE rihanna_jobs ALTER COLUMN priority SET NOT NULL;
  3. Upgrade to Rihanna v2 using Stop-the-World Downtime

    master

    If you can afford downtime, follow these steps to upgrade to Rihanna v2.0.0 or higher. This method ensures the required priority column and the new locking index are applied correctly via Ecto migrations.

    1. Prepare the Migration

    First, upgrade your dependencies to Rihanna v2 using mix. Then, generate a new migration file:

    mix ecto.gen.migration

    In the generated migration file, use the Rihanna.Migration.Upgrade module to handle the schema changes:

    defmodule MyApp.UpgradeRihannaJobs do
      use Rihanna.Migration.Upgrade
    end

    2. Deploy the Changes

    To prevent errors during the migration, follow this deployment sequence:

    1. Stop your application: Jobs cannot be enqueued or processed during the migration.
    2. Deploy the new code: Deploy the code containing the v2 upgrade, but do not start the application processes.
    3. Run migrations: Execute mix ecto.migrate to apply the schema changes.
    4. Restart the application: Start the application now running Rihanna v2.
  4. Install Rihanna with Ecto

    master

    To use Rihanna with Ecto, follow these steps:

    1. Add dependency: Add {:rihanna, "~> 2.3"} to your mix.exs.
    2. Migrate database: Create a migration using Rihanna.Migration to create the rihanna_jobs table.
      defmodule MyApp.CreateRihannaJobs do
        use Rihanna.Migration
      end
    3. Configure Ecto Repo: Set producer_postgres_connection in your config to reuse your Ecto Repo connection.
      config :rihanna,
        producer_postgres_connection: {Ecto, MyApp.Repo}
    4. Boot the Supervisor: Add Rihanna.Supervisor to your supervision tree, passing your Repo config under the postgrex key.
    # In your application.ex (Elixir 1.6+)
    children = [
      {Rihanna.Supervisor, [postgrex: My.Repo.config()]}
    ]
  5. How job ordering works in Rihanna

    master

    Rihanna uses a FIFO (First-In-First-Out) job queue, meaning jobs are processed roughly in the order they are enqueued.

    However, because Rihanna is a concurrent job queue with multiple workers processing jobs simultaneously, there is no guarantee of strict ordering in practice.

  6. Database connection requirements for Rihanna

    master

    Rihanna requires a specific number of database connections per node:

    • 1 connection for the external API (enqueuing and retrying jobs).
    • N connections for dispatchers (where N is the number of dispatchers).

    By default, Rihanna runs one dispatcher per node, meaning a total of 2 database connections are required per node.

  7. Set up the Rihanna jobs table without Ecto

    master

    If you are not using Ecto, you can create the jobs table manually by executing the SQL statements provided by Rihanna.

    Use Rihanna.Migration.sql/1 to get a single string containing all necessary semi-colon-terminated SQL statements. You can optionally pass a custom table name.

    # Returns the SQL string for the default table name
    sql_statements = Rihanna.Migration.sql()
    
    # Returns the SQL string for a custom table name
    sql_statements = Rihanna.Migration.sql("my_custom_jobs_table")
    Rihanna.Migration.sql("my_custom_jobs_table")
  8. Upgrade Rihanna jobs table without Ecto

    master

    If you are not using Ecto, you can manually upgrade your jobs table by executing the SQL statements provided by Rihanna.Migration.Upgrade.

    • Use Rihanna.Migration.Upgrade.sql/1 to get a single string of semi-colon-terminated SQL statements.
    • Use Rihanna.Migration.Upgrade.statements/1 to get a list of individual SQL strings.

    Both functions accept an optional table_name (string or atom). If no table name is provided, it defaults to the value of :jobs_table_name in your Rihanna configuration (which defaults to "rihanna_jobs").

    # To get the full SQL string for execution:
    sql_script = Rihanna.Migration.Upgrade.sql("my_custom_table")
  9. Setup Rihanna in your supervision tree

    master

    To use Rihanna, add Rihanna.Supervisor to your application's supervision tree. This will automatically start the Postgrex process required for enqueueing jobs and start the job dispatcher for processing them when your application boots.

    Database Configuration

    Rihanna requires a database configuration passed via the postgrex key. This configuration is passed directly to Postgrex.

    If you are using Ecto, you can avoid duplicating your configuration by using My.Repo.config() to extract your existing database settings.

    # In your application.ex
    children = [
      {Rihanna.Supervisor, [name: Rihanna.Supervisor, postgrex: My.Repo.config()]}
    ]
  10. Set up the Rihanna jobs table using Ecto

    master

    The easiest way to create the required database table for Rihanna is using Ecto migrations.

    1. Generate a new migration file:
      mix ecto.gen.migration create_rihanna_jobs
    2. Update the generated migration file to use Rihanna.Migration:
      defmodule MyApp.CreateRihannaJobs do
        use Rihanna.Migration
      end
    3. Run the migration:
      mix ecto.migrate

    By default, the table is named "rihanna_jobs". You can change this by setting :jobs_table_name in your application configuration or by passing a :table_name option to the use Rihanna.Migration macro.

    defmodule MyApp.CreateRihannaJobs do
      use Rihanna.Migration
    end
  11. Upgrade Rihanna jobs table using Ecto

    master

    If your project uses Ecto, the simplest way to upgrade your existing Rihanna jobs table is to use the Rihanna.Migration.Upgrade module within a standard Ecto migration. This handles the addition of new columns (due_at, rihanna_internal_meta, priority) and updates the necessary indexes.

    1. Generate a new migration: mix ecto.gen.migration upgrade_rihanna_jobs.
    2. Use Rihanna.Migration.Upgrade in the migration module.
    3. Run mix ecto.migrate to apply the changes.
    defmodule MyApp.UpgradeRihannaJobs do
      use Rihanna.Migration.Upgrade
    end
  12. Upgrade from Rihanna v2

    master

    If upgrading from v2, you may need to recreate the locking index to ensure due_at uses NULLS FIRST. Run the following SQL:

    CREATE INDEX CONCURRENTLY rihanna_jobs_locking_index_fixed ON rihanna_jobs (priority ASC, due_at ASC NULLS FIRST, enqueued_at ASC, id ASC);
    DROP INDEX rihanna_jobs_locking_index;
    ALTER INDEX rihanna_jobs_locking_index_fixed RENAME TO rihanna_jobs_locking_index;