DOCSEARCHbyAlgolia
Submit Docs

sidekiq-unique-jobs

repository·main·Indexed 23 days ago

https://github.com/mhenrixon/sidekiq-unique-jobs

A Ruby gem that prevents duplicate Sidekiq jobs from running using Redis-based locks. It supports various lock types (e.g., :until_executing, :until_expired, :while_executing) and conflict strategies to manage uniqueness constraints based on job arguments and lifecycle stages. Requires Ruby >= 3.2, Sidekiq >= 8.0, and Redis >= 6.2.

Tokens
10.1K
Snippets
26
Records
67
Agent score
81%

What's inside sidekiq-unique-jobs

  1. Understand the Redis command lifecycle for unique jobs

    main

    The current optimized implementation of sidekiq-unique-jobs achieves a minimum of 8 Redis commands per lock cycle. This breakdown represents the structural minimum required to maintain compatibility with the middleware ecosystem and the reaper/UI components.

    Lock Phase (queue_and_lock.lua - 5 commands)

    1. evalsha: Executes the atomic Lua script.
    2. HLEN locked: Checks if the lock limit has been reached.
    3. SET digest job_id: Creates the digest key (required by middleware).
    4. ZADD digests score digest: Registers the digest in the sorted set (required by the reaper/web UI).
    5. HSET locked job_id time: Acquires the actual lock.

    Unlock Phase (unlock.lua - 3 commands)

    1. evalsha: Executes the atomic Lua script.
    2. UNLINK digest info locked primed: Deletes all associated lock keys in a single batch command.
    3. ZREM digests digest: Removes the digest from the sorted set.

  2. Understand benchmark output metrics (IPS and Memory)

    main

    Benchmarks provide two primary metrics:

    1. Iterations Per Second (IPS): Measures how many times an operation can be performed per second. Higher is better.

      • Example: lock and unlock - normal flow 5.678k i/s

    2. Memory Usage: Shows memory allocated during operations. Lower is better.

      • Example: allocated: 45.2 MB

  3. How lock types work

    main

    Choose a lock type in your job options to define when a lock is acquired and released. This determines the lifecycle of the uniqueness constraint.

    class MyJob
  include Sidekiq::Job

  # Example: Prevent duplicates until the job completes
  sidekiq_options lock: :until_executed

  def perform(user_id)
    # ...
  end
end

  5. Manually compare benchmark results

    main

    If you prefer to run benchmarks manually without the automated comparison tool, follow these steps to compare your current branch against main using diff:

    # Save current branch name
CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)

# Benchmark main branch
git checkout main
bin/benchmark_improvements > results_main.txt

# Benchmark your branch
git checkout $CURRENT_BRANCH
bin/benchmark_improvements > results_current.txt

# Compare results
diff -y results_main.txt results_current.txt | less

  7. Configure sidekiq-unique-jobs middleware

    main

    To enable uniqueness, you must add the client and server middleware to your Sidekiq configuration in an initializer (e.g., config/initializers/sidekiq.rb).

    # config/initializers/sidekiq.rb
Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end
end

Sidekiq.configure_server do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end

  config.server_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Server
  end

  SidekiqUniqueJobs::Server.configure(config)
end

  8. Compare performance between git branches

    main

    The bin/compare_performance tool automates the process of comparing performance between two branches. It checks out the base branch, runs benchmarks, checks out the comparison branch, runs benchmarks again, and returns to your original branch while displaying a summary comparison.

    Usage Patterns:

    • Compare current branch with main:
    bin/compare_performance
    • Compare specific branches:
    bin/compare_performance main improved-exception-handling
    • Compare with custom base and comparison branches:
    bin/compare_performance v8.0.11 feature-branch
    # Compare current branch with main
bin/compare_performance

# Compare specific branches
bin/compare_performance main improved-exception-handling

# Compare with custom base and comparison branches
bin/compare_performance v8.0.11 feature-branch

  12. Benchmark the Redis command efficiency of sidekiq-unique-jobs

    main

    To evaluate the performance and Redis command overhead of the current implementation, you can run the autoresearch benchmark script. This script measures the redis_commands_per_cycle metric, which aims to minimize the number of Redis commands executed during a single lock lifecycle.

    Metrics tracked:

    • redis_commands_per_cycle: The primary metric (lower is better).
    • keys_while_locked: Number of keys active during the lock period.
    • ops_per_sec: Throughput for different job lifecycles (until_executed, until_expired, while_executing).
    bash autoresearch.sh