DOCSEARCHbyAlgolia
Submit Docs

parallel Ruby Library

repository·master·Indexed 26 days ago

https://github.com/grosser/parallel

A Ruby library for running code concurrently using Processes, Threads, or Ractors. It provides parallelized versions of collection methods such as map, each, any?, and all?, and is optimized for map-reduce patterns, parallel downloads, and uploads. It includes features for ActiveRecord connection management, progress monitoring via ruby-progressbar, worker isolation, and secure inter-process communication using an HMAC serializer.

Tokens
2.2K
Snippets
6
Records
22
Agent score
88%

What's inside parallel

  2. Handle ActiveRecord connections in parallel

    master

    When using Parallel with ActiveRecord, you must manage connections to avoid errors.

    • Multithreading: Requires connection pooling. Adjust pool size in config/database.yml.
    • Forks (Processes): Requires reconnecting in the child process.

    Common patterns:

    # Reconnect in the main process after the block
Parallel.each(User.all, in_processes: 8) do |user|
  user.update_attribute(:some_attribute, some_value)
end
User.connection.reconnect!

# Use the connection pool explicitly (Threads)
Parallel.each(User.all, in_threads: 8) do |user|
  ActiveRecord::Base.connection_pool.with_connection do
    user.update_attribute(:some_attribute, some_value)
  end
end

# Reconnect once inside every fork
Parallel.each(User.all, in_processes: 8) do |user|
  @reconnected ||= User.connection.reconnect! || true
  user.update_attribute(:some_attribute, some_value)
end

    Note on NameError: To avoid race conditions during autoloading in development/test environments, manually load your models before the parallel block using require '<modelname>' or ModelName.class.

  3. Run code in parallel using Processes, Threads, or Ractors

    master

    The Parallel module provides several methods to execute code concurrently. You can specify the concurrency model using in_processes, in_threads, or in_ractors options.

    • Processes: Best for CPU-bound tasks. Variables are protected from change, but extra memory is used.
    • Threads: Best for blocking I/O operations. Variables can be shared/modified, and no extra memory is used.
    • Ractors: Available in Ruby 3.0+. Very fast to spawn, but experimental and unstable. Variables must be passed explicitly or made shareable via Ractor.make_shareable.

  4. Produce items dynamically using a lambda or Queue

    master

    Instead of a static array, you can pass a lambda (anything responding to .call) or a Queue to Parallel to produce items one at a time.

    items = [1,2,3]
Parallel.each( -> { items.pop || Parallel::Stop }) { |number| ... }

  5. Use Parallel with different collection methods

    master

    In addition to map, you can use each, each_with_index, map_with_index, and flat_map to perform parallel operations on collections.

    Parallel.each(['a','b','c']) { |one_letter| ... }

  6. Secure worker communication with HMAC serializer

    master

    By default, worker processes communicate with the parent via Marshal over an anonymous pipe. In hardened environments where you want to prevent Marshal payload injection, use the Parallel::Serializer::Hmac serializer. It signs messages with an HMAC-SHA256 per-worker secret.

    Parallel.map(items, in_processes: 2, serializer: Parallel::Serializer::Hmac.new) { ... }

  7. Perform parallel predicate checks with any? and all?

    master

    You can use Parallel.any? and Parallel.all? to perform parallelized boolean checks on a collection.

    Parallel.any?([1,2,3,4,5,6,7]) { |number| number == 4 }
# => true

Parallel.all?([1,2,nil,4,5]) { |number| number != nil }
# => false

  8. Monitor progress and use hooks

    master

    You can display a progress bar using the ruby-progressbar gem or use :start and :finish hooks to execute code on the main thread.

    • :start hook provides (item, index).
    • :finish hook provides (item, index, result).
    • Use finish_in_order: true to ensure hooks are called in the order of the input array.

  9. Identify the current worker number

    master

    Use Parallel.worker_number to determine the specific worker slot (index) in which a task is currently running.

    Parallel.each(1..5, in_processes: 2) { |i| puts "Item: #{i}, Worker: #{Parallel.worker_number}" }

  10. Stop parallel execution with Break or Kill

    master

    You can control the flow of parallel execution by raising specific exceptions:

    • Parallel::Break: Stops after all currently running items are finished.
    • Parallel::Kill: Stops all sub-processes immediately (use with caution).

  11. Control worker isolation

    master
    When using in_processes, you can enable isolation: true. This causes the library to replace a worker process with a new one after it completes a task, which can help prevent memory leaks or resource exhaustion in long-running jobs.

  12. Use instrumentation hooks (`start` and `finish`)

    master

    Provide callbacks to execute at the start and end of each parallel task using the start and finish options.

    • start: A proc called with (item, index).
    • finish: A proc called with (item, index, result).

    Note: If you use finish_in_order: true, the finish callback will be triggered in the same order as the input collection, even if tasks complete out of order.