Chewy Documentation

repository·master·Indexed 23 days ago

https://github.com/toptal/chewy

Chewy is an Object Document Mapper (ODM) for the official Elasticsearch Ruby client. It provides an ActiveRecord-style DSL for querying, automatic index updates for ActiveRecord models, and efficient bulk import strategies. The library supports various update strategies (such as :atomic, :sidekiq, and :delayed_sidekiq), indexing configuration, and integration with non-Rails Ruby applications.

Tokens
21.9K
Snippets
62
Records
118
Agent score
80%

What's inside Chewy

  1. Import and Query data with Chewy

    master

    Chewy provides specialized tools for data ingestion and retrieval:

    • Import: Use various import options, perform raw imports, and manage journaling.
    • Querying: Execute search requests, handle pagination, use scopes, utilize scroll, and manage loading.
  2. Configure Chewy client and indexing

    master

    Chewy configuration and indexing capabilities are covered in the following documentation areas:

    • Configuration: Manage client settings, update strategies, notifications, and integrations.
    • Indexing: Define indexes, specify field types, use crutches, manage the compiled compose path, and perform index manipulation.
  3. Manage Chewy operations and testing

    master

    Operational tasks and testing workflows are documented as follows:

    • Rake Tasks: Manage reindexing, syncing, journal management, and parallelization.
    • Testing: Integration with RSpec, Minitest, and DatabaseCleaner.
    • Troubleshooting: Resolving common errors, debugging imports, and addressing Elasticsearch 8 specific issues.
  4. Choose an Index Update Strategy

    master

    When an object is saved (e.g., City.first.save!), Chewy requires an explicit update strategy to prevent UndefinedUpdateStrategy exceptions. You can wrap code blocks in Chewy.strategy(:name) { ... } to define how index updates are handled.

    Common strategies include:

    • :atomic: Delays updates until the end of the block and uses the Bulk API (highly optimized).
    • :sidekiq: Performs :atomic updates asynchronously via Sidekiq.
    • :lazy_sidekiq: Asynchronous updates that also defer the evaluation of update_index callbacks to improve response time. Note: records are re-fetched from the DB, so non-persistent state cannot be used.
    • :delayed_sidekiq: Accumulates record IDs in Redis and reindexes them in batches after a latency window. Best for frequently mutated records.
    • :active_job: Performs :atomic updates asynchronously using ActiveJob.
    • :urgent: Performs individual update requests for every object (useful for manual debugging in console).
    • :bypass: Disables automatic index updates on object save.
    • :default: To return to pre-0.7.0 behavior (automatic updates), set Chewy.root_strategy = :bypass.
    # Example using :atomic strategy
    Chewy.strategy(:atomic) do
      City.popular.map(&:do_some_update_action!)
    end
  5. Optimize indexing with Chewy Crutches

    master

    When dealing with complex associations that are slow to index via standard ActiveRecord includes, use Chewy Crutches. Crutches allow you to fetch data for an entire batch of objects using a single, lightweight query (e.g., using .pluck) and then map that data to the objects during the indexing process.

    This can increase indexing performance significantly by avoiding expensive object initialization for associated records.

    class ProductsIndex < Chewy::Index
      index_scope Product
    
      crutch :categories do |collection|
        # Fetch data for the whole batch efficiently
        data = ProductCategory.joins(:category)
          .where(product_id: collection.map(&:id))
          .pluck(:product_id, 'categories.name')
    
        # Format as a lookup hash: { product_id => [category_names] }
        data.each.with_object({}) { |(id, name), result| (result[id] ||= []).push(name) }
      end
    
      field :name
      # Access crutch data in the value proc
      field :category_names, value: ->(product, crutches) { crutches[:categories][product.id] }
    end
  6. Manage update strategies without Rails

    master

    In Rails, Chewy automatically manages update strategies. Without Rails, you must manually manage them to avoid UndefinedUpdateStrategy errors. You have two options:

    1. Wrap code in a strategy block: Use Chewy.strategy(:strategy_name) around your import or update logic.
    2. Set a root strategy: Set Chewy.root_strategy = :strategy_name globally.

    Common strategies include :atomic and :bypass.

    # Option 1: Strategy block
    Chewy.strategy(:atomic) do
      # your import / update code
    end
    
    # Option 2: Root strategy
    Chewy.root_strategy = :bypass
  7. Understand the Compiled Compose Path

    master

    Chewy uses a compiled compose path by default to speed up document creation. On the first import, Chewy generates a specialized __chewy_compose__ method for each index that inlines the field tree and accessors. This avoids iterating through the field list at runtime and typically results in 3-4× faster document composition.

    Key features:

    • Automatically handles (object), (object, crutches), or (object, crutches, context) arguments.
    • Supports Symbol#to_proc shorthand.
    • Transparently falls back to the legacy iterative path for edge cases like geo_point fields or ignore_blank fields.
  8. Choose the correct Chewy update strategy

    master

    Chewy requires an explicit strategy to prevent accidental, expensive reindexing. If update_index is called without a strategy block, it raises Chewy::UndefinedUpdateStrategy.

    StrategyUse Case
    :atomicDefault for web requests. Batches updates into a single bulk call at the end of the block.
    :urgentRails console / scripts. Updates Elasticsearch immediately after every save.
    :sidekiqBackground reindexing via Sidekiq.
    :active_jobBackground reindexing via ActiveJob.
    :bypassTests or migrations where you want to disable automatic indexing.

    Example usage:

    Chewy.strategy(:atomic) do
      Book.find_each { |b| b.update!(title: b.title.titleize) }
    end
  9. Nest Chewy strategies

    master

    Strategies can be nested to allow different update behaviors for different parts of a single operation. You can use block notation or non-block notation.

    Block notation:

    Chewy.strategy(:atomic) do
      city1.do_update! # Grouped
      Chewy.strategy(:urgent) do
        city2.do_update! # Individual
        city3.do_update! # Individual
      end
      city4.do_update! # Grouped with city1
    end

    Non-block notation:

    Chewy.strategy(:urgent)
    city1.do_update! # index updated immediately
    Chewy.strategy(:bypass)
    city2.do_update! # update bypassed
    Chewy.strategy.pop
    city3.do_update! # index updated again
    # Non-block notation example
    Chewy.strategy(:urgent)
    city1.do_update! # index updated
    Chewy.strategy(:bypass)
    city2.do_update! # update bypassed
    Chewy.strategy.pop
    city3.do_update! # index updated again
  10. Define multi-field and object field types

    master

    Object Fields

    To define an object type, nest fields within a field block. This automatically sets the type to object.

    field :projects do
      field :title
      field :description
    end

    Multi-fields

    To define a multi-field (e.g., a text field with a keyword sub-field for sorting), specify a type other than object or nested in the root field and provide a block containing the sub-fields.

    field :title, type: 'text' do
      field :sorted, type: 'keyword'
    end
    # Usage: BooksIndex.order('title.sorted': :asc)
    field :title, type: 'text' do
      field :sorted, type: 'keyword'
    end
  11. Enable Journaling to prevent data loss

    master

    Journaling records all create, update, and destroy actions into a separate Elasticsearch index. This is useful during zero-downtime index resets: while you are rebuilding a new index, any updates happening to the old data are captured in the journal, allowing you to replay them later using the Chewy::Journal interface.

    Journaling is disabled by default. You can enable it in three ways:

    1. Globally in config/chewy.yml by setting journal: true.
    2. For a specific import call: CityIndex.import journal: true.
    3. As a default for an index: default_import_options journal: true.

    Warning: Journaling can grow very large. It is recommended to periodically clean it using the chewy:journal:clean rake task.

    # As a default import option
    class CityIndex
      index_scope City
      default_import_options journal: true
    end
    
    # Or during a specific import
    CityIndex.import journal: true
  12. Use Raw Imports to speed up import time

    master

    When using the ActiveRecord adapter, you can use raw_import to bypass the overhead of full ActiveRecord model instantiation. Instead of loading full objects, Chewy can operate on raw hashes obtained directly from the database.

    To use this, provide a proc to the raw_import option that converts the database hash into a lightweight object that mimics the required behavior of your index fields.

    You can set this as a default in the index class or pass it explicitly to the import method.

    class LightweightProduct
      def initialize(attributes)
        @attributes = attributes
      end
    
      def created_at
        @attributes['created_at'].tr(' ', 'T') << 'Z'
      end
    end
    
    class ProductIndex < Chewy::Index
      index_scope Product
      default_import_options raw_import: ->(hash) {
        LightweightProduct.new(hash)
      }
    
      field :created_at, 'datetime'
    end