neighbor

For MariaDB 11.8+, vector columns must use null: false to support vector indexing.

Migration Example:

class CreateItems < ActiveRecord::Migration[8.1]
  def change
    create_table :items do |t|
      t.vector :embedding, limit: 3, null: false
      t.index :embedding, type: :vector
    end
  end
end

Supported Distance Metrics: euclidean, cosine, hamming.

Add neighbor to your application's Gemfile to enable nearest neighbor search for Rails. It supports Postgres (pgvector and cube), MariaDB 11.8+, MySQL 9.7+ (requires HeatWave), and SQLite.

gem "neighbor"

For pgvector, you can add approximate indexes (HNSW or IVFFlat) to speed up queries.

Indexing via migration:

class AddIndexToItemsEmbedding < ActiveRecord::Migration[8.1]
  def change
    add_index :items, :embedding, using: :hnsw, opclass: :vector_l2_ops
    # or
    add_index :items, :embedding, using: :ivfflat, opclass: :vector_l2_ops
  end
end

Distance Metrics:

• Use :vector_l2_ops for Euclidean distance.
• Use :vector_cosine_ops for Cosine distance.
• Use :vector_ip_ops for Inner Product distance.

Supported Distance Values in queries: euclidean, inner_product, cosine, taxicab, hamming, jaccard.

Tuning Search Parameters:

• HNSW: Set the dynamic candidate list size: Item.connection.execute("SET hnsw.ef_search = 100")
• IVFFlat: Set the number of probes: Item.connection.execute("SET ivfflat.probes = 3")

Neighbor supports two Postgres extensions.

For pgvector (supports more dimensions and approximate search):

1. Install the pgvector extension on your system.
2. Run the following commands:

rails generate neighbor:vector
rails db:migrate

For cube (ships with Postgres): Run the following commands:

rails generate neighbor:cube
rails db:migrate

Hybrid search combines keyword search (e.g., Postgres full-text search) and semantic search (vector search).

To combine results, you can use:

1. Reciprocal Rank Fusion (RRF): Use Neighbor::Reranking.rrf(keyword_results, semantic_results) to merge two result sets based on their rank.
2. Reranking Model: Use a dedicated reranking model (via Informers) to score the combined results from both methods.

# Combine using RRF
Neighbor::Reranking.rrf(keyword_results, semantic_results).first(5)

# Combine using a Reranking model
rerank = Informers.pipeline("reranking", "mixedbread-ai/mxbai-rerank-xsmall-v1")
results = (keyword_results + semantic_results).uniq
rerank.(query, results.map(&:content)).first(5).map { |v| results[v[:doc_id]] }

For the cube extension in Postgres:

Distance Metrics: euclidean, cosine, taxicab, chebyshev.

Normalization for Cosine Distance: If using cosine distance with cube, vectors must be normalized before storage. You can automate this in your model:

class Item < ApplicationRecord
  has_neighbors :embedding, normalize: true
end

Specifying Dimensions: It is recommended to specify the number of dimensions to ensure consistency:

class Item < ApplicationRecord
  has_neighbors :embedding, dimensions: 3
end

class Item < ApplicationRecord
  has_neighbors :embedding, normalize: true, dimensions: 3
end

For MySQL 9.7+, vector searching requires HeatWave.

Supported Distance Metrics: euclidean, cosine, hamming.

Binary Vectors: Use the binary type to store binary vectors.

class AddEmbeddingToItems < ActiveRecord::Migration[8.1]
  def change
    add_column :items, :embedding, :binary
  end
end

The neighbor gem provides ActiveRecord integration for several PostgreSQL vector-related types. These types allow for storing and querying high-dimensional embeddings and spatial data directly in your database. The supported types are:

• cube: For spatial data.
• vector: For standard high-dimensional vectors.
• halfvec: For half-precision vectors.
• sparsevec: For sparse vectors.

Neighbor integrates with ActiveRecord by hooking into the :active_record load event. Once loaded, it provides extensions to ActiveRecord models to support vector similarity searches. The gem automatically attempts to initialize adapters for PostgreSQL, MySQL, and SQLite based on the available gems (pg, mysql2, or sqlite3).

# Neighbor is typically used within an ActiveRecord model
class Product < ActiveRecord::Base
  # Neighbor methods will be available once the gem is loaded
end

To use Neighbor with SQLite, you must initialize the adapter and specify which vector extension to use. By default, it uses :sqlite_vec. This step is required to load the necessary Ruby types and prepare the ActiveRecord connection adapter.

Call Neighbor::SQLite.initialize!(extension: :sqlite_vec) to configure the extension. You can also pass a string representing a different extension name.