pgvector-elixir

Add pgvector to your application's mix.exs dependencies to enable support for the pgvector PostgreSQL extension in Elixir.

{:pgvector, "~> 0.4.0"}

To use pgvector directly with Postgrex (without Ecto):

1. Register the extension types using Postgrex.Types.define/3.
2. Pass the custom types to Postgrex.start_link/1.
3. Enable the extension in the database using CREATE EXTENSION IF NOT EXISTS vector.
4. Use standard SQL for table creation and queries.

# 1. Register types
Postgrex.Types.define(MyApp.PostgrexTypes, Pgvector.extensions(), [])

# 2. Start connection
{:ok, pid} = Postgrex.start_link(types: MyApp.PostgrexTypes)

# 3. Enable extension
Postgrex.query!(pid, "CREATE EXTENSION IF NOT EXISTS vector", [])

# 4. Create table and insert
Postgrex.query!(pid, "CREATE TABLE items (embedding vector(3))", [])
Postgrex.query!(pid, "INSERT INTO items (embedding) VALUES ($1)", [[1, 2, 3]])

# 5. Query nearest neighbors
Postgrex.query!(pid, "SELECT * FROM items ORDER BY embedding <-> $1 LIMIT 5", [[1, 2, 3]])

To use pgvector with Ecto, you must define custom Postgrex types that include the pgvector extensions and then configure your Repo to use those types.

1. Create lib/postgrex_types.ex to define your custom types.
2. Update config/config.exs to point your Repo to these types.
3. Create and run a migration to enable the vector extension in PostgreSQL.

# lib/postgrex_types.ex
Postgrex.Types.define(MyApp.PostgrexTypes, Pgvector.extensions() ++ Ecto.Adapters.Postgres.extensions(), [])

# config/config.exs
config :my_app, MyApp.Repo, types: MyApp.PostgrexTypes

# Migration to enable extension
defmodule MyApp.Repo.Migrations.CreateVectorExtension do
  use Ecto.Migration

  def up do
    execute "CREATE EXTENSION IF NOT EXISTS vector"
  end

  def down do
    execute "DROP EXTENSION vector"
  end
end

Once configured, you can use vector types in Ecto migrations and schemas.

Supported types:

• :vector (use Pgvector.Ecto.Vector in schema)
• :halfvec (use Pgvector.Ecto.HalfVector in schema)
• :bit (use Pgvector.Ecto.Bit in schema)
• :sparsevec (use Pgvector.Ecto.SparseVector in schema)

Example migration and schema usage:

# Migration
create table(:items) do
  add :embedding, :vector, size: 3
end

# Schema
schema "items" do
  field :embedding, Pgvector.Ecto.Vector
end

You can add HNSW or IVFFlat indexes to your vector columns within Ecto migrations. Use vector_l2_ops for L2 distance, vector_ip_ops for inner product, or vector_cosine_ops for cosine distance.

# HNSW index
create index("items", ["embedding vector_l2_ops"], using: :hnsw)

# IVFFlat index
create index("items", ["embedding vector_l2_ops"], using: :ivfflat, options: "lists = 100")

Sparse vectors represent vectors where most elements are zero, using Pgvector.SparseVector.

# Create from list
vec = Pgvector.SparseVector.new([1, 2, 3])

# Create from Nx tensor
vec = Nx.tensor([1.0, 2.0, 3.0]) |> Pgvector.SparseVector.new()

# Create from a map of non-zero elements (indices start at 0)
elements = %{0 => 1.0, 2 => 3.0}
vec = Pgvector.SparseVector.new(elements, 6)

# Metadata and extraction
dim = vec |> Pgvector.SparseVector.dimensions()
indices = vec |> Pgvector.SparseVector.indices()
values = vec |> Pgvector.SparseVector.values()

# Conversion
list = vec |> Pgvector.to_list()
tensor = vec |> Pgvector.to_tensor()

The Pgvector module provides tools to create vector objects from lists or Nx tensors and convert them back.

# Create from list
vec = Pgvector.new([1, 2, 3])

# Create from Nx tensor
vec = Nx.tensor([1.0, 2.0, 3.0]) |> Pgvector.new()

# Convert to list
list = vec |> Pgvector.to_list()

# Convert to Nx tensor
tensor = vec |> Pgvector.to_tensor()

Half vectors use Pgvector.HalfVector for reduced precision storage.

# Create from list
vec = Pgvector.HalfVector.new([1, 2, 3])

# Create from Nx tensor (f16)
vec = Nx.tensor([1.0, 2.0, 3.0], type: :f16) |> Pgvector.HalfVector.new()

# Conversion
list = vec |> Pgvector.to_list()
tensor = vec |> Pgvector.to_tensor()

Use Pgvector.Ecto.Query to perform nearest neighbor searches using various distance metrics.

Supported distance functions:

• l2_distance/2
• max_inner_product/2
• cosine_distance/2
• l1_distance/2
• hamming_distance/2
• jaccard_distance/2

Note: For cosine distance or inner product, use vector_cosine_ops or vector_ip_ops when creating indexes.

import Ecto.Query
import Pgvector.Ecto.Query

# Find 5 nearest neighbors using L2 distance
Repo.all(from i in Item, order_by: l2_distance(i.embedding, ^Pgvector.new([1, 2, 3])), limit: 5)

The Pgvector.Extensions.Vector module provides the necessary interface for the Postgrex driver to handle PostgreSQL vector types. It manages the encoding of Elixir vector structures into binary format for database storage and the decoding of binary data from PostgreSQL back into Elixir structures.

When configuring a Postgrex connection to support vectors, this extension handles the binary protocol requirements, including length-prefixed encoding.

# This module is typically used internally by Postgrex when the 
# pgvector extension is registered in your database connection.
# It implements the Postgrex extension protocol including:
# - init/1: Initializes decoding strategy
# - matching/1: Identifies the 'vector' type
# - encode/1: Converts Pgvector structs to binary
# - decode/1: Converts binary to Pgvector structs

Use Pgvector.new/1 to initialize a vector. It supports creating a vector from a standard Elixir list of floats or an Nx.Tensor (if Nx is loaded). If an Nx.Tensor is provided, it must have a rank of 1.

When using a list, the elements are converted to 32-bit floats.

# From a list
vector = Pgvector.new([1.0, 2.0, 3.0])

# From an Nx.Tensor (requires Nx)
vector = Pgvector.new(Nx.tensor([1.0, 2.0, 3.0]))

# From an existing Pgvector
new_vector = Pgvector.new(vector)

Use Pgvector.to_binary/1 to extract the raw binary data from a Pgvector, Pgvector.HalfVector, or Pgvector.SparseVector. This is typically used when preparing data for database drivers.

binary_data = Pgvector.to_binary(vector)