chromem-go

The chromem-go repository provides several example implementations to demonstrate different use cases for the vector database:

• Minimal Example: A bare-bones implementation using OpenAI for embeddings. Best for understanding the core API with minimal boilerplate.
• RAG Wikipedia Ollama: A Retrieval Augmented Generation (RAG) application for question answering. It uses Wikipedia lead sections as a knowledge base and runs both the embedding model and the LLM via Ollama, demonstrating a fully offline RAG setup.
• Semantic Search arXiv OpenAI: A semantic search application that indexes approximately 5,000 arXiv papers (Computer Science - Computation and Language category) using OpenAI embeddings.
• WebAssembly (WASM): Demonstrates how to compile chromem-go to WebAssembly to enable vector database functionality directly in a web browser via JavaScript.
• S3 Export/Import: Shows how to handle database persistence by exporting the database to and importing it from S3-compatible blob storage services.

This example demonstrates how to initialize an in-memory database, create a collection, add documents using OpenAI embeddings (requires OPENAI_API_KEY), and perform a similarity query. Note that passing nil as the embedding function defaults to OpenAI.

package main

import (
 "context"
 "fmt"
 "runtime"

 "github.com/philippgille/chromem-go"
)

func main() {
  ctx := context.Background()

  db := chromem.NewDB()

  // Passing nil as embedding function leads to OpenAI being used and requires
  // "OPENAI_API_KEY" env var to be set. Other providers are supported as well.
  // For example pass `chromem.NewEmbeddingFuncOllama(...)` to use Ollama.
  c, err := db.CreateCollection("knowledge-base", nil, nil)
  if err != nil {
    panic(err)
  }

  err = c.AddDocuments(ctx, []chromem.Document{
    {
      ID:      "1",
      Content: "The sky is blue because of Rayleigh scattering.",
    },
    {
      ID:      "2",
      Content: "Leaves are green because chlorophyll absorbs red and blue light.",
    },
  }, runtime.NumCPU())
  if err != nil {
    panic(err)
  }

  res, err := c.Query(ctx, "Why is the sky blue?", 1, nil, nil)
  if err != nil {
    panic(err)
  }

  fmt.Printf("ID: %v\nSimilarity: %v\nContent: %v\n", res[0].ID, res[0].Similarity, res[0].Content)
}

This example demonstrates how to export a chromem-go database to and import it from any S3-compatible blob storage service. It utilizes gocloud.dev (Google's Cloud Development Kit) to provide a generic interface for interacting with storage providers via readers and writers.

Prerequisites

• An S3-compatible storage service (e.g., MinIO).
• The OPENAI_API_KEY environment variable must be set.

Setup with MinIO (Local Example)

1. Start MinIO via Docker:

   docker run -d --rm --name minio -p 127.0.0.1:9000:9000 -p 127.0.0.1:9001:9001 quay.io/minio/minio:RELEASE.2024-05-01T01-11-10Z server /data --console-address ":9001"

2. Configure Bucket:
   • Access the MinIO Console at http://localhost:9001.
   • Log in with user minioadmin and password minioadmin.
   • Create a bucket named mybucket.
3. Run the Example:

   go run .

Verifying Output

After running, you can verify the exported database file (named chromem.gob.gz) in your bucket via the MinIO browser at http://localhost:9001/browser/mybucket.

# 1. Start MinIO
docker run -d --rm --name minio -p 127.0.0.1:9000:9000 -p 127.0.0.1:9001:9001 quay.io/minio/minio:RELEASE.2024-05-01T01-11-10Z server /data --console-address ":9001"

# 2. Set API Key
export OPENAI_API_KEY="your-key-here"

# 3. Run the application
go run .

This example demonstrates a semantic search application using chromem-go as a vector database. It loads approximately 5,000 arXiv papers from the 'Computer Science - Computation and Language' category and performs semantic searches using OpenAI embeddings.

Prerequisites

1. Dataset: Download arxiv-metadata-oai-snapshot.json from Kaggle.
2. Filtering: Use ripgrep (or grep) to filter the dataset for the cs.CL category and updates from 2023.

   rg '"categories":"cs.CL"' ~/Downloads/arxiv-metadata-oai-snapshot.json | rg '"update_date":"2023' > /tmp/arxiv_cs-cl_2023.jsonl

3. API Key: Set your OpenAI API key in your environment variables:

   export OPENAI_API_KEY='your-api-key-here'

Execution

Run the example using the Go CLI:

go run .

Note: The execution time is primarily dominated by the time taken to create embeddings via the OpenAI API.

# 1. Filter data
rg '"categories":"cs.CL"' ~/Downloads/arxiv-metadata-oai-snapshot.json | rg '"update_date":"2023' > /tmp/arxiv_cs-cl_2023.jsonl

# 2. Set API Key
export OPENAI_API_KEY='your-api-key-here'

# 3. Run
go run .