DOCSEARCHbyAlgolia
Submit Docs

ColiVara Documentation

repository·main·Indexed 23 days ago

https://github.com/tjmlabs/colivara

A retrieval system for Retrieval Augmented Generation (RAG) that uses Vision Language Models to index and search documents based on text and visual cues like tables, layouts, and figures. ColiVara avoids traditional OCR and parsing by treating documents as images and utilizing late-interaction embeddings. It provides Python and TypeScript SDKs for indexing via upsert_document and performing semantic searches, and supports local deployment via Docker Compose with pgVector.

Tokens
2.2K
Snippets
5
Records
9
Agent score
81%

What's inside ColiVara

  1. How ColiVara retrieval works

    main

    Unlike traditional RAG systems that rely on text extraction, parsing, and chunking, ColiVara uses Vision Language Models to generate embeddings.

    • Visual Awareness: It treats documents as images, allowing it to capture information from tables, figures, page layouts, and fonts that text-only parsers miss.
    • No Manual Chunking: There is no OCR or text-to-markdown conversion step; the model retrieves the most relevant pages directly based on visual and textual cues.
    • Late-Interaction Embeddings: It uses late-interaction style embeddings (based on the ColPali paper), which are more accurate than standard pooled embeddings for document retrieval.
    • Managed Infrastructure: ColiVara manages the vector storage (using Postgres and pgVector) so you don't need to manage embeddings yourself.

  3. Local Setup Guide

    main

    To run ColiVara locally, you must first set up the ColiVarE (Embeddings Service) as a separate component.

    1. Configure Environment: Create a .env.dev file in the root with:

      • EMBEDDINGS_URL: The URL of your embeddings service (e.g., http://host.docker.internal:8000/runsync).
      • EMBEDDINGS_URL_TOKEN: An authorization token.
      • AWS_S3_ACCESS_KEY_ID, AWS_S3_SECRET_ACCESS_KEY, AWS_STORAGE_BUCKET_NAME: S3 credentials for storage.

    2. Launch via Docker:

      git clone https://github.com/tjmlabs/ColiVara
docker-compose up -d --build
docker-compose exec web python manage.py migrate
docker-compose exec web python manage.py createsuperuser

    3. Retrieve Auth Token: Run the following to get the token for the newly created superuser:

      docker-compose exec web python manage.py shell
# Inside shell:
from accounts.models import CustomUser
user = CustomUser.objects.first().token 
print(user)

    4. Access: The app runs at http://localhost:8001 and Swagger docs at http://localhost:8001/v1/docs.

    docker-compose up -d --build
docker-compose exec web python manage.py migrate
docker-compose exec web python manage.py createsuperuser

  4. Run ColiVara using Docker Compose

    main

    You can deploy the ColiVara stack using Docker Compose. The setup includes a web service (running Uvicorn), a Gotenberg service for document processing, and a PostgreSQL database with the pgvector extension for vector similarity search.

    To run the services, ensure you have a .env.dev file present in your root directory as the web service depends on it.

  5. Index a document with upsert_document

    main

    Use upsert_document to index a file. ColiVara supports over 100 formats (PDF, DOCX, PPTX, etc.) via URLs, file paths, or base64 encoded strings. The system uses vision models to treat documents as images, avoiding traditional parsing or OCR. You can optionally specify a collection_name, add metadata, and set wait=True to wait for indexing to complete.

    from colivara_py import ColiVara
import os

client = ColiVara(api_key=os.environ.get("COLIVARA_API_KEY"))

document = client.upsert_document(
    name="sample_document",       
    document_url="https://example.com/sample.pdf",
    metadata={"author": "John Doe"},        
    collection_name="user_1_collection",    
    wait=True                               
)

  6. Search documents with ColiVara

    main

    Perform semantic searches using the search method. You can perform simple queries, restrict searches to a specific collection_name, or apply complex query_filter objects to filter by document or collection metadata.

    # Simple search
results = client.search("what is 1+1?")

# Search with a specific collection
results = client.search("what is 1+1?", collection_name="user_1_collection")

# Search with a filter on document metadata
results = client.search(
    "what is 1+1?",
    query_filter={
        "on": "document",
        "key": "author",
        "value": "John Doe",
        "lookup": "key_lookup",  # or 'contains'
    },
)

# Search with a filter on collection metadata
results = client.search(
    "what is 1+1?",
    query_filter={
        "on": "collection",
        "key": ["tag1", "tag2"],
        "lookup": "has_any_keys",
    },
)

  7. Search documents with ColiVara (TypeScript)

    main

    Perform semantic searches using the TypeScript SDK. The search method accepts an object containing the query and optional parameters like collection_name and query_filter.

    import { ColiVara } from 'colivara-ts';
const client = new ColiVara('your-api-key');

// Simple search
const results = await client.search({query: "what is 1+1?"})

// search with a specific collection
const results = await client.search({query: "what is 1+1?", collection_name: "user_1_collection"})

// Search with a filter on document metadata
const results = await client.search({
    query: "what is 1+1?",
    query_filter: {
        on: "document",
        key: "author",
        value: "John Doe",
        lookup: "key_lookup"
    }
})

// search with a filter on collection metadata
const results = await client.search({
    query: "what is 1+1?",
    query_filter: {
        on: "collection",
        key: ["tag1", "tag2"],
        lookup: "has_any_keys"
    }
})

  8. Reference the ColiVara Docker Compose service configuration

    main

    The following services are defined in the docker-compose.yml file:

    • web: The main application service. It builds from ./web, maps host port 8001 to container port 8000, and uses an environment file ./.env.dev. It runs the command uvicorn config.asgi:application --reload --host 0.0.0.0 --port 8000.
    • gotenberg: Uses the gotenberg/gotenberg:8 image.
    • db: Uses the pgvector/pgvector:pg16 image. It uses a named volume postgres_data for persistence and is configured with POSTGRES_HOST_AUTH_METHOD=trust for local development.
    services:
  web:
    build: ./web
    ports:
      - "8001:8000"
    command: uvicorn config.asgi:application --reload --host 0.0.0.0 --port 8000
    volumes:
      - ./web:/code
    depends_on:
      - db
    env_file:
      - ./.env.dev

  gotenberg:
    image: gotenberg/gotenberg:8

  db:
    image: pgvector/pgvector:pg16
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"

volumes:
  postgres_data:

  9. Update embeddings for all documents via update_embeddings command

    main

    The update_embeddings management command is used to batch update embeddings for all existing Page documents in the database. This is typically required when upgrading the base embedding model to ensure all document vectors are consistent with the new model.

    It processes pages in batches of 100, sends the base64 image data to the configured EMBEDDINGS_URL, and replaces existing PageEmbedding records with the new vectors within an atomic transaction.