DOCSEARCHbyAlgolia
Submit Docs

Zvec Vector Database

repository·main·Indexed 12 days ago

https://github.com/alibaba/zvec

An open-source, in-process vector database engine with a native C++ backend and bindings for Python, Node.js, Rust, Dart/Flutter, and Go. Zvec supports dense and sparse vectors, hybrid search (vector + full-text), and various index types including Flat and HNSW. It is designed to be embedded directly into applications for low-latency similarity search and scalable vector indexing with durable storage via Write-Ahead Logging (WAL).

Tokens
13K
Snippets
35
Records
49
Agent score
96%

What's inside Zvec

  1. Core features of Zvec

    main

    Zvec is a lightweight, in-process vector database designed for low-latency similarity search. Key capabilities include:

    • Dense + Sparse Vectors: Supports both dense and sparse embeddings, multi-vector queries, and various index types (Flat, HNSW, HNSW-RaBitQ, etc.).
    • Hybrid Search: Allows fusing vector similarity, full-text search (FTS), and structured filters in a single query.
    • Full-Text Search (FTS): Native keyword-based search for string fields using natural-language or structured expressions.
    • Durable Storage: Uses Write-Ahead Logging (WAL) to ensure data persistence during crashes or power failures.
    • Concurrency Model: Supports multiple simultaneous reader processes; however, writes are exclusive to a single process.
    • In-Process Execution: Runs within your application's process (notebooks, servers, edge devices) without requiring a separate server.

  2. Core concepts of Zvec

    main

    Zvec is designed as an embedded vector database with the following key characteristics:

    • In-Process Execution: No separate server deployment is required. It runs directly within your application process, making it suitable for Notebooks, edge devices, and high-performance servers.
    • Hybrid Search: Supports combining dense vectors, sparse vectors, and Full-Text Search (FTS) in a single query.
    • Persistence: Uses Write-Ahead Logging (WAL) to ensure data durability even during process crashes or power failures.
    • Concurrency Model: Supports multiple processes reading the same Collection simultaneously, while writing is restricted to a single process in exclusive mode.
    • Indexing: Offers various vector index types (e.g., Flat, HNSW, HNSW-RaBitQ, Sparse) that can reside in memory or on disk.

  3. Quickstart: Vector similarity search in Python

    main

    This example demonstrates the full lifecycle of using Zvec in Python: defining a schema, creating a collection, inserting documents with vectors, and performing a similarity search.

    1. Define Schema: Use zvec.CollectionSchema and zvec.VectorSchema to specify the collection name and vector dimensions/types.
    2. Create/Open Collection: Use zvec.create_and_open(path, schema) to initialize the database at a local path.
    3. Insert Documents: Use collection.insert() with a list of zvec.Doc objects. Each doc requires an id and a vectors dictionary mapping field names to vector arrays.
    4. Query: Use collection.query() with a zvec.Query object specifying the field_name and the target vector.

    Results are returned as a list of dictionaries containing 'id', 'score', and other metadata, sorted by relevance.

    import zvec

# Define collection schema
schema = zvec.CollectionSchema(
    name="example",
    vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4),
)

# Create collection
collection = zvec.create_and_open(path="./zvec_example", schema=schema)

# Insert documents
collection.insert([
    zvec.Doc(id="doc_1", vectors={"embedding": [0.1, 0.2, 0.3, 0.4]}),
    zvec.Doc(id="doc_2", vectors={"embedding": [0.2, 0.3, 0.4, 0.1]}),
])

# Search by vector similarity
results = collection.query(
    zvec.Query(field_name="embedding", vector=[0.4, 0.3, 0.3, 0.1]),
    topk=10
)

# Results: list of {'id': str, 'score': float, ...}, sorted by relevance
print(results)

  4. Quickstart: Create a collection and perform vector search in Python

    main

    This example demonstrates the complete lifecycle: defining a schema, creating/opening a collection, inserting documents, and performing a similarity search.

    1. Define Schema: Use zvec.CollectionSchema and zvec.VectorSchema to specify the collection name and vector dimensions/types.
    2. Create/Open Collection: Use zvec.create_and_open(path, schema) to initialize the database at a specific local path.
    3. Insert Documents: Use collection.insert() with a list of zvec.Doc objects. Each doc requires an id and a vectors dictionary.
    4. Query: Use collection.query() with a zvec.Query object specifying the field_name, the target vector, and topk results.

    Results are returned as a list of dictionaries containing id, score, and other metadata, sorted by relevance.

    import zvec

# Define collection schema
schema = zvec.CollectionSchema(
    name="example",
    vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4),
)

# Create collection
collection = zvec.create_and_open(path="./zvec_example", schema=schema)

# Insert documents
collection.insert([
    zvec.Doc(id="doc_1", vectors={"embedding": [0.1, 0.2, 0.3, 0.4]}),
    zvec.Doc(id="doc_2", vectors={"embedding": [0.2, 0.3, 0.4, 0.1]}),
])

# Vector similarity search
results = collection.query(
    zvec.Query(field_name="embedding", vector=[0.4, 0.3, 0.3, 0.1]),
    topk=10
)

# Results: list of {'id': str, 'score': float, ...}, sorted by relevance
print(results)

  5. Install Zvec via official SDKs

    main

    Zvec is an in-process vector database that can be embedded directly into your applications. You can install it using the following package managers depending on your language:

    • Python: pip install zvec (requires 64-bit Python 3.10–3.14)
    • Node.js: npm install @zvec/zvec
    • Rust: cargo add zvec-rust
    • Dart/Flutter: flutter pub add zvec
    • Go: Use the zvec-go bindings.

    Supported platforms include Linux (x86_64, ARM64), macOS (ARM64), and Windows (x86_64).

    pip install zvec

  6. Build zvec from source

    main

    To build the zvec source code, clone the repository, initialize submodules, and use cmake to generate the build files. Note that -DENABLE_SKYLAKE=ON is used in the example to enable specific optimizations.

    $ git clone git@github.com:alibaba/zvec.git
$ cd zvec
$ git submodule update --init

$ mkdir build
$ cd build
$ cmake -DENABLE_SKYLAKE=ON -DCMAKE_BUILD_TYPE=Release ..

  7. Run COHERE benchmarking in a Docker container

    main

    To avoid manual data setup, you can use a pre-configured Docker image that contains the COHERE benchmark datasets. The datasets are located at /tmp/cohere/ inside the container.

    Run the following commands to start the container and enter its shell:

    docker run -it --net=host -d -e DEBUG_MODE=true  --user root --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /home/zvec/:/home/zvec/  -w /home/zvec --name=cohere_bench zvec-registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data:0.0.1 bash

docker exec -it cohere_bench bash

  8. Run Index Building, Recall, and Benchmarking

    main

    After configuring your YAML files, use the compiled binaries in the build/bin directory to perform the following tasks:

    1. Build the Index: Uses the builder configuration.
    2. Conduct Recall: Evaluates search accuracy.
    3. Conduct Bench: Performs performance benchmarking.
    # Build the index
$ /home/zvec/workspace/zvec/build/bin/local_build_original ./build.yaml 

# Conduct recall
$ /home/zvec/workspace/zvec/build/bin/recall_original ./search.yaml

# Conduct benchmarking
$ /home/zvec/workspace/zvec/build/bin/bench_original ./search.yaml

  9. Convert COHERE dataset to zvec binary format

    main

    If you are using raw COHERE parquet data, you must first export the vector data using a Python script and then convert it to the binary .zvec.vecs format using the txt2vecs tool.

    # 1. Export vector data using python
$ mkdir 10m.output
$ python3 convert_cohere_parquet.py

# 2. Convert to binary formatted file
/home/zvec/workspace/zvec/bin/txt2vecs --input=cohere_train_vector_10m.txt --output=cohere_train_vector_10m.zvec.vecs --dimension=768

  10. Configure the Index Builder (YAML)

    main

    The BuilderCommon section defines how the index is constructed. Key parameters include:

    • BuilderClass: The streamer class to use (e.g., HnswStreamer).
    • BuildFile: Path to the .zvec.vecs binary file.
    • ConverterName: The quantization/conversion method (e.g., CosineInt8Converter).
    • MetricName: The distance metric (e.g., Cosine).
    • ThreadCount: Number of threads for the build process.
    BuilderCommon:
    BuilderClass: HnswStreamer
    BuildFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs
    NeedTrain: true 
    TrainFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs
    DumpPath:  /home/zvec/bench/config/cohere_train_vector_10m.dump.index
    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index

    ConverterName: CosineInt8Converter
    MetricName: Cosine

    ThreadCount: 16

BuilderParams: 
    proxima.general.builder.thread_count: !!int 16
    proxima.hnsw.builder.thread_count: !!int 16

  11. Manage mutable vector data with VectorDataBuffer

    main

    When you need to pass mutable vector data (e.g., for fetching vectors from an index), use the buffer structures. These use std::string internally to manage memory safely.

    • DenseVectorBuffer: Contains a std::string data field.
    • SparseVectorBuffer: Contains count, indices (string), and values (string). Use get_indices() to get a uint32_t* and get_values<T>() to get a T* for writing data into the buffer.
    struct SparseVectorBuffer {
  uint32_t count;
  std::string indices;
  std::string values;

uint32_t *get_indices() {
    return reinterpret_cast<uint32_t *>(indices.data());
}

template <typename T = void>
  T *get_values() {
    return reinterpret_cast<T *>(values.data());
  }
};

  12. Configure the Searcher (YAML)

    main

    The SearcherCommon section defines how queries are executed against the index. Key parameters include:

    • IndexPath: Path to the generated .index file.
    • TopK: A list of K values to evaluate (e.g., 1,10,50,100).
    • QueryFile: Path to the text file containing query vectors.
    • GroundTruthFile: Path to the file containing true neighbors for recall calculation.
    • BenchThreadCount: Number of threads for benchmarking.
    • SearcherParams: Contains engine-specific tuning like proxima.hnsw.streamer.ef.
    SearcherCommon:
    SearcherClass: HnswStreamer
    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index
    TopK: 1,10,50,100 
    QueryFile: /tmp/cohere/cohere_large_10m_zvec/cohere_test_vector_1000.new.txt
    QueryType: float 
    QueryFirstSep: ";" 
    QuerySecondSep: " "
    GroundTruthFile: /tmp/cohere/cohere_large_10m_zvec/neighbors.txt
    RecallThreadCount: 1
    BenchThreadCount: 16 
    BenchIterCount: 1000000000 
    CompareById: true

SearcherParams: 
    proxima.hnsw.streamer.ef: !!int 250