VectorDBBench (VDBBench)

repository·main·Indexed 22 days ago

https://github.com/zilliztech/vectordbbench

A comprehensive benchmarking tool for evaluating the performance, cost-effectiveness, and retrieval capabilities of various vector databases and cloud services. It supports a wide range of clients including PyMilvus, Qdrant, Pinecone, Weaviate, and others. Key features include an intuitive visual interface, support for real-world scenarios (insertion, searching, and filtered searching), and BM25-style full-text search benchmarking. It allows users to reproduce results using public datasets or custom Parquet-formatted data.

Tokens
38.6K
Snippets
131
Records
178
Agent score
78%

What's inside VectorDBBench

  1. What is VectorDBBench (VDBBench)

    main

    VectorDBBench (VDBBench) is a benchmarking tool designed to compare the performance and cost-effectiveness of mainstream vector databases and cloud services. It is intended for both professionals and non-professionals to reproduce benchmark results or test new systems.

    Key features include:

    • Intuitive Visual Interface: For initiating benchmarks and viewing comparative result reports.
    • Cost-Effectiveness Reports: Specifically for cloud services to provide realistic benchmarking.
    • Real-world Scenarios: Testing includes insertion, searching, and filtered searching.
    • Public Datasets: Uses datasets like SIFT, GIST, Cohere, and OpenAI-generated datasets from C4 to ensure credible data.
    • Full Text Search Support: As of June 2026, it supports benchmarking BM25-style retrieval (e.g., using MS MARCO and HotpotQA datasets) including metrics for payload profiles, recall, QPS, and load.
  2. Overview of Benchmark Case types

    main

    VDBBench categorizes benchmark cases into several types to evaluate different database capabilities:

    • Capacity Cases: Tests loading capacity using Large Dim (e.g., 960 dim) or Small Dim (e.g., 128 dim) vectors.
    • Search Performance Cases: Measures latency, recall, and QPS across different dataset scales (XLarge, Large, Medium, Small).
    • Filtering Search Performance: Evaluates performance with Int-Filter (integer comparisons) or Label-Filter (string/label comparisons).
    • Full Text Search (FTS): Measures BM25-style retrieval using MS MARCO or HotpotQA datasets. Supports comparing throughput with different payload profiles (IDs-only vs. text).
    • Streaming Cases: The Insertion-Under-Load case evaluates search performance while maintaining a constant stream of insert requests.
    • Cloud-Specific Cases: Specialized cases for managed services including CloudInsertCase, CloudPayloadSearchCase, CloudMultiTenantSearchCase, and CloudColdLatencyCase.
  3. Understand VectorDBBench timeout policies

    main

    VectorDBBench implements a multi-tiered timeout strategy to ensure benchmarks are representative of real-world production environments while remaining practical. Timeouts are applied to different stages of the benchmark process.

    Timeout Types

    • Capacity Case Timeout: An overall timeout for capacity-specific tests.
    • Data Loading Timeout: Filters out systems that are too slow at inserting data, ensuring they can handle real-world production demands.
    • Optimization Preparation Timeout: Prevents testing optimization strategies that might work in a benchmark environment but fail to deliver in real production scenarios.

    Timeout Values Reference

    CaseData SizeTimeout TypeValue
    Capacity CaseN/ALoading timeout24 hours
    Other Cases1M vectors (768d) / 500K vectors (1536d)Loading timeout2.5 hours
    Optimization timeout15 mins
    Other Cases10M vectors (768d) / 5M vectors (1536d)Loading timeout25 hours
    Optimization timeout2.5 hours
    Other Cases100M vectors (768d)Loading timeout250 hours
    Optimization timeout25 hours
  4. Understand the VDBBench scoring mechanism

    main

    VDBBench uses relative scoring to ensure different cases are weighted equally regardless of absolute values.

    Scoring Rules

    1. QPS and QP$: Based on the highest value (base_QPS or base_QP$).
      • Score = (Value / base_Value) * 100
    2. Latency: Based on the lowest value (base_Latency). A 10ms offset is added to prevent infinite scores when latency approaches zero.
      • Score = (base_Latency + 10ms) / (Latency + 10ms) * 100
    3. Failures/Timeouts: Systems that fail receive a score based on a value worse than the worst result by a factor of two (e.g., half the lowest QPS or twice the maximum Latency).
    4. Comprehensive Score: The final score for a system is the geometric mean of its scores across all cases.
  5. Add a new database client to VectorDBBench

    main

    To extend VectorDBBench with support for a new vector database, you must implement a client that follows the project's architecture. This involves creating client files, implementing the required abstract classes, and registering the client in the system.

    Step 1: Create Client Files

    1. Navigate to vectordb_bench/backend/clients/.
    2. Create a new directory for your client (e.g., new_client).
    3. Create new_client.py and config.py inside that directory.

    Step 2: Implement Logic

    • new_client.py: Define a class that inherits from VectorDB (found in clients/api.py). You must implement all abstract methods defined in the VectorDB class.
    • config.py: Implement DBConfig (required) and optionally DBCaseConfig. Use pydantic.SecretStr for sensitive data like tokens or passwords.

    Step 3: Register the Client

    Update clients/__init__.py to make the client discoverable:

    1. Add the client to the DB Enum.
    2. Update the init_cls property to return your NewClient class when the enum matches.
    3. Update the config_cls property to return your DBConfig class.
    4. Update case_config_cls if you implemented DBCaseConfig.

    Step 4: Enable CLI Support (Optional)

    To allow users to run your client via the command line:

    1. Create vectordb_bench/backend/clients/new_client/cli.py.
    2. Define a TypedDict (inheriting from CommonTypedDict) to map CLI options (using click) to configuration parameters.
    3. Implement a command function using the @cli.command() decorator and the run() helper.
    4. Register the command in vectordb_bench/cli/vectordbbench.py.
    # Example new_client.py
    from ..api import VectorDB
    class NewClient(VectorDB):
        # Implement the abstract methods defined in the VectorDB class
        ...
    
    # Example config.py
    from pydantic import SecretStr
    from clients.api import DBConfig, DBCaseConfig
    
    class NewDBConfig(DBConfig):
        token: SecretStr
        uri: str
    
    class NewDBCaseConfig(DBCaseConfig):
        # Implement optional case-specific configuration fields
        ...
  6. Install VectorDBBench for development

    main

    To install VectorDBBench and its dependencies for development, use pip to install the package in editable mode. You can include specific extras like [test] for testing or [pinecone] for Pinecone support.

    pip install -e '.[test]'
    pip install -e '.[pinecone]'
    pip install -e '.[test]'
    
    pip install -e '.[pinecone]'
  7. Install VectorDBBench

    main

    VectorDBBench requires python >= 3.11. You can install the base package which includes PyMilvus and Zilliz Cloud support by default, or install specific database clients using extras.

    Base installation (includes PyMilvus/Zilliz Cloud):

    pip install vectordb-bench

    Install specific database clients: Use the syntax pip install 'vectordb-bench[client_name]' to install support for your target database.

    pip install vectordb-bench
  8. Run the VectorDBBench test server

    main

    You can start the VDBBench test server using the Python module command or the init_bench utility.

    If you are using a dev container, ensure you have created a local dataset directory and mounted it to /tmp/vectordb_bench/dataset inside the container before running the server.

    # Option 1: Using python module
    python -m vectordb_bench
    
    # Option 2: Using init_bench
    init_bench
    python -m vectordb_bench
  9. How to run a benchmark test via the Web UI

    main

    VDBBench provides a web interface to configure and execute benchmarks. The workflow follows these steps:

    1. Select Systems: Choose one or more vector databases to test. A form will appear to collect connection details. Use db_label to differentiate instances of the same system (e.g., by specifying host size or instance type).
    2. Select Test Cases: Choose one or multiple benchmark cases. A form will appear to collect specific parameters for those cases.
    3. Submit Task: Provide a unique task label.
      • Warning: Using the same label for different tests will overwrite previous results for that label.
      • Note: Only one task can run at a time.

    Results can be viewed on the main Result Page, which allows comparing multiple tests simultaneously.

  10. Configure custom datasets for performance testing

    main

    You can use the /custom page to run performance cases using your own local datasets. The data must follow a strict Parquet format within a specified folder.

    Required Files

    • train.parquet: The vector data. Must contain columns id (incrementing int) and emb (array of float32).
      • For split files, use the format: train-[index]-of-[file_count].parquet (e.g., train-01-of-10.parquet).
    • test.parquet: The query vectors. Must contain columns id (incrementing int) and emb (array of float32).
    • neighbors.parquet: The ground truth. Must contain columns id (matching query vectors) and neighbors_id (array of int).

    Shuffled Data

    If you enable the Use Shuffled Data option, VDBBench expects files prefixed with shuffle_ (e.g., shuffle_train.parquet). In this mode, the id column can be in any order.

    # Required Parquet Schema:
    # train.parquet: [id (int), emb (float32[])]
    # test.parquet:  [id (int), emb (float32[])]
    # neighbors.parquet: [id (int), neighbors_id (int[])]
  11. How to implement a CLI command for a new client

    main

    When adding a new client, you can enable command-line execution by defining a CLI module. This uses click and pydantic to map command-line arguments to the client's configuration.

    Key components:

    • CommonTypedDict: Used to define the expected CLI arguments.
    • @click.option: Used within the Annotated type hints to define flags (e.g., --uri, --password).
    • run(): The core function that executes the benchmark with the provided db, db_config, and db_case_config.

    Note: For databases with multiple index configurations (like pgvector or milvus), you should repeat this process for each index configuration type.

    from typing import Annotated, Unpack
    import click
    import os
    from pydantic import SecretStr
    from vectordb_bench.cli.cli import (
        CommonTypedDict,
        cli,
        click_parameter_decorators_from_typed_dict,
        run,
    )
    from vectordb_bench.backend.clients import DB
    
    class ZillizTypedDict(CommonTypedDict):
        uri: Annotated[
            str, click.option("--uri", type=str, help="uri connection string", required=True)
        ]
        user_name: Annotated[
            str, click.option("--user-name", type=str, help="Db username", required=True)
        ]
        password: Annotated[
            str,
            click.option(
                "--password",
                type=str,
                help="Zilliz password",
                default=lambda: os.environ.get("ZILLIZ_PASSWORD", ""),
                show_default="$ZILLIZ_PASSWORD",
            ),
        ]
        level: Annotated[
            str, click.option("--level", type=str, help="Zilliz index level", required=False),
        ]
    
    @cli.command()
    @click_parameter_decorators_from_typed_dict(ZillizTypedDict)
    def ZillizAutoIndex(**parameters: Unpack[ZillizTypedDict]):
        from .config import ZillizCloudConfig, AutoIndexConfig
    
        run(
            db=DB.ZillizCloud,
            db_config=ZillizCloudConfig(
                db_label=parameters["db_label"],
                uri=SecretStr(parameters["uri"]),
                user=parameters["user_name"],
                password=SecretStr(parameters["password"]),
            ),
            db_case_config=AutoIndexConfig(
                params={parameters["level"]},
            ),
            **parameters,
        )
  12. Run AWS OpenSearch (Standard and Serverless)

    main

    Standard OpenSearch

    Use the awsopensearch command.

    Key Options:

    • --number-of-shards: Number of primary shards.
    • --number-of-replicas: Number of replica copies.
    • --engine: Type of engine (faiss, lucene, s3vector).
    • --quantization-type: Type of quantization (fp32, fp16, bq).

    OpenSearch Serverless (AOSS)

    To run on AOSS, use the --serverless flag. This uses AWS SigV4 authentication.

    Prerequisites:

    • AWS credentials configured.
    • pip install 'vectordb-bench[opensearch]' installed.
    • IAM policy aoss:APIAccessAll granted.

    Note: In Serverless mode, --user and --password are not needed. Options like --force-merge-enabled and --refresh-interval are ignored as AOSS manages these.

    # Example: OpenSearch Serverless
    vectordbbench awsopensearch --db-label aoss \
      --serverless --aws-region us-east-1 \
      --host <collection-id>.aoss.us-east-1.on.aws --port 443 \
      --case-type Performance768D1M \
      --m 16 --ef-construction 200 --ef-search 40 \
      --number-of-shards 8 --number-of-replicas 0 \
      --engine faiss --metric-type cosine \
      --num-concurrency 80,100,120