DOCSEARCHbyAlgolia
Submit Docs

OasysDB

repository·main·Indexed 18 days ago

https://github.com/edwinkys/oasysdb

A Rust-based database project implementing Approximate Nearest Neighbor Search (ANNS) using a modified IVF algorithm with dynamic cluster splitting via KMeans. It supports vector similarity queries with Euclidean and Cosine metrics, metadata filtering, and persistence via snapshots. Note: This repository is currently unmaintained and kept for historical purposes.

Tokens
6.3K
Snippets
30
Records
33
Agent score
63%

What's inside oasysdb

  3. Follow the OasysDB contribution workflow

    main

    To avoid wasting effort on features or fixes that may not align with the project roadmap, follow this workflow:

    1. Check existing issues/PRs: Ensure your intended work isn't already being addressed.
    2. Open an issue first: If you want to work on a new feature or a fix not currently in the tracker, open an issue to discuss it with maintainers and the community before writing code.
    3. Provide use cases: For new features, include real-world use cases in your issue to help prioritize the request.

  4. Construct filters from strings

    main

    OasysDB allows you to create complex query filters by parsing strings. You can define individual Filter objects or group them into Filters using logical join operators.

    Individual Filters

    A single filter follows the format: <key> <operator> <value>.

    • Example: name CONTAINS Ada or age >= 21.
    • Note: The string must contain exactly three parts separated by spaces.

    Filter Groups (Logical Joins)

    You can join multiple filters using AND or OR operators.

    • Supported Joins: AND, OR.
    • Limitation: You cannot mix AND and OR in a single filter string (e.g., a = 1 AND b = 2 OR c = 3 is invalid).
    • Example: gpa >= 3.0 OR age < 21.
    use oasysdb::types::filter::{Filters, Filter};

// Create a single filter
let single_filter = Filter::try_from("name CONTAINS Ada")?;

// Create a group of filters using OR
let filter_group = Filters::try_from("gpa >= 3.0 OR age < 21")?;

// Create a group of filters using AND
let filter_group_and = Filters::try_from("age >= 20 AND gpa < 4.0")?;

  5. Configure the OasysDB database

    main

    Use Database::configure to initialize a new database instance with specific settings. This method sets up the database directory and creates an initial snapshot.

    Warning: If the database directory already exists, the process will prompt you in the terminal to overwrite the existing configuration. If you choose not to overwrite, the configuration will not proceed.

    Required configuration settings are provided via the Parameters struct.

    let params = Parameters {
    dimension: 128,
    metric: Metric::Euclidean,
    density: 64,
};

Database::configure(&params);

  6. Set the database directory via ODB_DIR

    main
    OasysDB determines its storage location using the ODB_DIR environment variable. If this variable is set, the database will use that path. If it is unset, the database defaults to a directory named oasysdb in the current working directory.

  7. OasysDB coding style and dependencies

    main

    Style Guide

    • Linting: Uses default Rust linting with specific overrides defined in rustfmt.toml.
    • Comments: Write clear, concise comments using proper English sentence capitalization and punctuation.

    Key Dependencies

    Familiarity with these third-party libraries is helpful for contributing:

    • gRPC
    • Tonic
    • Tokio

  8. Get the string representation of a Metric

    main

    Use the .as_str() method on a Metric instance to retrieve its name as a lowercase string slice. This is useful for serialization or logging.

    Supported return values:

    • "euclidean" for Metric::Euclidean
    • "cosine" for Metric::Cosine
    let metric = Metric::Euclidean;
assert_eq!(metric.as_str(), "euclidean");

  9. Configure Index parameters

    main

    When creating an Index, you can use a builder-style pattern to configure its behavior:

    • with_metric(metric: Metric): Sets the distance metric used for calculations (e.g., Metric::Euclidean).
    • with_density(density: usize): Sets the maximum number of records allowed in a cluster before a split is triggered. The default is 256.
    let index = Index::new()
    .with_metric(Metric::Euclidean)
    .with_density(512);

  10. Open an existing OasysDB instance

    main

    Use Database::open() to load an existing database from disk. This method restores the Parameters, Index, and Storage from the files stored in the database directory.

    It uses the directory specified by the ODB_DIR environment variable, or defaults to oasysdb if the variable is not set.

    let db = Database::open().expect("Failed to open database");

  11. Manage vector search with the Index struct

    main

    The Index struct is the core component for Approximate Nearest Neighbor Search (ANNS) in OasysDB. It uses a modified IVF (Inverted File) algorithm that allows clusters to grow and split dynamically using KMeans to maintain a balanced structure.

    Key capabilities:

    • Initialization: Create an index with custom distance metrics and cluster density.
    • Insertion: Add records to the index. If a cluster exceeds the configured density, the index automatically splits the cluster using KMeans.
    • Deletion: Remove records by their RecordID.
    • Querying: Perform vector similarity searches with support for metadata filtering and search radius constraints.
    // Initialize an index with custom configuration
let mut index = Index::new()
    .with_metric(Metric::Euclidean)
    .with_density(256);

// Insert a record
// Note: 'records' must contain the full mapping of RecordID to Record
index.insert(&id, &record, &records)?;

// Query the index
let results = index.query(
    &query_vector,
    10,               // k: number of neighbors
    &Filters::None,  // metadata filters
    &params,          // query parameters (probes, radius)
    &records          // full record storage
)?;

  12. Convert between Vector and protos::Vector

    main

    You can convert between the internal Vector type and the protos::Vector type (used for serialization/protobufs) using the From and TryFrom traits.

    // Vector to protos::Vector
let proto_vec: protos::Vector = Vector::from(my_vector);

// protos::Vector to Vector
let my_vector = Vector::try_from(proto_vec).expect("Conversion failed");