VennDB

repository·main·Indexed 40 days ago

https://github.com/plabayo/venndb

An append-only, in-memory database in Rust optimized for high-performance querying of static datasets using bit-flag filters. It provides a derive macro to define schemas with primary keys and filterable columns, a fluent query builder API, and support for custom row validation and wildcard matching via the Any trait.

Tokens
4.8K
Snippets
13
Records
20
Agent score
49%

What's inside VennDB

  1. Overview of VennDB

    main
    VennDB is an append-only in-memory database written in Rust. It is specifically optimized for datasets that are mostly static (loaded at startup) and require high-frequency querying using simple bit (flag) column filters. It is designed to be faster and more compact than traditional solutions like SQLite for this specific use case.
  2. Make filter properties optional using `Option<T>`

    main

    Both boolean filters and filter maps (properties with the #[venndb(filter)] attribute) can be made optional by wrapping the type in an Option.

    • Option<bool> for boolean filters.
    • Option<T> for filter maps.

    Rows containing Option::None for an optional column cannot be filtered on that specific property, but they remain valid rows in the database.

  3. Use the `Any` trait to allow rows to match any filter value

    main

    You can allow certain rows to match any value provided in a filter by implementing the Any trait for your property type and using the #[venndb(filter, any)] attribute.

    When a row's property implements Any and its value is considered 'any' (via is_any()), that row will be returned even if the query specifies a different specific value.

    Note: Filtering for an 'any' value will only return rows that have been explicitly marked with that 'any' value. It will not match rows with specific, non-any values.

    use venndb::{Any, VennDB};
    
    #[derive(Debug, PartialEq, Eq, Clone, Hash)]
    pub enum Department {
      Any,
      Hr,
      Engineering,
    }
    
    impl Any for Department {
      fn is_any(&self) -> bool {
        self == Department::Any
      }
    }
    
    #[derive(Debug, VennDB)]
    pub struct Employee {
      name: String,
      #[venndb(filter, any)]
      department: Department,
    }
    
    let db = EmployeeDB::from_iter([
      Employee { name: "Jack".to_owned(), department: Department::Any },
      Employee { name: "Derby".to_owned(), department: Department::Hr },
    ]);
    let mut query = db.query();
    
    // will match Jack and Derby, as Jack is marked as Any
    let hr_employees: Vec<_> = query.department(Department::Hr).execute().unwrap().iter().collect();
    assert_eq!(hr_employees.len(), 2);
  4. Define a database schema with the `VennDB` macro

    main

    To use VennDB, apply the #[derive(VennDB)] macro to a struct. This macro automatically generates a database struct, a query builder, and error types based on your struct's fields.

    You can customize the generated database using the #[venndb(...)] attribute on the struct:

    • name = "Name": Sets the name of the generated database struct (defaults to [StructName]DB).
    • validator = path::to::function: Specifies a function to validate rows before they are appended/inserted.

    Field attributes:

    • #[venndb(key)]: Marks a field as a unique identifier. This enables direct lookups via get_by_[field_name]. Keys must implement Clone and be unique.
    • #[venndb(filter)]: Marks a field as a searchable filter. Non-boolean types (like enums or strings) become 2D filters. These must implement PartialEq + Eq + Hash + Clone.
    • #[venndb(skip)]: Prevents a field (typically a boolean) from being automatically turned into a query filter.

    Note: A property cannot be both a key and a filter simultaneously; doing so will cause a compile-time error.

    #[derive(Debug, VennDB)]
    #[venndb(name = "EmployeeInMemDB", validator = employee_validator)]
    pub struct Employee {
        #[venndb(key)]
        id: u32,
        name: String,
        is_manager: bool,
        is_admin: bool,
        is_active: Option<bool>,
        #[venndb(skip)]
        foo: bool,
        #[venndb(filter)]
        department: Department,
        #[venndb(filter)]
        country: Option<String>,
    }
    
    fn employee_validator(employee: &Employee) -> bool {
        employee.id > 0
    }
  5. Query data using the VennDB Query Builder

    main

    To search for data, call .query() on your database instance to get a query builder.

    1. Compose Filters: Use generated methods named after your fields (e.g., .is_manager(true) or .department(Department::Engineering)) to add constraints.
    2. Execute: Call .execute() to run the query. It returns Option<QueryResult>. If None is returned, no rows matched the filters.
    3. Process Results: If Some(result) is returned, you can:
      • .iter(): Iterate over all matching rows.
      • .first(): Get a reference to the first matched row.
      • .any(): Get a reference to a randomly selected matched row.
      • .filter(|row| ...): Apply an additional closure-based filter to the result set.
    4. Reset: Use .reset() on the query builder to clear all current filters and start a new query composition.

    Note on Optional Booleans: If a field is Option<bool>, the filter methods allow for three states: Some(true), Some(false), and None. None is distinct from false and means the row will not be found if filtered on that property.

    let mut query = db.query();
    query.is_active(true)
         .is_manager(true)
         .department(Department::Engineering);
    
    let manager = query.execute()
        .expect("found at least one")
        .any();
  6. Understand VennDB field types and generated methods

    main

    When using #[derive(VennDB)], the macro parses struct fields into three distinct types, each resulting in different generated API methods. Understanding these types helps you predict how your struct will be used for querying and mapping.

    1. Key Fields

    Used for primary lookups. The macro generates getter and mapping methods.

    • Generated Methods: get_by_<field_name> and map_<field_name>.

    2. Filter Fields

    Used for simple equality or existence filtering.

    • Generated Methods: filter_<field_name> and filter_not_<field_name>.
    • Optionality: If the field is an Option<T>, the filter behavior adjusts accordingly.

    3. FilterMap Fields

    Used for more complex filtering, such as matching against sets or handling 'any' logic.

    • Generated Methods: filter_map_<field_name> and filter_vec_<field_name>.
    • Special Logic: If the field is configured with any logic, a filter_any_<field_name> method is also generated.
  7. Query the VennDB database

    main

    Once a database is initialized (e.g., via TDB::from_iter(...)), you can perform queries using a fluent API. The generated query builder allows you to chain filters based on your struct's fields and then execute the query to retrieve results.

    Common execution methods include .any() to find if any record matches the criteria.

    let db = EmployeeDB::from_iter(/* .. */);
    
    let mut query = db.query();
    let employee = query
        .is_admin(true)
        .is_manager(false)
        .department(Department::Engineering)
        .execute()
        .expect("to have found at least one")
        .any();
    
    println!("non-manager admin engineer: {:?}", employee);
  8. Query multiple variants of a filter map property

    main

    To perform an 'OR' query on a single filter map property (matching rows that have any of several specific values), call the query method for that property multiple times on the same query object.

    use venndb::{Any, VennDB};
    
    #[derive(Debug, VennDB)]
    pub struct Value {
       #[venndb(filter)]
       pub foo: String,
       pub bar: u32,
    }
    
    let db = ValueDB::from_iter([
        Value { foo: "a".to_owned(), bar: 8 },
        Value { foo: "b".to_owned(), bar: 12 },
        Value { foo: "c".to_owned(), bar: 16 },
    ].into_Iter()).unwrap();
    
    let mut query = db.query();
    query.foo("a".to_owned());
    query.foo("c".to_owned());
    
    let values: Vec<_> = query.execute().unwrap().iter().collect();
    assert_eq!(values.len(), 2);
    assert_eq!(values[0].bar, 8);
    assert_eq!(values[1].bar, 16);
  9. Define a database schema with `#[derive(VennDB)]`

    main

    To use VennDB, define a named field struct and derive VennDB. You use attribute macros to configure how fields are treated by the database:

    • #[venndb(key)]: Marks a field as the primary key.
    • #[venndb(filter)]: Marks a field as a filterable column.
    • #[venndb(filter, any)]: Marks a field as a filterable column using an 'any' matching strategy.
    • #[venndb(skip)]: Excludes the field from the database index/storage.

    When you derive VennDB on a struct named T, it generates a corresponding database type named TDB (e.g., Employee becomes EmployeeDB).

    use venndb::VennDB
    
    #[derive(Debug, VennDB)]
    pub struct Employee {
        #[venndb(key)]
        id: u32,
        name: String,
        is_manager: Option<bool>,
        is_admin: bool,
        #[venndb(skip)]
        foo: bool,
        #[venndb(filter, any)]
        department: Department,
        #[venndb(filter)]
        country: Option<String>,
    }
  10. Initialize a VennDB database

    main

    You can create a new database instance using several methods:

    • new() / default(): Creates an empty database with zero capacity.
    • capacity(usize): Creates an empty database with a pre-allocated capacity.
    • from_rows(Vec<T>): Creates a database from a vector of existing instances. If #[venndb(key)] is used, this returns a Result to handle potential duplicate keys.
    • from_iter(impl IntoIterator<Item = impl Into<T>>): Creates a database from an iterator. This is useful for converting different types (like CSV rows) into your database struct on the fly. This also returns a Result if keys are defined.

    If a validator is defined, insertion methods will return an error if the data does not satisfy the validator.

    // Creating from an iterator (e.g., converting RawCsvRow to Employee)
    let db = EmployeeInMemDB::from_iter([
        RawCsvRow("1,John Doe,true,false,true,false,Engineering,USA"),
        RawCsvRow("2,Jane Doe,false,true,true,true,Sales,"),
    ])
    .expect("DB created without errors");
  11. Implement custom row validation with `#[venndb(validator = ...)]`

    main

    To prevent invalid data from being appended to the database, you can provide a custom validation function using the #[venndb(validator = function_name)] attribute on your struct. The validation function is called before a row is appended; if it returns false, the append operation will return an error.

    #[derive(Debug, VennDB)]
    #[venndb(validator = my_validator_fn)]
    pub struct Value {
       pub foo: String,
       pub bar: u32,
    }
    
    fn my_validator_fn(value: &Value) -> bool {
        !value.foo.is_empty() && value.bar > 0
    }
    
    let mut db = ValueDB::default();
    assert!(db.append(Value {
        foo: "".to_owned(),
        bar: 42,
    }).is_err()); // fails because foo == empty