VennDB
repository·main·Indexed 40 days ago
https://github.com/plabayo/venndbAn append-only, in-memory database in Rust optimized for high-performance querying of static datasets using bit-flag filters. It provides a derive macro to define schemas with primary keys and filterable columns, a fluent query builder API, and support for custom row validation and wildcard matching via the Any trait.
What's inside VennDB
- VennDB is an append-only in-memory database written in Rust. It is specifically optimized for datasets that are mostly static (loaded at startup) and require high-frequency querying using simple bit (flag) column filters. It is designed to be faster and more compact than traditional solutions like SQLite for this specific use case.
Make filter properties optional using `Option<T>`
mainBoth boolean filters and filter maps (properties with the
#[venndb(filter)]attribute) can be made optional by wrapping the type in anOption.Option<bool>for boolean filters.Option<T>for filter maps.
Rows containing
Option::Nonefor an optional column cannot be filtered on that specific property, but they remain valid rows in the database.Use the `Any` trait to allow rows to match any filter value
mainYou can allow certain rows to match any value provided in a filter by implementing the
Anytrait for your property type and using the#[venndb(filter, any)]attribute.When a row's property implements
Anyand its value is considered 'any' (viais_any()), that row will be returned even if the query specifies a different specific value.Note: Filtering for an 'any' value will only return rows that have been explicitly marked with that 'any' value. It will not match rows with specific, non-any values.
use venndb::{Any, VennDB}; #[derive(Debug, PartialEq, Eq, Clone, Hash)] pub enum Department { Any, Hr, Engineering, } impl Any for Department { fn is_any(&self) -> bool { self == Department::Any } } #[derive(Debug, VennDB)] pub struct Employee { name: String, #[venndb(filter, any)] department: Department, } let db = EmployeeDB::from_iter([ Employee { name: "Jack".to_owned(), department: Department::Any }, Employee { name: "Derby".to_owned(), department: Department::Hr }, ]); let mut query = db.query(); // will match Jack and Derby, as Jack is marked as Any let hr_employees: Vec<_> = query.department(Department::Hr).execute().unwrap().iter().collect(); assert_eq!(hr_employees.len(), 2);Define a database schema with the `VennDB` macro
mainTo use VennDB, apply the
#[derive(VennDB)]macro to a struct. This macro automatically generates a database struct, a query builder, and error types based on your struct's fields.You can customize the generated database using the
#[venndb(...)]attribute on the struct:name = "Name": Sets the name of the generated database struct (defaults to[StructName]DB).validator = path::to::function: Specifies a function to validate rows before they are appended/inserted.
Field attributes:
#[venndb(key)]: Marks a field as a unique identifier. This enables direct lookups viaget_by_[field_name]. Keys must implementCloneand be unique.#[venndb(filter)]: Marks a field as a searchable filter. Non-boolean types (like enums or strings) become 2D filters. These must implementPartialEq + Eq + Hash + Clone.#[venndb(skip)]: Prevents a field (typically a boolean) from being automatically turned into a query filter.
Note: A property cannot be both a
keyand afiltersimultaneously; doing so will cause a compile-time error.#[derive(Debug, VennDB)] #[venndb(name = "EmployeeInMemDB", validator = employee_validator)] pub struct Employee { #[venndb(key)] id: u32, name: String, is_manager: bool, is_admin: bool, is_active: Option<bool>, #[venndb(skip)] foo: bool, #[venndb(filter)] department: Department, #[venndb(filter)] country: Option<String>, } fn employee_validator(employee: &Employee) -> bool { employee.id > 0 }Query data using the VennDB Query Builder
mainTo search for data, call
.query()on your database instance to get a query builder.- Compose Filters: Use generated methods named after your fields (e.g.,
.is_manager(true)or.department(Department::Engineering)) to add constraints. - Execute: Call
.execute()to run the query. It returnsOption<QueryResult>. IfNoneis returned, no rows matched the filters. - Process Results: If
Some(result)is returned, you can:.iter(): Iterate over all matching rows..first(): Get a reference to the first matched row..any(): Get a reference to a randomly selected matched row..filter(|row| ...): Apply an additional closure-based filter to the result set.
- Reset: Use
.reset()on the query builder to clear all current filters and start a new query composition.
Note on Optional Booleans: If a field is
Option<bool>, the filter methods allow for three states:Some(true),Some(false), andNone.Noneis distinct fromfalseand means the row will not be found if filtered on that property.let mut query = db.query(); query.is_active(true) .is_manager(true) .department(Department::Engineering); let manager = query.execute() .expect("found at least one") .any();- Compose Filters: Use generated methods named after your fields (e.g.,
Install VennDB
mainAdd
venndbas a dependency to your Rust project usingcargo add.cargo add venndbUnderstand VennDB field types and generated methods
mainWhen using
#[derive(VennDB)], the macro parses struct fields into three distinct types, each resulting in different generated API methods. Understanding these types helps you predict how your struct will be used for querying and mapping.1. Key Fields
Used for primary lookups. The macro generates getter and mapping methods.
- Generated Methods:
get_by_<field_name>andmap_<field_name>.
2. Filter Fields
Used for simple equality or existence filtering.
- Generated Methods:
filter_<field_name>andfilter_not_<field_name>. - Optionality: If the field is an
Option<T>, the filter behavior adjusts accordingly.
3. FilterMap Fields
Used for more complex filtering, such as matching against sets or handling 'any' logic.
- Generated Methods:
filter_map_<field_name>andfilter_vec_<field_name>. - Special Logic: If the field is configured with
anylogic, afilter_any_<field_name>method is also generated.
- Generated Methods:
Query the VennDB database
mainOnce a database is initialized (e.g., via
TDB::from_iter(...)), you can perform queries using a fluent API. The generated query builder allows you to chain filters based on your struct's fields and then execute the query to retrieve results.Common execution methods include
.any()to find if any record matches the criteria.let db = EmployeeDB::from_iter(/* .. */); let mut query = db.query(); let employee = query .is_admin(true) .is_manager(false) .department(Department::Engineering) .execute() .expect("to have found at least one") .any(); println!("non-manager admin engineer: {:?}", employee);Query multiple variants of a filter map property
mainTo perform an 'OR' query on a single filter map property (matching rows that have any of several specific values), call the query method for that property multiple times on the same query object.
use venndb::{Any, VennDB}; #[derive(Debug, VennDB)] pub struct Value { #[venndb(filter)] pub foo: String, pub bar: u32, } let db = ValueDB::from_iter([ Value { foo: "a".to_owned(), bar: 8 }, Value { foo: "b".to_owned(), bar: 12 }, Value { foo: "c".to_owned(), bar: 16 }, ].into_Iter()).unwrap(); let mut query = db.query(); query.foo("a".to_owned()); query.foo("c".to_owned()); let values: Vec<_> = query.execute().unwrap().iter().collect(); assert_eq!(values.len(), 2); assert_eq!(values[0].bar, 8); assert_eq!(values[1].bar, 16);Define a database schema with `#[derive(VennDB)]`
mainTo use VennDB, define a named field struct and derive
VennDB. You use attribute macros to configure how fields are treated by the database:#[venndb(key)]: Marks a field as the primary key.#[venndb(filter)]: Marks a field as a filterable column.#[venndb(filter, any)]: Marks a field as a filterable column using an 'any' matching strategy.#[venndb(skip)]: Excludes the field from the database index/storage.
When you derive
VennDBon a struct namedT, it generates a corresponding database type namedTDB(e.g.,EmployeebecomesEmployeeDB).use venndb::VennDB #[derive(Debug, VennDB)] pub struct Employee { #[venndb(key)] id: u32, name: String, is_manager: Option<bool>, is_admin: bool, #[venndb(skip)] foo: bool, #[venndb(filter, any)] department: Department, #[venndb(filter)] country: Option<String>, }Initialize a VennDB database
mainYou can create a new database instance using several methods:
new()/default(): Creates an empty database with zero capacity.capacity(usize): Creates an empty database with a pre-allocated capacity.from_rows(Vec<T>): Creates a database from a vector of existing instances. If#[venndb(key)]is used, this returns aResultto handle potential duplicate keys.from_iter(impl IntoIterator<Item = impl Into<T>>): Creates a database from an iterator. This is useful for converting different types (like CSV rows) into your database struct on the fly. This also returns aResultif keys are defined.
If a validator is defined, insertion methods will return an error if the data does not satisfy the validator.
// Creating from an iterator (e.g., converting RawCsvRow to Employee) let db = EmployeeInMemDB::from_iter([ RawCsvRow("1,John Doe,true,false,true,false,Engineering,USA"), RawCsvRow("2,Jane Doe,false,true,true,true,Sales,"), ]) .expect("DB created without errors");Implement custom row validation with `#[venndb(validator = ...)]`
mainTo prevent invalid data from being appended to the database, you can provide a custom validation function using the
#[venndb(validator = function_name)]attribute on your struct. The validation function is called before a row is appended; if it returnsfalse, theappendoperation will return an error.#[derive(Debug, VennDB)] #[venndb(validator = my_validator_fn)] pub struct Value { pub foo: String, pub bar: u32, } fn my_validator_fn(value: &Value) -> bool { !value.foo.is_empty() && value.bar > 0 } let mut db = ValueDB::default(); assert!(db.append(Value { foo: "".to_owned(), bar: 42, }).is_err()); // fails because foo == empty