corenn

To use CoreNN in Rust, use CoreNN::create to initialize a new database at a specified path. You must provide a Cfg object where the dim field specifies the dimensionality of your vectors. You can then use db.insert to add entries. To reload an existing database, use CoreNN::open. Querying is performed via db.query(query_vector, k), which returns a Vec of (key, distance) pairs.

fn main() {
  let db = CoreNN::create("/path/to/db", Cfg {
    // Specify the dimensionality of your vectors.
    dim: 3,
    // All other config options are optional.
    ..Default::default()
  });
  let key = "my_entry".to_string();
  let vec: Vec<f32> = vec![0.3, 0.6, 0.9];
  db.insert(&key, &vec);
  // For per-vector quantized int8: `db.insert_qi8(&key, &qvec_i8, scale, zero_point)`.

  // Later...
  let db = CoreNN::open("/path/to/db");
  let query: Vec<f32> = vec![1.0, 1.3, 1.7];
  // Returns Vec of (key, distance) pairs.
  let k100 = db.query(&query, 100);
  assert_eq!(k100[0].0.as_str(), "my_entry");
}

InMemoryVectorStore provides methods to save and load a dense matrix of vectors keyed by NodeId order. The on-disk format consists of a bincode header (dtype, dim, max_nodes, node_count) followed by raw row-major scalar bytes in little-endian format.

To persist vectors, you must pass the current number of nodes in the HNSW index to save_to.

use hnswlib_rs::{Hnsw, HnswConfig, InMemoryVectorStore, L2, Result};

fn save_and_load() -> Result<()> {
  let dim = 128;
  let max_nodes = 100_000;

  let hnsw = Hnsw::new(L2::new(), HnswConfig::new(dim, max_nodes));
  let store = InMemoryVectorStore::<f32>::new(dim, max_nodes);
  hnsw.insert(&store, "doc-1".to_string(), &vec![0.0; dim])?;
  let node_count = hnsw.len();

  let mut f = std::fs::File::create("vectors.bin")?;
  store.save_to(&mut f, node_count)?;

  let mut f = std::fs::File::open("vectors.bin")?;
  let (loaded, loaded_count) = InMemoryVectorStore::<f32>::load_from(&mut f)?;
  assert_eq!(loaded_count, node_count);
  Ok()
}

You can save and load the graph structure, key mappings, and configuration using Hnsw::save_to() and Hnsw::load_from().

Important Notes:

• Vectors are NOT included in the graph save. You must persist your VectorStore separately.
• The metric/space is not stored; you must provide the same Metric when calling load_from.
• The graph file includes dim and dtype. load_from validates the dtype against the provided Metric's vector type.

use hnswlib_rs::{Hnsw, HnswConfig, InMemoryVectorStore, L2, Result};

fn save_and_load() -> Result<()> {
  let dim = 128;
  let max_nodes = 100_000;

  let hnsw = Hnsw::new(L2::new(), HnswConfig::new(dim, max_nodes));
  let vectors = InMemoryVectorStore::<f32>::new(dim, max_nodes);
  hnsw.insert(&vectors, "doc-1".to_string(), &vec![0.0; dim])?;

  let mut f = std::fs::File::create("hnsw.bin")?;
  hnsw.save_to(&mut f)?;

  let mut f = std::fs::File::open("hnsw.bin")?;
  let loaded = Hnsw::load_from(L2::new(), &mut f)?;
  assert_eq!(loaded.len(), hnsw.len());
  Ok()
}

CoreNN uses a trait-based system to handle different vector representations (Dense vs Quantized) through VectorFamily and VectorView.

• VectorFamily: Defines the type of vector data being used (e.g., Dense<S> for standard scalars or Qi8 for quantized int8). It specifies the associated Ref type used for viewing the data.
• VectorView: A trait used to obtain a reference (Ref) to the underlying vector data. This allows the library to work with various data containers (like slices &[S]) by providing a unified way to access the vector's contents.
• VectorRef: A trait implemented by the reference types (like &[S] or Qi8Ref) to provide metadata such as the vector's length via .len().

Supported Vector Families

1. Dense<S>: Represents standard dense vectors using a scalar type S (where S: Scalar). The associated reference type is a simple slice &'a [S].
2. Qi8: Represents quantized 8-bit integer vectors. The associated reference type is Qi8Ref<'a>, which contains:
   • data: The &'a [i8] slice.
   • scale: An f32 scaling factor.
   • zero_point: An i8 offset.

CoreNN automatically transitions from an uncompressed state to a compressed state to save memory and disk space as the dataset grows.

Transition Lifecycle

1. Threshold Check: When the number of inserted nodes exceeds cfg.compression_threshold, a background task is spawned to enable compression.
2. Compression Modes:
   • CompressionMode::PQ (Product Quantizer): Trains a model on the existing data and stores it in the database.
   • CompressionMode::Trunc: Uses dimensionality truncation based on cfg.trunc_dims.
3. Hybrid Querying: During the transition, CoreNN can handle distance calculations between uncompressed query vectors and compressed stored vectors by compressing the query on-the-fly.

Caching

To avoid expensive database roundtrips, CoreNN uses a CVCache (Compressed Vector Cache) when in compressed mode, similar to how it uses a NodeCache in uncompressed mode.

You can save and load InMemoryVectorStore (dense) and InMemoryQi8VectorStore (quantized) using the save_to and load_from methods.

Requirements & Constraints:

• Endianness: Persistence requires little-endian architecture. It will return Error::InvalidIndexFormat on big-endian systems.
• Capacity: When saving, you must specify the current node_count. This count cannot exceed max_nodes.
• Loading: load_from returns a tuple containing the reconstructed store and the node_count used during serialization.

Error Cases:

• Error::InvalidIndexFormat: Occurs if the version, dtype, or endianness is incorrect, or if dimensions/counts exceed u32::MAX.

// Saving
let mut file = std::fs::File::create("vectors.bin")?;
store.save_to(&mut file, current_node_count)?;

// Loading
let mut file = std::fs::File::open("vectors.bin")?;
let (loaded_store, loaded_count) = InMemoryVectorStore::<f32>::load_from(&mut file)?;