Vectorlite

repository·main·Indexed 18 days ago

https://github.com/1yefuwang1/vectorlite

A high-performance, runtime-loadable SQLite extension providing fast vector search capabilities using the HNSW algorithm. It supports k-Nearest Neighbor (kNN) searches via virtual tables and is portable across Windows, Linux, and MacOS. The library includes Python (vectorlite_py v0.3.0) and Node.js bindings, offering optimized vector distance implementations and predicate pushdown for metadata filtering on SQLite 3.38+.

Tokens
19.8K
Snippets
64
Records
89
Agent score
62%

What's inside vectorlite

  1. Use Vectorlite Node.js bindings

    main
    Vectorlite provides Node.js bindings for high-performance vector operations. The core native library (vectorlite.so, vectorlite.dll, or vectorlite.dylib) is distributed via platform-specific packages (e.g., @1yefuwang1/vectorlite-linux-x64, @1yefuwang1/vectorlite-darwin-arm64, etc.) to ensure compatibility with the host operating system and architecture.
  2. Implemented Vector Operations

    main

    Vectorlite implements several core mathematical operations optimized using Google's Highway SIMD library. The primary operations available are:

    1. InnerProductDistance
    2. L2DistanceSquared
    3. Normalize (L2 Norm)

    These operations are designed for high-performance vector arithmetic, leveraging SIMD instructions like AVX2's fused Multiply-Add (FMA) to achieve significant speedups over standard implementations.

  3. Key features of vectorlite

    main

    Vectorlite provides several advanced capabilities for vector search in SQLite:

    • Fast ANN Search: Powered by hnswlib for high-performance approximate nearest neighbor search.
    • SIMD Acceleration: Uses Google's highway library for fast vector distance calculations.
    • HNSW Parameter Tuning: Provides full control over HNSW parameters for performance optimization.
    • Predicate Pushdown: Supports filtering by rowid (metadata) during the HNSW graph traversal (requires SQLite >= 3.38).
    • Index Serde: Supports saving and reloading vectorlite tables and loading existing hnswlib index files.
    • JSON Support: Includes vector_from_json() and vector_to_json() for easy data handling.
  4. Determinism strategy for Python testing

    main

    When writing or interpreting tests for vectorlite's Python integration, the suite follows a seeded-strict strategy to ensure mathematical correctness without being brittle to internal HNSW graph changes:

    • Data Generation: All test data is generated using a fixed numpy.random.default_rng(seed).
    • Graph Determinism: Tables use a fixed random_seed= parameter within the hnsw(...) option to ensure consistent graph construction.
    • Ordering/Recall Assertions: To verify KNN results, use a small number of elements N with ef >= N. This forces HNSW to return the exact KNN, allowing for a direct comparison of the returned rowid ordering against a NumPy brute-force reference.
    • Distance Assertions: Distance values are compared against NumPy. Note that hnswlib returns squared L2 distance (it does not apply a square root).
    • Exact-match Retrieval: For l2 or cosine metrics, inserting a vector v at rowid r and querying knn_param(v, 1) should return r with a distance $\approx 0$. Note that exact-match assertions are skipped for the ip (inner product) metric as it is not a true metric.
  5. Benchmark naming convention for ops

    main

    When analyzing or running benchmarks for vector operations, the naming pattern follows: Name/Dimension/Whether to do self-product.

    • Name: The operation being tested (e.g., BM_InnerProduct_Scalar, BM_InnerProduct_Vectorlite).
    • Dimension: The number of elements in the vector.
    • Whether to do self-product: A flag (0 or 1) indicating if the operation is performed on a vector against itself.
    Example: `BM_InnerProduct_Scalar/128/0` benchmarks scalar inner product on 128-dimension vectors without self-product.
  6. Understand the hnswlib lifetime trap

    main

    When working with hnswlib and vectorlite, it is critical to understand that the hnswlib index caches a pointer into its space object (specifically, it returns the address of the dimension variable &dim_ inside the SpaceInterface).

    Constraint: The index and its corresponding NamedVectorSpace must live together. An index must never be paired with a different or rebuilt space.

    Safe Patterns:

    • Moving a NamedVectorSpace or an IndexHandle is safe because the SpaceInterface is held by unique_ptr; the move transfers the pointer, but the pointee's address remains stable.
    • Storing handles as std::unique_ptr<IndexHandle> in a map ensures that each handle's address remains stable across map mutations.
  7. Use hidden command columns for index persistence

    main

    Vectorlite supports hidden command columns that act as a write-only channel for managing index persistence via SQL INSERT statements. These columns allow you to trigger index saving or loading without changing the table's primary data schema.

    Available hidden columns:

    • operation: A TEXT column used to specify the command.
    • path: A TEXT column used to specify the file path for the operation.

    Note: These columns are write-only; attempting to read them will return NULL.

  8. Understand vector quantization and data integrity

    main

    When using quantized vector types in a vectorlite virtual table, reading the data back as float32 may result in precision loss. The following tolerances apply when comparing dequantized values to original float32 values:

    TypeRelative Tolerance (RTOL)
    float320.0 (Exact)
    bfloat161e-2
    float161e-3

    Column Behavior:

    • cosine columns: Vectors are automatically normalized to unit length upon being read back.
    • Non-cosine columns: Vectors are stored and returned verbatim as the specified byte format.
  9. Safety Guards for Index Reuse

    main

    The Connection-Scoped Index Registry employs two specific guards to prevent data corruption or name collisions when tables are dropped and recreated with different configurations:

    1. xCreate Replacement Guard: When a CREATE VIRTUAL TABLE command is issued, the registry always treats this as a fresh start. If a stale entry exists for that table name, it is overwritten with a brand-new, empty index. This prevents a new table from accidentally inheriting vectors from a previously dropped table of the same name.
    2. xConnect Validation Guard: During a reparse, the registry only reuses an existing index if the stored vector_space_str and index_options_str exactly match the current argv (the arguments used to define the table). If you attempt to recreate a table with the same name but a different dimension or index configuration, the guard detects the mismatch and forces a fresh index build, preventing a dimension mismatch (e.g., trying to use a dim-4 index for a dim-128 table).
  10. Important technical notes on distance metrics and data types

    main

    When working with Vectorlite, keep the following technical constraints in mind:

    • L2 Distance: Vectorlite returns the squared L2 distance. This is because the underlying hnswlib implementation does not take the square root.
    • Precision: Reading a quantized vector back as float32 is a lossy operation. Using float32 directly is exact.
    • Data Alignment: Be aware that a 3-byte blob is not a multiple of sizeof(float), which may cause issues when handling raw vector data.
  11. Update VirtualTable to use Registry-Owned Handles

    main

    When refactoring VirtualTable for registry support, the class members must be updated to use references to the IndexHandle owned by the IndexRegistry.

    Important: The order of member declarations in vectorlite/virtual_table.h is critical. The registry pointer, the key, and the handle pointer must be declared before the reference members (space_, index_, allow_replace_deleted_) to ensure valid initialization. The IndexHandle must be stored in the registry via a stable mechanism (like std::unique_ptr in a map) so that the references held by VirtualTable remain valid until the entry is explicitly erased from the registry.

  12. Understand the IndexRegistry and IndexHandle abstractions

    main

    To ensure vector indexes survive SQLite schema reparses (which occur when DDL is executed on a connection), vectorlite uses a connection-scoped registry pattern.

    • IndexHandle: A struct representing the stateful core of a vector table. It bundles the NamedVectorSpace and the hnswlib::HierarchicalNSW index together. This bundling is critical because the hnswlib index maintains a pointer into the space. It also stores the original vector_space_str and index_options_str to detect table-name collisions during re-connection.
    • IndexRegistry: A per-connection map that manages the lifetime of IndexHandle objects. Because IndexRegistry is owned by the connection rather than the short-lived VirtualTable object, the indexes persist even when the virtual table is destroyed and recreated during a reparse.

    Note: IndexRegistry is not thread-safe, as it relies on SQLite's serialization of access to a single connection.

    // Example of the conceptual relationship
    // IndexHandle holds the data; IndexRegistry manages the lifetime.
    struct IndexHandle {
      NamedVectorSpace space;
      std::unique_ptr<hnswlib::HierarchicalNSW<float>> index;
      bool allow_replace_deleted;
      std::string vector_space_str;
      std::string index_options_str;
    };
    
    class IndexRegistry {
     public:
      IndexHandle* Find(const RegistryKey& key);
      IndexHandle* Insert(const RegistryKey& key, IndexHandle handle);
      void Erase(const RegistryKey& key);
    };