HyperspaceDB

repository·main·Indexed 40 days ago

https://github.com/yarlabs/hyperspace-db

A high-performance Spatial AI Engine for autonomous agents and robotics featuring native support for hyperbolic geometry, hierarchical knowledge graphs, and edge-to-cloud synchronization. It provides a Rust SDK (v3.1.0) for managing collections, performing vector, lexical (BM25), and hybrid searches, and integrating embedding backends including Local ONNX, HuggingFace Hub, and cloud APIs.

Tokens
163.3K
Snippets
527
Records
669
Agent score
50%

What's inside HyperspaceDB

  1. What is HyperspaceDB?

    main

    HyperspaceDB is a Spatial AI Engine designed for Autonomous Agents, Robotics, and Continuous Learning. Unlike traditional vector databases optimized for static RAG (Retrieval-Augmented Generation), HyperspaceDB models information as hierarchical, spatial, and dynamic graphs.

    Key technical differentiators include:

    • Hyperbolic Geometry: Uses Poincaré and Lorentz models to compress massive hierarchical structures (like codebases or taxonomies) into 64-dimensional spaces, reducing RAM usage by up to 50x.
    • Edge-to-Cloud Architecture: Supports offline-first operation for edge hardware (drones, robots) using a Merkle Tree Delta Sync protocol (SyncHandshake, SyncPull, SyncPush) to sync episodic memory with the cloud.
    • Continuous Learning: Provides primitives for agents to organize memories via a Fast Upsert Path, CDC Event Streams, and a Riemannian Math SDK (supporting Fréchet mean and parallel transport).
  2. HyperspaceDB ROS2 Integration Overview

    main
    The HyperspaceDB ROS2 integration provides a bridge between HyperspaceDB and the ROS2 environment. It enables robots to utilize hyperbolic vector memory and advanced cognitive math through a set of specialized ROS2 services. This integration is designed to allow robotic systems to interact with the HyperspaceDB engine for memory, search, and hierarchical navigation tasks.
  3. Understand the HyperspaceDB System Architecture

    main

    HyperspaceDB follows a Command-Query Separation (CQS) pattern to optimize for high-throughput ingestion and low-latency search. The architecture is divided into several layers:

    1. Client Layer: Supports multiple SDKs (Python, TypeScript, Go, C++, Dart) and ROS2 nodes.
    2. API Layer: Provides gRPC (via Tonic) and REST (via Axum) interfaces.
    3. Real-Time Ingestion Tier: Uses a WriteBuffer (DashMap RAM) and a segmented Write-Ahead Log (WAL) for immediate data availability, alongside an L0 Hot Tier Cache for ultra-fast lookups.
    4. Core Engine: An LSM-Tree based engine using HNSW for indexing, a Meta-Router for IVF centroids, and a Cascade Pipeline for Matryoshka Representation Learning (MRL) truncation.
    5. Storage Layer: Manages MemTables, immutable HNSW Chunks (.hyp files), and S3 cloud tiering.
    6. Replication: Uses a 256-bucket Merkle Tree and a Gossip Swarm (UDP) for anti-entropy and synchronization.
    graph TB
        subgraph "Client Layer"
            PY[Python SDK]
            TS[TypeScript SDK]
            GO[Go SDK]
            CPP[C++ SDK]
            DART[Dart SDK]
            ROS2[ROS2 Node]
        end
        
        subgraph "API Layer"
            GRPC[gRPC Server<br/>Tonic]
            REST[REST API Server<br/>Axum]
        end
        
        subgraph "Real-Time Ingestion Tier"
            WB[WriteBuffer<br/>DashMap RAM]
            WAL[Write-Ahead Log<br/>Segmented]
            CACHE[L0 Hot Tier Cache<br/>L1 DashMap / L2 HNSW]
        end
        
        subgraph "Core Engine (LSM-Tree + Cascade)"
            MEM[MemTable<br/>Active HNSW]
            CHUNK[Immutable Chunks<br/>SSTables]
            ROUTER[Meta-Router<br/>IVF Centroids]
            CASCADE[Cascade Pipeline<br/>MRL Truncation]
            BACKEND[Chunk Backend<br/>Local/S3]
            QUANT[Quantization<br/>Anisotropic SQ8/Binary]
        end
        
        subgraph "Storage Layer"
            FLUSH[Flush Worker]
            SNAP[Snapshots<br/>rkyv mmap]
            S3[(S3 / Cloud Storage)]
        end
        
        subgraph "Replication & Anti-Entropy"
            MERKLE[256-Bucket Merkle Tree]
            GOSSIP[Gossip Swarm<br/>UDP socket]
        end
    
        subgraph "Visualization (Phase 4)"
            DASH[React Dashboard]
            DASH -->|/api/graph/explore| REST
            DASH -->|Canvas| POINCARE[Poincaré Disk]
            DASH -->|Canvas| MOMENTUM[Momentum HUD]
        end
        
        PY --> GRPC
        TS --> GRPC
        GO --> GRPC
        CPP --> GRPC
        DART --> GRPC
        ROS2 --> GRPC
        
        GRPC --> WB
        GRPC --> WAL
        REST --> CACHE
        
        WB --> MEM
        WAL -.-> FLUSH
        FLUSH --> CHUNK
        CHUNK --> BACKEND
        BACKEND --> S3
        
        style S3 fill:#f9f,stroke:#333,stroke-width:2px
        style ROUTER fill:#9f9,stroke:#333,stroke-width:2px
        style MEM fill:#99f,stroke:#333,stroke-width:2px
  4. Implement Zero-Knowledge Client-Side Encryption (ZK-Privacy)

    main

    The ROS2 node links against the C++ SDK, providing native support for Zero-Knowledge client-side encryption. For high-security applications (e.g., protecting sensor readings, maps, or private logs), developers can extend their custom ROS2 nodes by accessing the underlying HyperspaceClient instance.

    To ensure data is encrypted before it reaches the cloud or a DePIN server, use the following methods on the HyperspaceClient:

    • RegisterCollectionKey: To set up encryption keys for a collection.
    • InsertSecure: To perform encrypted insertions.
  5. Core Pillars of HyperspaceDB

    main

    HyperspaceDB is built on several core architectural pillars designed for high-performance spatial AI workloads:

    • Reflex-Level Speed: Built on Nightly Rust with an ArcSwap lock-free architecture and f32 SIMD intrinsics, capable of up to 12,000 Search QPS and 60,000 Ingest QPS per node.
    • RBAC & Multi-Tenancy: Granular Role-Based Access Control (Admin, ReadWrite, ReadOnly) using redb storage. Supports namespaced URL collection referencing (e.g., owner:name or owner/name) for secure cross-tenant sharing.
    • L0 Hot Tier Caching: A transparent two-level cache combining a 64-shard L1 DashMap Cache (~1 µs) and an L2 HNSW Fallback Graph (~100 µs).
    • Real-Time WriteBuffer: Ensures zero-latency searchability by allowing queries to perform Rayon-parallelized linear scans on newly inserted vectors before they are fully integrated into the HNSW graph.
    • Schema-Driven Cascade: Native MRL (Matryoshka) support. Uses a CascadePipeline to query truncated dimensions in RAM and automatically rerank results using full-resolution vectors from NVMe/S3.
    • Cognitive Math Engine: Supports Euclidean (L2/Cosine), Poincaré Ball, and Lorentz Hyperboloid metrics. Includes advanced geometric filters like InBall, InBox, and InCone.
    • S3 Cloud Tiering: Seamlessly offloads cold segments to S3/MinIO. Enable via Cargo feature s3-tiering and setting HS_STORAGE_BACKEND=s3.
    • Zero-Knowledge ZK-Privacy: Supports client-side encryption where vectors are projected, noise-injected, and payloads are AES-GCM encrypted before leaving the client.
  6. Configure Leader-Follower Replication (Cloud/HA)

    main

    For high-availability cloud deployments, use the Leader-Follower architecture.

    • Leader: Acts as the Coordinator. It handles all Writes, manages cluster topology, and streams Write-Ahead Log (WAL) events.
    • Follower: Acts as a read-only replica. Followers can be promoted to Leaders if the current leader fails.

    Each node is identified by a unique node_id and uses a Lamport logical clock for ordering.

    # Start a Leader node
    ./hyperspace-server --port 50051 --role leader
    
    # Start a Follower node pointing to the leader
    ./hyperspace-server --port 50052 --role follower --leader http://127.0.0.1:50051
  7. How Agent Memory and Hyperbolic Hierarchy work in HyperspaceDB

    main

    HyperspaceDB uses a hybrid $H^{33} imes ext{R}^{768}$ vector geometry. For agents managing long-term memory or complex reasoning, the following tools enable interaction with hierarchical cognitive structures:

    • Hierarchical Navigation: Use hyperspace_get_concept_parents to find hypernyms in Lorentz space ($H^{33}$) or hyperspace_get_subsumption_tree to trace full taxonomic trees.
    • Reasoning Stability: Use hyperspace_analyze_thought_stability to calculate Lyapunov convergence of a Chain of Thought (CoT) trajectory. This helps detect if a model is hallucinating or converging on a stable logical attractor.
    • Semantic Retrieval: hyperspace_search_text performs natural language lookups using a 96D MRL RAM HNSW scan combined with an 801D Disk payload rerank.
  8. How Matryoshka Representation Learning (MRL) works

    main

    HyperspaceDB supports MRL via a Cascade Pipeline. This allows for a two-stage retrieval process: performing a fast initial search on a truncated low-dimensional vector (e.g., 64D) and then reranking the top results using the full-dimensional vector (e.g., 1024D).

    To implement this, define a cascadePipeline in your CollectionSchema containing MrlLayer objects. You specify the cutoffDimension for the fast search and rerankTopK for the second stage.

    final schema = pb.CollectionSchema(
      components: [
        pb.VectorComponent(name: 'main', metric: 'lorentz', fullDimension: 1025)
      ],
      cascadePipeline: [
        pb.MrlLayer(
          componentName: 'main',
          cutoffDimension: 129, // 128D (+1) fast search
          storeInRam: true,
          rerankTopK: 100
        )
      ]
    );
    await client.createCollection('mrl_docs', schema);
  9. How Cognitive Tools Work in MCP

    main

    Cognitive tools in the HyperspaceDB MCP server are implemented as client-side computations. This allows them to work against any version of the HyperspaceDB server.

    The Workflow:

    1. The tool calls getPoints() to fetch the stored vectors for the specific trajectory IDs.
    2. Mathematical analysis (such as Lyapunov, Koopman, or trust scores) is performed locally within the MCP server process.
    3. The results are returned to the AI host without requiring additional server-side RPC for the mathematical computations.

    Geometry Awareness: When a collection uses the hybrid metric, momentum extrapolation handles the vector by splitting it into Lorentz (the first 33 dimensions) and Euclidean (the remaining dimensions) parts. It extrapolates each part independently and then recombines them.

  10. Configure Matryoshka Representation Learning (MRL) Cascades

    main

    MRL allows for high-performance cascading searches. You define an MrlLayer in the CascadePipeline of your CollectionSchema. This specifies a CutoffDimension for the initial fast search on a truncated vector, and a RerankTopK value for re-ranking results using the full-dimensional vector.

    schema := &pb.CollectionSchema{
        Components: []*pb.VectorComponent{
            {Name: "primary", Metric: "lorentz", FullDimension: 1025, Weight: 1.0},
        },
        CascadePipeline: []*pb.MrlLayer{
            {
                ComponentName:   "primary",
                CutoffDimension: 129, // Initial search on 128D (+1)
                StoreInRam:      true,
                RerankTopK:      100,
            },
        },
    }
    client.CreateCollection(ctx, "mrl_collection", schema)
  11. How Thought Trajectories work in HyperspaceDB

    main

    A thought trajectory is a sequence of vector IDs representing the progression of an agent's reasoning (e.g., each step of a Chain of Thought stored as a vector).

    By storing Chain of Thought (CoT) steps in HyperspaceDB, you can apply mathematical analysis to detect hallucinations, measure convergence, and predict future reasoning direction. These tools are implemented as client-side computations in the SDK: they fetch stored vectors via getPoints() and then apply mathematical analysis locally, making them compatible with any version of the HyperspaceDB server.

    Supported geometries for cognitive tools:

    • lorentz, poincare — full Lorentz/hyperbolic math
    • hybrid — applies Lorentz math to first 33 dims, Euclidean to the rest
    • cosine, l2 — Euclidean approximations
    [id_1: "observe problem"] → [id_2: "form hypothesis"] → [id_3: "test hypothesis"] → ...
  12. Use the Implicit Graph Engine

    main

    HyperspaceDB treats vectors as nodes in a dynamic graph where relationships are inferred from geometry (e.g., Lorentz/Poincare for hierarchy, L2/Cosine for similarity).

    • getSubsumptionTree(id1, id2): Extracts directed hierarchies from Lorentz-encoded data.
    • traverse({ startId, traversalMode, breadthLimit }): Navigates the graph. traversalMode can be 0: GREEDY, 1: DIFFUSIVE, or 2: MOMENTUM.
    • getNeighbors(id): (Implicitly available via graph APIs) Retrieves adjacent nodes.
    // Extract directed hierarchies
    const tree = await client.getSubsumptionTree(1, 5);
    
    // Navigate the graph
    const results = await client.traverse({
        startId: 1,
        traversalMode: 2, // MOMENTUM
        breadthLimit: 5
    });