DOCSEARCHbyAlgolia
Submit Docs

RuView: WiFi-based Human Pose Estimation and Sensing

repository·main·Indexed 33 days ago

https://github.com/ruvnet/ruview

A cross-platform platform for WiFi-based human pose estimation, vital signs monitoring, and occupancy sensing using CSI data. Includes the WiFi-DensePose mobile interface, AetherArena benchmarking for camera-free spatial intelligence, and WASM edge modules for ESP32 hardware. Features LoRA adapter calibration for room-specific pose accuracy and a Rust-compatible pose engine.

Tokens
721.8K
Snippets
852
Records
2.9K
Agent score
100%

What's inside ruview

  1. Overview of FieldView Mobile App Architecture

    main

    The FieldView mobile application (built with Expo and React Native) is a companion for WiFi-DensePose field deployments. It provides real-time visualization of human pose estimation, vital signs, and occupancy maps.

    Key Connectivity Details:

    • No direct ESP32 connection: The mobile app does not communicate with ESP32 nodes directly. It connects to a Rust sensing server over local WiFi.
    • WebSocket API: Used for real-time streaming of SensingFrame JSON data via ws://host:3001/ws/sensing.
    • REST API: Used for configuration, history, and health checks via http://host:3000/api/v1/....
    • Fallback Mechanism: If the sensing server is unreachable, the app automatically switches to simulation.service to generate synthetic data for testing and demos.

  2. Overview of the homecore crate

    main

    The homecore crate is a Rust port of the Home Assistant core architecture. It serves as the foundational layer for the HOMECORE ecosystem, providing a high-performance, async-first implementation of a state machine, event bus, service registry, and entity registry.

    Key characteristics include:

    • Concurrency: Uses DashMap for lock-free, concurrent state reads and per-entity write serialization.
    • Async-first: Built on Tokio for high-performance event broadcasting and asynchronous operations.
    • Zero-copy: State reads return Arc<State> to avoid expensive cloning.
    • P1 Status: Currently provides in-memory state and registries; persistence (SQLite) and full service dispatch are planned for P2.

  3. Overview of wifi-densepose-train

    main

    The wifi-densepose-train crate provides a complete training pipeline for the WiFi-to-DensePose model. It integrates all five ruvector crates to handle dataset loading, subcarrier interpolation, loss functions, evaluation metrics, and the training loop orchestration.

    Key capabilities include:

    • MM-Fi dataset loader: Loads the NeurIPS 2023 MM-Fi multimodal dataset using memory-mapped .npy files.
    • Synthetic dataset: Provides deterministic, fixed-seed CSI generation for unit tests and reproducible experiments.
    • Subcarrier interpolation: Compresses 114 subcarriers down to 56 using ruvector-solver sparse interpolation with variance-based selection.
    • Training & Evaluation: When the tch-backend feature is enabled, it provides PyTorch-based training loops, loss functions (MSE, OKS), and metrics (PCKh, OKS-AP) with ruvector-mincut-based person matching.

  4. Overview of WiFi-DensePose Rust Crates

    main
    The wifi-densepose workspace is a modular Rust-based platform for WiFi-based human pose estimation, vital sign monitoring, and disaster response using Channel State Information (CSI). It is built on the RuVector graph algorithms and provides high-performance signal processing (up to 810x speedup over Python implementations).

  5. Overview of wifi-densepose-mat features

    main

    wifi-densepose-mat is a specialized crate for WiFi-based disaster survivor detection and localization. Key capabilities include:

    • Vital signs detection: Detects breathing patterns, heartbeat signatures, and movement using an ensemble classifier.
    • Survivor localization: Estimates 3D positions through debris using triangulation, depth estimation, and position fusion.
    • Triage classification: Automatically performs triage compatible with the START protocol, generating priority-based alerts.
    • Event sourcing: Emits domain events such as DetectionEvent, AlertEvent, and ZoneEvent via a pluggable EventStore.
    • ML debris model: Predicts signal attenuation and classifies debris materials to improve detection accuracy.
    • Real-time API: Provides an axum-based REST and WebSocket API for monitoring dashboards.
    • Mathematical integration: Uses ruvector-solver for triangulation and ruvector-temporal-tensor for compressed CSI buffering.

  6. Overview of HOMECORE-UI Operational Dashboard

    main

    HOMECORE-UI is a single-surface operational dashboard for the two-tier Cognitum stack (Appliance $\rightarrow$ SEED $\rightarrow$ ESP32). It replaces manual API calls or SSH access to SEED nodes by providing a unified interface for monitoring system health, entity states, and automation.

    Key architectural features:

    • BFF Pattern: Uses homecore-server as a Backend-for-Frontend (BFF) gateway to aggregate data from multiple tiers (API, Calibration, and SEED nodes).
    • Real-time Updates: Delivered via WebSockets (subscribe_events) rather than polling.
    • No Mock Data in Production: The UI is designed to render typed empty or error states (e.g., 503 upstream_unavailable) if the required upstream service is not available, ensuring operators are not misled by fabricated data.
    • Design System: Built using Cognitum design tokens (CSS custom properties) for visual consistency across the platform.

  7. Overview of WiFi-DensePose API Architecture

    main

    The WiFi-DensePose API provides interfaces for accessing pose estimation data, controlling system operations, and integrating with external platforms. The architecture is multi-protocol, supporting:

    • REST APIs: Standard HTTP methods for resource manipulation.
    • WebSockets: Real-time streaming for live pose data.
    • MQTT & Webhooks: Event-driven notifications for IoT and external services.
    • GraphQL: Flexible queries for complex data requirements.
    • Protocols: Support for HTTP/2, WebSocket, MQTT, and gRPC.
    • Security: OAuth2, JWT, API keys, and rate limiting.

  8. Overview of rUv Neural ecosystem

    main

    rUv Neural is a 12-crate Rust ecosystem designed for brain network topology analysis. It extends RuView's signal processing to transform neural magnetic field measurements (from NV diamond magnetometers or OPMs) into dynamic connectivity graphs. It uses minimum cut algorithms to detect cognitive state transitions in real time.

    Key Components:

    • ruv-neural-signal: Signal processing.
    • ruv-neural-graph: Graph construction.
    • ruv-neural-memory: HNSW-indexed pattern memory.
    • ruv-neural-embed: Graph embeddings.
    • ruv-neural-decoder: Cognitive state decoding.
    • ESP32/WASM edge targets for deployment.

  9. Overview of HOMECORE architecture

    main

    HOMECORE is a native Rust, WASM, and TypeScript implementation of the Home Assistant (HA) hub contract, designed to integrate natively with the RuView sensing platform. Unlike the Python-based Home Assistant, HOMECORE is built for real-time RF sensing, AI-augmented automation, and edge-native deployment on constrained hardware.

    Key architectural components include:

    • Core State Machine & Event Bus: A Tokio-based async Rust core (ADR-127).
    • WASM Plugin System: An integration system where plugins run in sandboxed WASM environments (ADR-128).
    • Automation Engine: A Rust-native engine with a WASM expression evaluator for templates (ADR-129).
    • Wire-Compatible API: An Axum-based REST and WebSocket API that is wire-compatible with Home Assistant, allowing existing HA companion apps and clients to work without modification (ADR-130).
    • Semantic Recorder: A storage layer using SQLite combined with ruvector for semantic search capabilities (ADR-132).
    • ruflo Agent Orchestration: A voice/text assistance layer using ruflo and ruvector intent matching (ADR-133).

  10. Overview of Exotic & Research Modules

    main

    The Exotic & Research Modules are experimental sensing applications for WiFi-DensePose that explore novel uses of RF sensing. These modules are designed for edge deployment with the following constraints:

    • no_std Compliance: No heap allocation; designed to run on the WASM3 interpreter on ESP32-S3.
    • Stack-Allocated State: All modules use const fn new() for stack-allocated, const-constructible state.
    • Static Event Buffers: Events are returned via &[(i32, f32)] from a static array (typically 3-5 events per frame).
    • Budget-Aware: Each module declares a per-frame time budget (Low/Medium/High).
    • Frame Rate: Assumes a 20 Hz CSI frame rate from the host Tier 2 DSP.

    Maturity Levels

    • Proven: Based on published research with validated results.
    • Experimental: Working implementation, needs real-world validation.
    • Research: Proof of concept, exploratory.

  11. Overview of the nvsim crate

    main
    The nvsim crate is a standalone, deterministic NV-diamond magnetometer pipeline simulator. It is designed to model the full sensing pipeline: magnetic source synthesis $\rightarrow$ propagation $\rightarrow$ NV-ensemble response $\rightarrow$ ADC quantization/demodulation. It is intended for use in magnetic-anomaly modeling, NV-physics teaching, and COTS-sensor noise-floor sanity checks, and can be used independently of the WiFi-CSI context.

  12. Overview of Security & Safety Modules

    main

    The Security & Safety modules provide perimeter monitoring and threat detection using WiFi Channel State Information (CSI). These modules work through walls and in darkness without cameras. Each module is designed to run on an $8 ESP32-S3 chip at a 20 Hz frame rate, is no_std-compatible, and compiles to WASM for hot-loading via ADR-040 Tier 3.

    Available Modules

    ModuleFileDescription
    Intrusion Detectionintrusion.rsPhase/amplitude anomaly intrusion alarm with arm/disarm
    Perimeter Breachsec_perimeter_breach.rsMulti-zone perimeter crossing with approach/departure
    Weapon Detectionsec_weapon_detect.rsConcealed metallic object detection via RF reflectivity ratio
    Tailgating Detectionsec_tailgating.rsDouble-peak motion envelope for unauthorized following
    Loitering Detectionsec_loitering.rsProlonged stationary presence with 4-state machine
    Panic Motionsec_panic_motion.rsErratic motion, struggle, and fleeing patterns