DOCSEARCHbyAlgolia
Submit Docs

XR Blocks

website·Indexed 29 days ago

https://xrblocks.github.io/docs/

A framework for creating AI and XR applications, specifically for Android XR. It includes a comprehensive API for UI elements, scene simulation, and interaction, as well as 'netblocks' for multiplayer functionality featuring voice, chat, presence, and lipsync.

Tokens
234.1K
Snippets
1.7K
Records
2.3K
Agent score
72%

What's inside XR Blocks

  1. Overview of the AI class

    The AI class (extending Script) provides an interface to interact with AI models, primarily Gemini. It supports text and multimodal queries, real-time audio/video AI sessions (Gemini Live), and manages session locking to prevent concurrent operations.

  2. Overview of PlaneDetector class

    The PlaneDetector class extends Script and is responsible for detecting and managing real-world planes provided by the WebXR Plane Detection API. It handles the creation, updating, and removal of Plane mesh objects within the scene.

  3. Overview of the XR Blocks framework

    XR Blocks is an open-source, web-based cross-platform framework designed for rapid prototyping of human-centered AI+XR applications. It aims for "minimum code from idea to reality" by providing a modular architecture with plug-and-play components for user representation, environmental perception, gestures, and spatial UI. It is built using WebXR, three.js, TensorFlow, and Gemini.

  4. Overview of xrblocks lipsync functionality

    The lipsync addon parses incoming audio streams (from a microphone or remote peer) and maps frequency formants to real-time mouth shapes on a 3D avatar face. It uses a heuristic viseme mapper that runs locally, requiring no machine learning runtimes or model downloads.

  5. Overview of GazeController

    The GazeController class supplies a camera-aligned gaze ray for XR interactions. Since WebXR Eye Tracking is not yet widely available, this API simulates a reticle at the center of the field of view to enable gaze-based interaction. It extends Script<GazeControllerEventMap> and implements the Controller interface.

  6. Overview of XR Blocks capabilities

    XR Blocks is a cross-platform library built on three.js for rapid prototyping of AI and XR experiences. It targets Chrome v136+ with WebXR support for Android XR devices (e.g., Galaxy XR, XREAL Aura) and includes a desktop simulator. Key features include:

    • Hand Tracking & Gestures: Support for pinch, open-palm, fist, thumbs-up, point, and spread detection via options.enableGestures(), with gesturestart, gestureupdate, and gestureend events.
    • Head Gesture Recognition: Detection of nod and shake motions via options.enableHeadGestures(), accessible through xb.input.headGestures.
    • World Understanding: Depth sensing, geometry-aware physics, planes, scene meshes, object recognition, human pose, face landmarks, and semantic segmentation.
    • Unified Interaction: A single pipeline for mouse, gaze, hand rays, controllers, direct touch, spatial UI, and object manipulation.
    • Spatial UI: World-space cards and view-space overlays using flex-layout panels, text, images, icons, buttons, sliders, and themes.
    • Agent Context: Access to semantic trees, visible-object state, and Set-of-Mark screenshots via xb.context for AI agents.
    • AI Integration: Direct connection to Gemini for multimodal understanding and conversational experiences.

  7. Use FollowObject to sync position and rotation

    The FollowObject class is used to copy the position, rotation, or both from another object to its parent object. It extends TransformScript.
    // Example instantiation
const followObject = new FollowObject(options);

  8. Use HumanRecognizer for human body pose estimation

    The HumanRecognizer class is a detector script that orchestrates human body pose estimation. It manages the lifecycle of backend pose detectors (such as MediaPipe) and provides detected poses, including 3D joint landmarks, mapped into the world coordinate space.

  9. Reference the SetOfMarkContext interface properties

    The SetOfMarkContext interface (defined in src/context/shared/SemanticTypes.ts) provides metadata about a captured simulation snapshot, including the timing, a reference image, the set of marks, and a unique identifier.
    interface SetOfMarkContext {
  snapshotId: string;
  capturedAt: number; // Elapsed simulation time in milliseconds
  image: string;
  marks: SetOfMark[];
}

  10. XRPass class overview

    The XRPass class extends the Pass class from Three.js. It is used to implement post-processing effects specifically for XR environments. It is defined in src/core/components/XREffects.ts.

  11. Use ModelViewer to display 3D models

    The ModelViewer class loads and presents a single interactive glTF or Gaussian Splat model. It extends the Script class and provides specialized properties and methods for model manipulation and loading.

  12. ObjectTouchEvent Interface Reference

    The ObjectTouchEvent interface is used to handle touch-based interaction events. It is extended by ObjectTouchStartEvent and provides detailed information about the touch interaction, such as the source of the interaction, the target object, and the spatial position of the touch.