DOCSEARCHbyAlgolia
Submit Docs

XR Blocks SDK

repository·main·Indexed 19 days ago

https://github.com/google/xrblocks

A lightweight, cross-platform JavaScript library built on three.js for rapid prototyping of AI-integrated XR experiences. It supports hand/head gesture recognition, world understanding, and multimodal AI integration with Gemini. The SDK includes capabilities for 3D object box generation using depth data, VRM lipsync, and the Generative aNthropometric Model (GNM) for human head modeling.

Tokens
129.9K
Snippets
336
Records
548
Agent score
64%

What's inside xrblocks

  1. Overview of the VRM Avatar Demo

    main

    The VRM Avatar demo is a point-to-walk implementation built on XRBlocks. It demonstrates how to integrate VRM (Virtual Reality Model) avatars into an XR environment.

    Key features include:

    • VRM Loading: Uses @pixiv/three-vrm to load .vrm files.
    • Animation Retargeting: Retargets CC0 Mesh2Motion GLB animations onto the VRM humanoid skeleton.
    • Interaction: Users can click (mouse) or pinch (XR) on the floor to command the avatar to walk to that location and return to an idle state.
    • Procedural Animation: Includes procedural eye blinking via the VRM expression manager and crossfades between idle and walk animations.
    • Compatibility: Works in both the XRBlocks desktop simulator and WebXR environments.

  2. What is uiblocks?

    main

    uiblocks is a 3D UI toolkit for XRBlocks that brings 2D web-like UI patterns into spatial computing. It uses three.js and @pmndrs/uikit to provide:

    • Spatial UI: Components that can be anchored to objects, follow the user via Head Leash/Billboard, or be manipulated in 3D space.
    • Flexbox Layout: Automatic positioning using yoga-layout (supporting flex-direction, justify, align, and gap).
    • Rich Styling: Support for gradients, shadows (inner/outer), rounded corners, and borders.
    • Interactions: Built-in raycast click callbacks and visual states for hover and click.

  3. Overview of the agenthands addon

    main

    The agenthands addon provides reusable components for creating embodied agents in XR. It enables an agent to gesture and point at objects in a user's space in sync with spoken text. The embodiment consists of a calm, semi-transparent orb (the head) and translucent hands, driven by inline gesture markup from a language model.

    Key modules include:

    • AgentHands / AgentHand: Animates hands toward poses, plays motions (e.g., beat, wave, size, count), and aims fingers at world points.
    • AgentHead: A semi-transparent orb that breathes while idle, pulses while speaking, and gazes at point targets.
    • AgentGestures: Parses inline gesture markup via parseAgentGestures and converts it into executable steps via buildGestureSteps.
    • AgentGestureAnimator: Drives AgentHands using gesture steps and tracks the active pointing hand.
    • AgentSpeechConductor: Synchronizes the gesture timeline with spoken text using speech synthesizer word boundaries.
    • AgentWorld: Handles object detection grounded to 3D points against the depth mesh, with caching and background re-scanning.

  4. Overview of the Webcam Gestures Demo

    main
    The Webcam Gestures Demo allows you to control XR simulator hand gestures using a webcam and MediaPipe hand tracking. It bridges real-world hand movements to XR Blocks gesture events by mapping webcam landmarks into a HandContext via a local pose estimator and gesture recognizer.

  5. Use the Virtual Keyboard component

    main
    The Keyboard component provides a 3D on-screen QWERTY keyboard for spatial interfaces in XR. It features a 6-row grid layout including symbols, numbers, letters, and special keys (Tab, Backspace, Caps Lock, Enter, Shift, and Space). It handles complex typing logic such as transient shifts (where Shift turns off after one character) and Caps Lock (permanent uppercase).

  6. Use the Simulator Hand Poses Demo for hand pose authoring

    main

    The Simulator Hand Poses Demo is a developer tool for inspecting and authoring simulator hand poses. It allows you to:

    • Inspect hands: View both simulator hands simultaneously.
    • Tune poses: Use per-joint semantic rotation controls to adjust poses.
    • Verify kinematics: Display resolved raw joint transforms produced by forward kinematics (FK).
    • Export data: Copy rotation JSON to use in simulator pose data.
    • Test gestures: Check gesture recognition behavior by manipulating hand shapes.

    This tool is particularly useful when tuning preset hand poses or verifying how specific rotations affect the final joint transforms.

  7. Integrate Generative AI with xb-ai

    main

    The xb.ai (also known as xb.core.ai) module allows you to integrate Gemini or OpenAI into your XR Blocks application. You can perform text queries, multimodal queries (text + images), real-time audio/video sessions via Gemini Live, and image generation.

    Important Security Note: For local prototyping, you can use URL parameters or a keys.json file. However, never embed API keys in production client code. In production, you must proxy AI calls through a secure server you control.

    const options = new xb.Options();
options.enableAI();
xb.init(options);

  8. Netblocks Basic Samples Overview

    main

    The netblocks package provides several basic samples for exploring multiplayer primitives:

    • presence (basic/presence/index.html): Demonstrates joining a room using BroadcastChannelTransport (the default for basic samples, limited to same-origin tabs). It shows peer avatars (heads and hands) in the shared space. Use WebRTCTransport for cross-device testing.
    • objects (basic/objects/index.html): Demonstrates the NetObject class. It features a shared cube where ownership transfers automatically upon interaction (click/drag or pinch), and transform updates are replicated to all peers.
    • events (basic/events/index.html): Demonstrates the canonical pattern for one-shot signals like chat or reactions using session.events to broadcast data between peers.
    • voice (basic/voice/index.html): Demonstrates push-to-hold spatial voice chat, where microphone audio is spatialized around remote avatars.
    • transports (basic/transports/index.html): A utility sample to test connectivity using BroadcastChannel, WebRTC, or WebSocket transports.

  9. Overview of available Sound SDK modules

    main

    The Sound SDK is composed of several specialized modules, all of which are automatically initialized and accessible via xb.core.sound:

    • AudioListener: Handles microphone capture and AI streaming.
    • AudioPlayer: Manages AI audio playback and queuing.
    • BackgroundMusic: Manages background music tracks.
    • SpatialAudio: Provides 3D positional audio capabilities.
    • SpeechRecognizer: Provides speech-to-text functionality.
    • SpeechSynthesizer: Provides text-to-speech functionality.

  10. The Core Pattern for building XR Blocks apps

    main

    XR Blocks is a script-driven engine. Instead of managing your own requestAnimationFrame loop, camera, or WebXR session, you must subclass xb.Script, register it using xb.add(), and then initialize the engine with xb.init(options).

    Per-frame logic should be placed inside the update(time, frame) lifecycle hook. The Core subsystem manages the engine lifecycle and rendering loop.

    import * as THREE from 'three';
import * as xb from 'xrblocks';

class MainScript extends xb.Script {
  init() {
    // Setup logic goes here
    this.add(new THREE.Mesh(...));
  }

  update(time, frame) {
    // Per-frame logic goes here
  }

  onSelectEnd() {
    // Handle interaction events
  }
}

document.addEventListener('DOMContentLoaded', () => {
  xb.add(new MainScript());
  xb.init(new xb.Options());
});

  11. Understand the XR Blocks Engine Model

    main

    XR Blocks operates on a singleton engine model managed by xb.core.

    Key Concepts:

    • xb.core: The singleton engine. Calling xb.init(options) initializes the renderer, camera, XR session, subsystems, and the frame loop.
    • xb.Script: The primary extension point for application logic. An xb.Script is a THREE.Object3D. You add scene content to it using this.add(object).
    • Registration: You must register every application script using xb.add(script) before calling xb.init(options).
    • The Frame Loop: Do not create your own render loop or requestAnimationFrame. Instead, implement per-frame behavior in the update(time, frame) lifecycle hook.
    • State Access: Access engine-created state (like the renderer or camera) in or after the init() phase, never inside a constructor.
    • Measurement: Use meters for all spatial logic. For consistent placement, use xb.user.height, xb.user.objectDistance, and xb.user.panelDistance instead of hardcoded constants.

    Singleton Aliases: You can access high-level engine services via these aliases: xb.scene, xb.user, xb.world, xb.ai, xb.depth, xb.sound, xb.input, and xb.camera.

    class MainScript extends xb.Script {
  static dependencies = {world: xb.World};

  init({world}) {
    this.world = world;
  }
}

  12. Manage multiple pages using Sessions

    main

    A single relay instance can host multiple independent browser pages. To isolate them, assign each page and its corresponding client to the same sessionId. If no sessionId is provided, it defaults to default.

    // In the XR Blocks page
xb.add(
  new RemoteControl({
    url: 'ws://127.0.0.1:8791',
    sessionId: 'run-1',
  })
);

// In the external client
const client = new RemoteControlClient({
  url: 'ws://127.0.0.1:8791',
  sessionId: 'run-1',
});