DOCSEARCHbyAlgolia
Submit Docs

Immersive Web SDK Documentation

repository·main·Indexed 18 days ago

https://github.com/facebook/immersive-web-sdk

A framework for building high-performance VR/AR web experiences using Three.js and an Entity Component System (ECS). The SDK provides production-ready systems for locomotion, physics, spatial audio, and interaction, featuring automatic desktop emulation for development. It includes specialized capabilities for AR depth-based occlusion via the DepthSensingSystem and real-world surface alignment using the EnvironmentRaycastTarget component.

Tokens
164.7K
Snippets
421
Records
713
Agent score
63%

What's inside Immersive Web SDK

  1. Overview of the Audio Example capabilities

    main

    The Audio Example demonstrates 3D positional audio within the Immersive Web SDK (IWSDK). Key features include:

    • Positional Audio: Interactive objects (e.g., robots) with spatialized sound.
    • Playback Modes: Support for restart, overlap, and fade-restart modes.
    • Spatial Properties: Implementation of directional audio cones and distance-based falloff.
    • XR Interface: A UIKitML panel for managing entry and exit from XR sessions.

  2. Overview of @iwsdk/scene-composition

    main

    The @iwsdk/scene-composition package provides IWSDK-native scene document primitives for declarative scene authoring. It defines the browser-independent scene JSON contract used by the native scene editor and agentic scene composition tools.

    Key features include:

    • Schema & Types: Exports SCENE_DOCUMENT_JSON_SCHEMA, SCENE_DOCUMENT_SCHEMA_ID, and TypeScript types for the iwsdk.scene.v1 format.
    • Parsing & Validation: Includes a parser, serializer, and runtime validator for scene documents.
    • Composition: Provides deterministic, resolver-driven scene module composition for Node and browsers.
    • Operations: Supports reversible scene patch operations and command history.
    • Helpers: Includes deterministic lookAt, snap-to-grid, and bounds-aware alignment helpers.
    • Procedural Generation: Provides deterministic, browser-independent RGBA generation and hashes for versioned periodic procedural PBR maps.

    Note: This package is intentionally decoupled from three, Vite, Playwright, or IWSDK runtime packages.

  3. Overview of the Immersive Web SDK

    main

    The Immersive Web SDK (IWSDK) is a collection of frameworks and tools designed to simplify WebXR development. It provides a complete foundation for building immersive web experiences by handling complex tasks like 3D math, physics, spatial audio, and interaction systems.

    Core Architecture

    • Three.js Foundation: Built on top of Three.js for 3D rendering.
    • Entity Component System (ECS): Uses a high-performance ECS to manage application state and scale from prototypes to complex scenes.
    • Cross-Platform Emulation: Applications run in VR/AR headsets but automatically provide mouse-and-keyboard emulation on desktop browsers without requiring special extensions or hardware.

  4. Overview of Immersive Web SDK packages

    main

    The Immersive Web SDK is a collection of tools built on Three.js using a high-performance Entity Component System (ECS). It is designed to provide immersive VR/AR experiences while automatically emulating mouse-and-keyboard input for desktop browsers.

    Core Packages

    PackageDescription
    @iwsdk/coreCore SDK containing the ECS, systems, and WebXR integration
    @iwsdk/createCLI tool for scaffolding new projects
    @iwsdk/locomotorLocomotion engine specifically for Three.js WebXR
    @iwsdk/xr-inputWebXR input system for Three.js
    @iwsdk/vite-plugin-devProvides XR emulation, AI agent tooling, and a development browser

  5. Overview of Immersive Web SDK features

    main

    The Immersive Web SDK is an AI-native WebXR development framework designed to help developers build immersive web experiences. Key features include:

    • Modern ECS Architecture: Built on Three.js using a high-performance Entity Component System (ECS) for scalable 3D applications.
    • First-Class Spatial UI: Author 3D interfaces using HTML-like syntax with UIKitML. This is compiled at build time via a Vite plugin for high-performance layout and text rendering.
    • Production-Ready XR Systems: Includes pre-built, integrated systems for Grab interactions, Locomotion, Spatial Audio, Physics, and Scene Understanding.
    • Agent-First Development: Built-in support for AI agents to see, interact with, and debug XR scenes through screenshots, controller input simulation, scene inspection, and engine-level debugging.
    • Versatile AI Integration: Supports multiple agent modes, including headless background execution, real-time observation, or hands-on collaboration.
    • Powerful AI Tooling: Provides 39 tools for XR emulation, native scene composition, scene graph inspection, and ECS-level debugging.

  6. Understand the IWSDK Native 3D Scene Editor Replacement Plan

    main

    The IWSDK is transitioning from a Meta Spatial Editor integration to a native, IWSDK-powered 3D scene composition editor. The goal is to move away from a 2D canvas projection placeholder to a real 3D viewport that uses an actual IWSDK World rendered via WebGL/Three.js.

    Key architectural shifts include:

    • Source of Truth: The editor will use IWSDK scene JSON and the IWSDK runtime as the primary source of truth, rather than local or external formats.
    • Viewport: A real 3D viewport supporting GLTF assets, meshes, materials, and lights.
    • Unified Tooling: Human UI tools and AI agent tools will share the same scene session and patch path, ensuring agents do not edit JSON files directly but instead use the established tool surface.
    • Deprecation: Meta Spatial Editor integration (including @iwsdk/vite-plugin-metaspatial) is deprecated and being replaced by this native editor.

  7. License and usage of @iwsdk/example-assets

    main

    The assets provided in the @iwsdk/example-assets package (specifically the environment-desk, plant-sansevieria, and robot models) are first-party assets owned by Meta Platforms, Inc. and affiliates. They are distributed under the MIT License.

    No attribution beyond the copyright and permission notice included in the package is required. The package ships with the repository's MIT LICENSE.

  8. Locomotion Overview

    main

    The Immersive Web SDK (IWSDK) provides a modular locomotion stack designed for XR comfort and predictability. It is built on top of a lightweight physics engine (@iwsdk/locomotor) that handles grounded motion, collisions, gravity, jumping, and parabolic ray hits.

    Key components include:

    • Locomotor Engine: Computes physics (grounding, collisions via BVH, gravity) either on the main thread or in a Web Worker for stability.
    • ECS Systems: Map user input to movement via specific systems like Slide, Teleport, and Turn.
    • Environment Component: Allows you to mark specific meshes as walkable (either STATIC or KINEMATIC).
    • Player Collider: A 0.5m radius capsule with a floating spring-damper to handle slopes and steps.

  10. What is a StatefulGamepad and why use it?

    main

    A StatefulGamepad is a wrapper around the standard WebXR Gamepad API. It simplifies XR input handling by providing:

    • Edge-triggered events: Methods like getButtonDown() and getButtonUp() allow you to respond to button presses and releases without manually tracking previous/next state arrays.
    • Component ID mapping: Instead of using hard-coded array indices, you can access buttons and axes using semantic component IDs (e.g., 'xr-standard-trigger') resolved from the active WebXR Input Profile.
    • Axes utilities: Provides 2D magnitude calculations and directional state machines (Up, Down, Left, Right) with configurable thresholds for thumbsticks and touchpads.

  11. What is UIKit (3D UI Runtime)?

    main

    UIKit is a native 3D UI runtime built on Three.js that uses Yoga (Flexbox) for layout. It is designed for XR-grade performance and provides web-like layout semantics in a 3D environment.

    Key features include:

    • Web-aligned layout: Uses Yoga to implement Flexbox properties like flexDirection, gap, and padding.
    • Crisp text: Uses MSDF (Multi-channel Signed Distance Field) text rendering with instancing for high performance.
    • Batching: Groups panels and glyphs to minimize draw calls.
    • Interactivity: Supports pointer events (hover, active, focus) and conditional styling.

  12. Introduction to Spatial UI with UIKitML

    main

    IWSDK uses uikitml (based on pmndrs/uikit and @drawcall/uikitml) to build spatial user interfaces. It allows for declarative, HTML-style markup that is parsed and rendered at runtime in WebXR.

    Key Features:

    • Declarative markup: Describe UI structure using HTML-like syntax.
    • 3D layout system: Uses a Flexbox-like layout system designed for 3D space.
    • Component Kits: Access pre-built components like buttons, panels, and sliders.
    • Event system: Handle interactions like clicks and hovers.
    • Theming: Built-in support for light and dark modes.