DOCSEARCHbyAlgolia
Submit Docs

ViroReact Documentation

repository·main·Indexed 23 days ago

https://github.com/reactvision/viro

An open-source library for building Augmented Reality (AR) and Virtual Reality (VR) experiences using React Native and TypeScript. Supports iOS (ARKit), Android (ARCore), and Meta Horizon OS. Features include world mesh configuration, monocular depth for iOS, front-camera AR, and integration with ReactVision Studio for visual scene editing. Includes specialized packages for on-device object detection (@reactvision/react-viro-onnx) and face-tracking (@reactvision/react-viro-face-tracking).

Tokens
17.4K
Snippets
29
Records
72
Agent score
80%

What's inside @reactvision/react-viro

  1. Configure physics for world meshes and JS physics bodies

    main

    There is a distinction between how world meshes and standard JS physics bodies are handled:

    1. World Mesh Physics: Managed directly by VROARWorldMesh in C++. You do not need to add a physicsWorld prop to ViroARScene for this to work.
    2. JS Physics Bodies: If you are using physicsBody components on nodes, you must provide a physicsWorld prop to the ViroARScene (e.g., physicsWorld={{ gravity: [0,-9.81,0] }}).

  2. Coordinate systems in ViroObjectDetector

    main

    The detector provides three ways to locate objects:

    1. boundingBox: Always present. It is normalized [0,1] in the model's (portrait) input space.
    2. screenBoundingBox: Provided in density-independent points (dp), aligned to the on-screen AR preview. You can use these values directly in an absolutely-positioned React Native View (e.g., { left, top, width, height }).
      • iOS: Uses ARKit's displayTransform and inverts the center-square crop.
      • Android: Maps the uncropped camera frame through the viewport crop rectangle and converts to dp.
    3. worldPosition: Provided in metres via an ARKit hit-test of the box center. This is iOS only and requires projectToWorld={true}.

  3. Use ViroCameraTexture for Mirror Effects

    main

    If you want to display a live camera feed on a specific piece of geometry (like a mirror) rather than using the camera as the full-screen AR background, use ViroCameraTexture.

    Comparison: frontCameraEnabled vs ViroCameraTexture

    FeaturefrontCameraEnabledViroCameraTexture
    Camera FeedAR session background (full screen)Material texture on any geometry
    AR TrackingWorld-locked (gravity)Back camera AR still active
    Typical UseSelfie AR, face effectsMirror surfaces, PiP, VR selfie
    PlatformiOS + AndroidiOS + Android
    import {
  ViroMaterials,
  ViroQuad,
  ViroCameraTexture,
} from "@reactvision/react-viro";

ViroMaterials.createMaterials({
  mirrorMat: { lightingModel: "Constant" },
});

function MirrorScene() {
  const cameraRef = useRef(null);

  return (
    <ViroARScene>
      <ViroCameraTexture
        ref={cameraRef}
        material="mirrorMat"
        cameraPosition="front"
        paused={false}
        onCameraReady={() => console.log("Camera ready")}
      />
      <ViroQuad
        width={0.54}
        height={0.96}
        position={[0, 0, -1]}
        materials={["mirrorMat"]}
      />
    </ViroARScene>
  );
}

  4. ViroObjectDetector support on Meta Quest

    main

    On Meta Quest 3 / 3S, the detector automatically switches to using the Meta Passthrough Camera API (Camera2, Horizon OS v74+) when no ViroViewARCore is found in the tree.

    Key details for Quest:

    • Permissions: Requires horizonos.permission.HEADSET_CAMERA and android.permission.CAMERA. These must be requested at runtime before mounting the detector.
    • Capabilities: It emits label and normalized boundingBox only. It does not currently support worldPosition or screenBoundingBox because the passthrough camera FOV differs from the rendered view.
    • Usage: It can run alongside an immersive ViroXRSceneNavigator. You can bridge onDetection data into your VR scene using a module-level store.

  5. Compare ViroCameraTexture and frontCameraEnabled

    main

    Choosing between ViroCameraTexture and the frontCameraEnabled prop on ViroARSceneNavigator depends on your specific use case.

    Use ViroCameraTexture when:

    • You want to show a camera feed on a specific surface (e.g., a mirror, a screen, or a viewport) using a material.
    • You need to keep the back-camera AR world active simultaneously.
    • You want to use the built-in capturePhoto() or startRecording() methods.
    • You want a selfie feed without triggering Apple's TrueDepth API review (Guideline 2.5.1).

    Use frontCameraEnabled when:

    • The entire scene should use the front camera as the background.
    • You want face-tracking behavior (anchoring content to a face or gravity).
    • You are using the @reactvision/react-viro-face-tracking package on iOS.
    • You need to use the TrueDepth API (on iOS).
    FeatureViroCameraTexturefrontCameraEnabled
    Feed destinationMaterial texture on geometryFull-screen AR background
    AR trackingBack camera AR remains activeFront camera, no world tracking
    Selfie mirror effect✅ Automatic mirroring❌ Not a mirror
    Face tracking❌ No✅ Supported (via extra package)
    Photo/Video Capture✅ Supported❌ Not supported

  6. How camera position interacts with AR and non-AR scenes

    main

    The behavior of cameraPosition depends on the type of scene being used:

    In a ViroARScene (AR Mode)

    ViroARScene uses the back camera for tracking and background rendering. ViroCameraTexture opens its own independent camera session.

    • cameraPosition="front": ✅ Works. Two cameras run concurrently (back for AR tracking, front for selfie).
    • cameraPosition="back": ❌ Fails. The OS will deny a second exclusive session on the same sensor already held by ARCore/ARKit.

    In a ViroScene (Non-AR Mode)

    Since no AR framework is running, the camera is not held by tracking services.

    • cameraPosition="front" or "back": ✅ Works. Either camera can be used freely.

  7. Understand Monocular Depth performance and thermal management

    main

    Monocular depth inference runs at a default rate of 5fps. On supported hardware like the A18 Pro, this uses less than 50% of the ANE (Apple Neural Engine) capacity.

    Thermal management is handled automatically by the system. If the device becomes hot, the inference rate will reduce automatically. During this reduction, the system continues to use the depth texture from the last successful frame to maintain occlusion.

  8. Embed ReactVision Studio scenes in your app

    main
    ReactVision Studio is a browser-based visual editor for AR/VR. You can build scenes visually (including placement, AI asset generation, animations, and physics) and then embed them into your React Native app using the StudioSceneNavigator component. This allows designers to update scenes in the Studio without requiring a new native app build.

  9. How ViroObjectDetector modes work

    main

    The mode prop determines how detections are filtered:

    • prompt-free: The default mode. The model detects everything in its baked vocabulary (e.g., yoloe-26n has ~4,585 classes). This is rich but can be noisy.
    • text: A label post-filter. It keeps only detections whose label matches one of the provided categories (whole-word, case-insensitive).
      • Note: This does not prompt the model at runtime. It only filters what the model already emits. For best results, use a model exported specifically for your target classes using react-viro-onnx's export_text_model.py.
    • visual: Reference-image prompting (SAVPE). This mode is currently reserved and not yet implemented.

  10. Technical notes for ViroCameraTexture

    main

    When using ViroCameraTexture, keep the following technical constraints in mind:

    • Material Requirements: The material assigned to ViroCameraTexture only needs a lightingModel. You do not need to provide a texture property; ViroCameraTexture will overwrite the diffuse texture while mounted and clear it on unmount.
    • Resolution:
      • Android: Fixed at 1280 × 720.
      • iOS: Follows the AVFoundation session preset (defaults to AVCaptureSessionPresetHigh).
    • Mirroring: The front camera feed is automatically mirrored by the OS. No manual coordinate transforms are required on your geometry.
    • Conflicts: Multiple ViroCameraTexture components targeting the same material name will conflict. Only the last one to mount will successfully hold the texture.

  11. Configure ReactVision Platform features

    main
    Advanced platform features like Cloud Anchors (persistent, multi-user shared AR) and Geospatial Anchors (pinning content to real-world coordinates) are powered by the ReactVision Platform. To use these, you must provide your rvApiKey and rvProjectId to handle the managed cloud infrastructure.

  12. Navigate between VR scenes

    main

    You can navigate between scenes in two ways:

    1. From inside a VR scene

    ViroVRSceneNavigator passes a sceneNavigator prop to every scene. Use this to push, pop, replace, or jump without needing a ref.

    export function MyVRScene({ sceneNavigator }: any) {
  return (
    <ViroScene>
      <ViroNode onClick={() => sceneNavigator.push({ scene: DetailScene })}>
        {/* … */}
      </ViroNode>
    </ViroScene>
  );
}

    2. From panel-side code (via ref)

    Use a ref on the ViroXRSceneNavigator. Note that the ref accessor is named arSceneNavigator for historical reasons, but it works for VR on Quest as well.

    const navRef = useRef<any>(null);

// Push a new scene
navRef.current?.arSceneNavigator?.push({ scene: DetailScene });

// Pop back
navRef.current?.arSceneNavigator?.pop();

<ViroXRSceneNavigator ref={navRef} vrInitialScene={{ scene: MyVRScene }} />
    // From inside a VR scene
export function MyVRScene({ sceneNavigator }: any) {
  return (
    <ViroScene>
      <ViroNode onClick={() => sceneNavigator.push({ scene: DetailScene })}>
        {/* … */}
      </ViroNode>
    </ViroScene>
  );
}

// From panel-side code (via ref)
const navRef = useRef<any>(null);

// Push a new scene
navRef.current?.arSceneNavigator?.push({ scene: DetailScene });

// Pop back
navRef.current?.arSceneNavigator?.pop();

<ViroXRSceneNavigator ref={navRef} vrInitialScene={{ scene: MyVRScene }} />