VisionProTeleop Documentation

VisionProTeleop is an ecosystem for robotics research using Apple Vision Pro. It consists of three main components:

1. Tracking Streamer (VisionOS app): Streams hand/head tracking data to a Python client, receives video/audio/simulation streams from the client, presents AR simulation scenes (MuJoCo/Isaac Lab), and records egocentric video using UVC cameras.
2. avp_stream (Python library): The client-side library used to receive tracking data and stream video/audio/simulation back to the Vision Pro.
3. Tracking Manager (iOS app): A companion app for managing cloud recordings, configuring VisionOS app settings, calibrating mounted cameras, and sharing datasets.

Primary Workflows:

• Real-World Teleoperation: Control physical robots via hand tracking while viewing robot camera feeds.
• Simulation Teleoperation: Control simulated robots (MuJoCo/Isaac Lab) with hand tracking, viewing either 2D renderings or native AR scenes.
• Egocentric Video Recording: Record first-person manipulation videos with synchronized tracking data.

The examples/ directory contains various implementations demonstrating different features of VisionProTeleop. Use this table to find an example that matches your requirements:

Legend:

• Hardware Camera: Requires a physical camera device (✅) or generates synthetic video (❌).
• Update Method: Uses callback-based or direct frame update method (N/A for MuJoCo-only).
• Audio: Includes audio streaming (✅) or not (❌).
• Stereo Video: Demonstrates stereoscopic 3D video (✅) or standard video (❌).
• MuJoCo: Streams MuJoCo simulation to Vision Pro (✅) or not (❌).

You can inject custom logic into the video stream by registering a callback function using register_frame_callback. The callback function receives a frame as a numpy array (H, W, 3) in RGB format and must return the processed frame.

Key patterns:

• Overlays: Use OpenCV to draw text, bounding boxes, or filters.
• Chaining: You can chain multiple processing steps by having one function call another or by applying multiple transformations sequentially within a single callback.
• Synthetic/Simulation Frames: To stream frames from a simulator (like Isaac Gym) instead of a physical camera, register a callback that returns the rendered frame and call start_streaming with device=None and format=None.

import cv2
import numpy as np

def add_overlay(frame):
    """Add information overlay to frame."""
    # frame is a numpy array (H, W, 3) in RGB format
    cv2.putText(frame, "Robot View", (10, 30),
                cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2)
    return frame

s.register_frame_callback(add_overlay)
s.start_streaming(
    device="/dev/video0",
    format="v4l2",
    size="640x480",
    fps=30
)

The direct method gives you explicit control over the video stream by pushing frames manually from your main loop. This is best when you need the video frame rate to be perfectly synchronized with your data processing or when you need precise, frame-by-frame control.

Pros:

• Explicit control: You decide exactly when frames are sent.
• Synchronized timing: Frame rate matches your loop rate.
• Easy debugging: All logic remains in the main thread.
• Dynamic frame rate: You can change the rate on-the-fly.

Cons:

• Coupled timing: Video frame rate is tied to the main loop speed.
• Main loop overhead: Frame generation happens in the main thread and can slow down other tasks.

streamer.start_streaming(device=None, fps=60)

while True:
    # Get data
    hand_data = streamer.get_latest()
    
    # Process data
    processed_data = process(hand_data)
    
    # Generate visualization synced with data processing
    frame = visualize(processed_data)
    
    # Send frame immediately after processing
    streamer.update_frame(frame)
    
    time.sleep(1/60.)  # Everything runs at 60Hz together

The callback method allows you to decouple video streaming from your main application logic. You register a function that the video streaming system calls automatically at a specified FPS. This is ideal for applications where the control loop (e.g., robot control) needs to run at a different frequency than the video stream (e.g., 100Hz control vs 60Hz video).

Pros:

• Decoupled timing: Video FPS is independent of the main loop.
• Multithreading: Frame generation occurs in a dedicated video thread.
• Consistent frame rate: Not affected by main loop delays.

Cons:

• Harder to debug: Callback runs in a separate thread.
• State management: Requires closures to maintain state.
• Less explicit: Timing is managed by the system, not the user.

# Robot control running at 100Hz
streamer.register_frame_callback(visualizer_callback())
streamer.start_streaming(device=None, fps=60)

while True:
    # Control loop runs at 100Hz
    robot_state = get_robot_state()
    action = compute_action(robot_state)
    robot.execute(action)
    time.sleep(1/100.)  # Control rate: 100Hz
    
    # Video automatically streams at 60Hz in background