DOCSEARCHbyAlgolia
Submit Docs

Meta Wearables Device Access Toolkit (DAT) for Android

repository·main·Indexed 43 days ago

https://github.com/facebook/meta-wearables-dat-android

A developer preview SDK enabling Android applications to connect to Meta AI glasses. It provides capabilities for video streaming, photo capture, and display integration, including support for version 0.9.0. The toolkit includes tools for managing session lifecycles, configuring camera stream resolution and frame rates, and integrating with AI-assisted development tools.

Tokens
16.4K
Snippets
44
Records
68
Agent score
57%

What's inside Meta Wearables DAT

  1. Overview of the Camera Access App Sample

    main

    The Camera Access App is a sample Android application designed to demonstrate how to integrate the Meta Wearables Device Access Toolkit. It implements the SDK's camera lifecycle through explicit steps: starting a session, starting the preview, capturing or recording media, stopping the preview, and ending the session.

    Key capabilities demonstrated include:

    • Connecting to Meta AI glasses.
    • Streaming live camera feeds.
    • Capturing photos and recording video (with optional sound-in-video).
    • Using a foreground service to maintain recording while the app is in the background.
    • Managing firmware update flows.

  2. Understand Camera Stream State

    main

    Camera streaming has its own independent state flow that occurs after a stream is attached to a capability. The state transitions follow this sequence:

    STOPPED $\rightarrow$ STARTING $\rightarrow$ STARTED $\rightarrow$ STREAMING $\rightarrow$ STOPPING $\rightarrow$ STOPPED $\rightarrow$ CLOSED

    lifecycleScope.launch {
    camera.stream.state.collect { state ->
        // React to camera capability state changes
    }
}

  3. Handle errors using DatResult

    main

    The SDK uses DatResult<T, E> for typed success and failure handling. Instead of using getOrThrow(), which can crash your app, use functional operators like .onSuccess { ... } and .onFailure { error, _ -> ... } to surface errors to the user. This is particularly important for permission checks and session/capability operations.

    Wearables.checkPermissionStatus(Permission.CAMERA)
    .onSuccess { status -> /* handle status */ }
    .onFailure { error, _ -> /* handle error */ }

  4. Handle Session Pauses and Resumes

    main

    A session may enter a PAUSED state if another experience takes over the device, the user removes/folds the glasses, Bluetooth connectivity drops, or access is revoked.

    When paused:

    • The device connection may remain active.
    • Attached capabilities will stop performing useful work.
    • Do not try to force a restart. Your app should wait for the next observed session state change.

  5. Map DAT evidence to specific lifecycle stages

    main

    When debugging, correlate observed behaviors with these specific DAT components and states:

    Lifecycle StageRelevant DAT Evidence / API Tokens
    InitializationApp logs around Wearables.initialize(context)
    RegistrationWearables.registrationState and wearables.registrationState
    Device AvailabilityWearables.devices, wearables.devices debug events, device.compatibility, and device.properties
    Device Linkdevice.linkState
    PermissionsWearables.checkPermissionStatus(...), device.permission, check_permissions, and get_permissions
    SessioncreateSession(...), session.start(), deviceSession.state, session.state, and session errors
    Streamsession.addCamera(...), camera.stream.start(), camera.stream.state, frame counters, and stream errors
    Transportsdk.usesDam and DAM/DWA error values

  7. Register your app with Meta AI

    main

    The DAT SDK separates app usage into two distinct phases: Registration, where the user connects your app to Meta AI, and Device permissions, where your app requests specific capabilities (like camera access) after registration is complete. Both flows require the Meta AI app to be installed on the device.

    To initiate the registration process, use Wearables.startRegistration(activity). You should observe the registration state via the Wearables.registrationState flow to update your UI accordingly. To remove the connection, use Wearables.startUnregistration(activity).

    // Start registration
Wearables.startRegistration(activity)

// Observe registration state
lifecycleScope.launch {
    Wearables.registrationState.collect { state ->
        // Update your registration UI
    }
}

// Unregister
Wearables.startUnregistration(activity)

  8. Best practices and rules for using Display Access

    main

    When implementing Display Access, follow these operational rules to ensure stability and correct device interaction:

    Lifecycle and Initialization

    • Registration: Initialize DAT and complete registration before creating a session.
    • Session State: Wait for DeviceSessionState.STARTED before calling session.addDisplay(). Wait for DisplayState.STARTED before enabling or sending user-triggered content.
    • Cleanup: When the display experience ends, cancel state/error collection jobs, close or replace active VideoPlayer instances, detach with session.removeDisplay(), and then stop the session.

    Error Handling and Observation

    • Async Failures: DeviceSession.start() returns Unit. You must observe session.errors to catch asynchronous failures.
    • Streams to Monitor: Observe Wearables.registrationErrorStream, session.errors, Display.sendContent failures, and VideoPlayer.error.

    Device Selection

    • Use SpecificDeviceSelector for a user-picked device.
    • Use AutoDeviceSelector(filter = { it.isDisplayCapable() }) only when automatic selection is acceptable.

    Firmware and App Updates

    • If device metadata reports DeviceCompatibility.DEVICE_UPDATE_REQUIRED, call Wearables.openFirmwareUpdate(activity).
    • If session creation or session.errors reports DAT_APP_ON_THE_GLASSES_UPDATE_REQUIRED, call Wearables.openDATGlassesAppUpdate(activity).

    Performance

    • Keep display callbacks fast; move app state changes back onto your UI layer as needed.

  9. Understand the Display Access App architecture

    main

    The sample app is organized into several key components:

    • MainActivity.kt: Serves as the app entry point and handles runtime permission requests.
    • WearablesViewModel.kt: Manages the state for device registration and device observation.
    • DisplayViewModel.kt: Manages the session lifecycle, attaches display capabilities, and handles display content.
    • AppScaffold.kt: Handles navigation between settings and samples, including the automatic handoff to the samples list once a session starts.

  10. How camera streaming works in Meta Wearables DAT

    main

    Camera streaming is built around four primary abstractions:

    • Session: Represents the device connection lifecycle. It is initialized using Wearables.createSession(...).
    • Stream: The actual camera stream accessed via camera.stream after a camera is attached to a session.
    • StreamConfiguration: Defines the resolution and frame rate for the stream.
    • PhotoData: The data object containing a still image captured from the glasses during an active stream.

    To use the camera, you must first create a session, add a camera with a specific configuration, and then start the stream.

    val session = Wearables.createSession(AutoDeviceSelector()).getOrElse { error ->
    throw IllegalStateException(error.description)
}
session.start()

val camera: Camera = session.addCamera(
    StreamConfiguration(
        videoQuality = VideoQuality.MEDIUM,
        frameRate = 24,
    ),
).getOrElse { error ->
    throw IllegalStateException(error.description)
}

camera.stream.start().getOrElse { error ->
    throw IllegalStateException(error.description)
}

  11. Execute the Live DAT Debugging agent loop

    main

    When using the live debugging skill, follow this sequence to diagnose issues:

    1. Discovery: Connect to the running app debug server using discover_debug_servers and connect_to_debug_server.
    2. Baseline Establishment: Call get_connection_status, get_sdk_state, and get_dat_readiness in order. Note: Do not skip get_connection_status, as it distinguishes MCP/debug-server connectivity from the actual SDK, registration, and session state.
    3. Boundary Inspection: If the issue involves Meta AI app or device boundary behavior, inspect get_companion_boundary_diagnosis, get_device_path, get_permissions, and get_device_properties.
    4. Reproduction: Ask the developer to reproduce the failing Android flow while connected.
    5. Evidence Collection: Instead of polling, wait for specific evidence using wait_for_events (filtered by category or source), get_errors, or get_event_digest.
    6. Handoff: Collect a redacted bundle using export_diagnostic_bundle.

  12. Set up the Display Access App sample

    main

    To run the Display Access App sample, you must provide your application credentials. You can do this by adding the following keys to your local.properties file or by exporting them as environment variables.

    Prerequisites

    • Android Studio Narwhal (2025.1.1) or newer
    • JDK 17 or newer
    • Android SDK 36 or newer
    • Meta Wearables Device Access Toolkit (included as a dependency)
    • Display-capable Meta AI glasses for end-to-end testing
    mwdat_application_id=YOUR_APPLICATION_ID
mwdat_client_token=YOUR_CLIENT_TOKEN