DOCSEARCHbyAlgolia
Submit Docs

MixedReality-WebRTC

repository·master·Indexed 21 days ago

https://github.com/microsoft/mixedreality-webrtc

Libraries for integrating peer-to-peer real-time audio, video, and data communication into mixed reality applications. It provides a native C API (mrwebrtc), a .NET Standard 2.0 C# wrapper (Microsoft.MixedReality.WebRTC), and a Unity package (com.microsoft.mixedreality.webrtc). It specifically targets Windows (UWP/Desktop), HoloLens 1, HoloLens 2, and Unity on Android, offering features like scene video streaming and Mixed Reality Capture (MRC) integration.

Tokens
31.5K
Snippets
42
Records
151
Agent score
75%

What's inside MixedReality-WebRTC

  1. Overview of MixedReality-WebRTC documentation structure

    master

    The documentation is divided into two primary integration paths:

    C# Library

    For developers working in native C# environments (e.g., Desktop or UWP).

    • Core Concepts: Feature Overview, Peer Connection, and Signaling.
    • Tutorials: Available for both Desktop and UWP.

    Unity Integration

    For developers using the Unity engine.

    • Core Concepts: Feature Overview, Peer Connection, and Signaler.
    • Audio Components: MicrophoneSource and AudioReceiver.
    • Video Components: WebcamSource, VideoReceiver, and VideoRenderer.
    • Tutorials: Getting started with Unity.

  2. Overview of MixedReality-WebRTC

    master

    MixedReality-WebRTC is a collection of libraries designed to help mixed reality developers integrate peer-to-peer real-time audio, video, and data communication into their applications.

    Key features include:

    • Multi-track communication: Supports real-time audio, video, and data between peers.
    • Abstracted signaling: Provides an interface to easily switch signaling implementations.
    • Multi-platform APIs: Exposes C++ and C# APIs for integration into existing applications.
    • Unity3D Integration: Provides Unity components for rapid prototyping.
    • HoloLens Support: Supports Microsoft HoloLens (x86) and HoloLens 2 (ARM).
    • Mixed Reality Capture (MRC): Allows streaming the user's viewpoint for multi-device experiences.

    ⚠️ DEPRECATION NOTICE: MR-WebRTC has been deprecated. Microsoft is no longer committing development resources, taking pull requests, or planning a replacement. If you need to continue development, you must fork the repository and maintain your own branch.

  3. Overview of MixedReality-WebRTC for Unity features

    master

    The com.microsoft.mixedreality.webrtc package enables peer-to-peer real-time audio, video, and data communication in Unity. Key features include:

    • Multi-track communication: Support for multiple simultaneous audio/video/data tracks with remote peers.
    • Drop-in Components:
      • Peer connection and automated media track management.
      • Local audio/video capture (webcam and microphone).
      • Remote audio output via AudioSource (supports 2D and spatial audio).
      • Remote video rendering via Unity textures on any mesh.
      • Scene video streaming: Stream content from any Unity Camera component (stream-what-you-see).
    • Mixed Reality Capture (MRC): Easy integration to stream the user's viewpoint for multi-device experiences.
    • Platform Support: Supports in-editor Play mode, UWP devices, Microsoft HoloLens (x86), and HoloLens 2 (ARM).

  4. Overview of MixedReality-WebRTC Unity Samples

    master

    The samples package provides two primary demonstration scenes to help developers understand the WebRTC implementation in Unity:

    StandaloneDemo

    • Purpose: A self-contained introduction to WebRTC concepts and PeerConnection components.
    • Mechanism: Uses the multi-track API to connect two peers within the same Unity scene.
    • Signaling: Uses a simple, local-only, in-process signaler. This avoids the need for external networking setup and is ideal for learning the core components without networking complexity.

    VideoChatDemo

    • Purpose: Demonstrates a concrete audio and video chat client application.
    • Mechanism: Allows two clients on different machines to communicate.
    • Signaling: Requires external setup using the NodeDssSignaler. This is a testing/debugging solution based on the NodeJS-based node-dss server.
    • Warning: NodeDssSignaler is not production-ready and provides no security guarantees (uses plain text over HTTP).

  5. MixedReality-WebRTC Documentation Overview

    master

    The MixedReality-WebRTC documentation is organized into two primary development paths: the C# Library and the Unity Library.

    • C# Library: Focuses on core WebRTC primitives like PeerConnection and Signaling mechanisms.
    • Unity Library: Provides high-level integration components for Unity developers, including specialized classes for peer connections, signaling, video rendering (VideoRenderer), audio handling (MicrophoneSource, AudioReceiver), and video sources (WebcamSource, VideoReceiver).

  7. Overview of MixedReality-WebRTC Unity components

    master

    The MixedReality-WebRTC Unity package provides idiomatic C# wrappers over the core library to simplify WebRTC integration in Unity. The architecture relies on a few core components working together:

    • PeerConnection: The central component that encapsulates a single peer-to-peer connection.
    • Signaler: An abstract base class used to manage signaling messages required to establish a connection. A concrete implementation must be assigned to the PeerConnection.
    • VideoSource: Acts as the bridge for video tracks, handling both local video sending and remote video receiving.
    • VideoTrackPlayer: A rendering component that bridges a VideoSource to a Unity MeshRenderer to display video frames.

    Component Interaction Model

    1. A PeerConnection uses a Signaler (e.g., NodeDssSignaler) to handle session establishment.
    2. A VideoSource is associated with a PeerConnection to manage video-related signals.
    3. VideoTrackPlayer components are optionally attached to a VideoSource to render local or remote tracks using a custom shader and a VideoFrameQueue.

  8. Manage object lifetime and ownership

    master

    The library manages object lifetimes through a hierarchical ownership model:

    1. Root Objects: The library maintains a global collection of all PeerConnection objects. These act as the roots of the ownership tree.
    2. Ownership Tree: All other objects are kept alive via an ownership tree with a PeerConnection object at its root.
    3. Reference Counting: Internally, the library uses rtc::scoped_refptr for reference counting.

    Critical Note for Unity Developers: When using the library in the Unity editor, the library may be unloaded and reloaded during Play sessions. It is vital to ensure all threads are stopped and resources are released to prevent leaks or crashes. If a language wrapper (like C#) is still holding a reference to an object (e.g., due to an incomplete asynchronous call), that object may persist even after the library is unloaded.

  11. How remote video and audio work in Unity

    master

    In MixedReality-WebRTC, remote media is controlled by the remote peer. The receiver side does not initiate the track; instead, it decides whether to handle the incoming data.

    To handle remote media in Unity, you use receiver components that act as bridges between the WebRTC connection and Unity's rendering/audio systems:

    • VideoReceiver: Exposes a remote video track as a frame queue. It does not render anything on its own.
    • VideoRenderer: Consumes the frames from a VideoReceiver and converts them into textures for display.
    • AudioReceiver: Receives remote audio data and requires a Unity AudioSource to inject that data into the Unity DSP pipeline for playback.

    These components must be associated with a PeerConnection via transceiver media lines to function.

  12. Understand Peer Connection and its components

    master

    The Peer connection is the central entity in MixedReality-WebRTC. It manages the connection to a single remote peer. A Peer connection is composed of two main types of communication pipes:

    1. Transceivers: Used for transporting Media (audio and/or video).
    2. Data channels: Used for sending and receiving arbitrary blobs of bytes.

    A Peer connection can have zero or more of these components.