MetaVoiceChat

MetaVoiceChat is a high-performance, modular audio system for Unity with the following characteristics:

Audio Specifications

• Sampling Frequency: 48kHz
• Bandwidth: Wideband Opus (SILK mode)
• Channels: Single audio channel (Mono)
• Bit Depth: 16-bit audio
• Loop Time: 1 second input and output audio clip loop
• Average Latency: ~200 ms with default settings

Key Features

• Zero Garbage: No memory garbage is created at runtime due to the use of pooled data buffers.
• Latency Compensation: Uses a dynamic buffer latency compensation system (P-controller) that adjusts based on RMS jitter, sender/host/receiver FPS, and server FPS.
• Microphone Management:
  • Automatic reconnection via VcMicAudioInput.SetSelectedDevice(string device).
  • VcMicAudioInput.OnActiveDeviceChanged event for monitoring device changes.
• Reactive Properties: Provides events for Speaking, Deafening yourself, Input muting yourself, and Output muting others.

To implement networked voice chat in Unity, follow these steps to configure your networked player object:

1. Create a Voice Chat Container: Create a new GameObject (e.g., named "Voice Chat") as a child of your networked player object.
2. Add Core Components:
   • Add the MetaVc component.
   • Add an audio input component (e.g., VcMicAudioInput).
   • Add an audio output component (e.g., VcAudioSourceOutput).
   • Add a network provider component (e.g., MirrorNetProvider, NgoNetProvider, FishNetNetProvider, or NetickNetProvider).
3. Connect Components:
   • Link the input and output components to their respective fields on the MetaVc component.
   • Link the network provider to the MetaVc component.
4. Configure Performance:
   • Adjust the Complexity field based on your user count/device capabilities.
   • Use Max Codec Milliseconds to manage complexity. Set this higher than your desired frame period during testing to avoid spamming warnings, then increase it to the maximum to disable it.
   • Enable Allow Multiple Codec Warnings Per Frame to see total frame time used by the codec during warnings.

Advanced audio tuning is available through the following configuration categories:

General Settings

• Jitter calculation time window size: Duration in seconds.
• Jitter mean time offset calculation window size: Measured in updates.
• Output min buffer size: Measured in fractional frames.

Audio Pipeline (Filters)

The pipeline is modular and allows for input and output filtering:

• VcAudioInput: Supports a First input filter (nullable).
• VcInputFilter: Supports a Next input filter (nullable).
• VcAudioOutput: Supports a First output filter (nullable).
• VcOutputFilter: Supports a Next output filter (nullable).

VcAudioSourceOutput Settings

Used for managing audio source playback and latency compensation:

• Audio source: The Unity Audio Source to use.
• Buffer frame lifetime: Duration in seconds.
• Buffer max negative latency: Duration in seconds.
• Pitch proportional gain: Percent per second of latency error.
• Pitch max correction: Maximum allowed correction in percent.

You can extend the functionality of MetaVoiceChat by implementing the following interfaces:

Custom Audio Input (VcAudioInput)

Extend VcAudioInput and call SendAndFilterFrame(int index, float[] samples) whenever audio data is ready.

• index: Starts at zero and never resets.
• samples: Pass the audio data array. Pass null when not sending audio data to maintain timing via empty heartbeats.
• Reference: VcMicAudioInput.cs

Custom Audio Output (VcAudioOutput)

Extend VcAudioOutput and implement ReceiveFrame(int index, float[] samples, float targetLatency).

• Reference: VcAudioSourceOutput.cs

Custom Input Filter (VcInputFilter)

Extend VcInputFilter and implement Filter(int index, ref float[] samples).

• To stop the pipeline (e.g., for Push-To-Talk), set samples to null when disabled.

Custom Output Filter (VcOutputFilter)

Extend VcOutputFilter and implement Filter(int index, float[] samples, float targetLatency).

• Directly modify the samples array to apply effects (e.g., for mouth movement visualization).