audiolab Audio Processing Monorepo

Sherpa Voice is an open-source demo application designed to showcase on-device speech and audio AI capabilities. It operates entirely offline without cloud requirements, making it suitable for developers evaluating mobile inference or researchers exploring offline AI.

Key features available for testing include:

• Speech Recognition: Transcribing speech to text (files or live microphone).
• Text-to-Speech: Generating speech from text with various voices.
• Voice Activity Detection: Real-time detection of speech vs. silence.
• Keyword Spotting: Listening for specific custom keywords.
• Speaker Identification: Recognizing specific speakers by voice.
• Speaker Diarization: Identifying who spoke when in a conversation.
• Audio Tagging: Classifying sounds (music, speech, noise).
• Language Identification: Detecting the spoken language.
• Speech Enhancement: Removing background noise.
• Punctuation: Adding punctuation to raw transcripts.

Each feature utilizes a downloadable model catalog (typically 15-100 MB) to allow comparison of different models regarding speed, size, and accuracy.

The @siteed/audio-studio API is organized into several functional domains to handle the full audio lifecycle:

• Recording and Playback: Tools for capturing and playing audio, including the useAudioRecorder hook and AudioRecorderProvider for state management.
• Audio Device Management: APIs for detecting and selecting audio input devices via useAudioDevices or the audioDeviceManager singleton.
• Audio Analysis: Capabilities for extracting features, waveforms, and raw PCM data (e.g., extractAudioAnalysis, extractPreview).
• Specialized Audio Processing: Advanced tasks like generating mel spectrograms (extractMelSpectrogram) or trimming audio (trimAudio).
• Events and Results: Data structures for handling audio streams (AudioDataEvent) and completed recordings (AudioRecording).
• Utility Functions: Low-level helpers for PCM conversion and WAV file manipulation.

The stopRecording operation in @siteed/audio-studio (specifically within the Android AudioRecorderManager.kt implementation) has undergone significant optimization. Previously, stop times for 10-minute recordings could take up to 2554ms on iOS and 1070ms on Android. Optimized versions achieve sub-125ms stop times across platforms.

Key Performance Metrics

• Android: ~124ms for 10-minute recordings (92% improvement).
• iOS: ~18ms for 10-minute recordings (99.3% improvement).

Implementation Overview

The stopRecording method executes the following sequence:

1. Thread Synchronization: Uses audioRecordLock to prevent race conditions.
2. Buffer Reading: Reads remaining data from the audio buffer.
3. Service Cleanup: Stops notification and background recording services.
4. Thread Joining: Waits for the recording thread to complete (with a timeout).
5. Final Buffer Read: Captures any remaining data.
6. Cleanup Operations: Releases resources and updates the WAV header.
7. Result Calculation: Computes duration and prepares metadata.

The @siteed/audio-studio package provides a unified API for audio tasks across iOS, Android, and web. Key capabilities include:

Recording

• Zero-latency recording: Use the prepareRecording API to eliminate startup delays.
• Dual-stream recording: Record simultaneous raw PCM and compressed formats (OPUS/AAC).
• Background recording: Supports keep-awake functionality.
• Interruption handling: Automatic pause/resume during phone calls.

Device Management

• Enumeration: List all available audio input devices.
• Capabilities: Inspect supported sample rates, channels, and bit depths.
• Type Detection: Identify if a device is Bluetooth, USB, wired, or built-in.

Audio Processing & Analysis

• Feature Extraction: Extract energy, RMS, MFCC, spectral features, chromagram, and pitch detection.
• Mel Spectrograms: Generate spectrograms for ML or visualization.
• Trimming: Perform precision audio trimming with multi-segment support.
• Format Conversion: Normalize and convert audio formats.

Sherpa Voice uses different model stacks depending on whether the goal is high-accuracy offline diarization or low-latency live UX. Do not use the offline diarization stack for live UX.

Offline Long Diarization (Quality Focus)

Used for processing existing files where accuracy is paramount.

• Models: pyannote-segmentation-3-0/model.onnx and speaker-id-3dspeaker-eres2net-en.
• Settings: 5-minute native file windows, fixed numClusters (if known), and 2 threads.
• Note: Avoid quantized int8 segmentation for high-quality needs; model.onnx provides significantly better DER/JER results.

Live Transcription + Turn Attribution (Latency Focus)

Used for real-time streaming with bounded memory.

• Models: streaming-zipformer-en-20m-mobile, silero-vad-v5, and speaker-id-en-voxceleb.
• Settings: 100 ms chunks, minTurnDurationMs=1000, speechPadMs=120, and speakerThreshold=0.55.
• Threading: 1-2 ASR/Speaker threads and single-threaded VAD.
• Validation Criteria: Replay realtime factor $\le 1.0$, mic max queue depth $\le 2$, and zero drops.

When enableProcessing: true is set, the library provides two ways to access analysis data:

1. analysisData in the hook state (a rolling window for visualizers).
2. onAudioAnalysis callback (emits chunks during recording).

By default, the library retains the full history of analysis data in memory and attaches it to the result of stopRecording().analysisData. For long-running sessions, this can consume significant memory. To prevent this, set keepFullAnalysis: false.

Recommendation: For long sessions where you need full analysis, do not rely on keepFullAnalysis. Instead, use extractAudioAnalysis() on the saved file after the recording has stopped.

await startRecording({
  sampleRate: 16000,
  channels: 1,
  enableProcessing: true,
  keepFullAnalysis: false,
  intervalAnalysis: 250,
  onAudioAnalysis: async (analysis) => {
    // Use this chunk for live VAD, meters, or streaming decisions.
    updateVoiceActivity(analysis.dataPoints)
  },
})

const result = await stopRecording()
console.log(result.analysisData) // undefined when keepFullAnalysis is false

The web implementation of expo-audio-studio is a multi-threaded audio recording system designed for browsers. It uses a dual-path approach to provide both high-quality uncompressed audio and efficient compressed audio simultaneously.

Key Components

• AudioStudioWeb: The main entry point providing the public API and managing the recording lifecycle.
• WebRecorder: The core implementation that manages AudioContext, AudioWorklet, and the MediaRecorder API.
• AudioWorklet: A low-level processor running in a dedicated thread for real-time PCM audio processing.
• FeatureExtractor: A Web Worker dedicated to audio analysis (amplitude, frequency) for visualizations.

Data Flow

1. Capture: MediaDevices API captures microphone input.
2. Processing: AudioWorklet processes raw PCM data.
3. Compression: MediaRecorder API (optional) compresses audio chunks.
4. Analysis: FeatureExtractor analyzes audio for UI visualization.
5. Emission: AudioStudioWeb emits events containing the processed data to the client application.