DOCSEARCHbyAlgolia
Submit Docs

LiveKit WakeWord

repository·main·Indexed 42 days ago

https://github.com/livekit/livekit-wakeword

An open-source library for high-accuracy wake word detection in voice-enabled applications. It features a specialized Conv-Attention architecture to reduce false positives and supports Python, Rust, and Swift (iOS/macOS) via ONNX Runtime. The toolkit includes a full pipeline for training custom wake words, from TTS synthesis and augmentation to evaluation and export to ONNX or TFLite formats.

Tokens
23.7K
Snippets
69
Records
100
Agent score
51%

What's inside LiveKit WakeWord

  1. Overview of the livekit-wakeword module map

    main

    The project is organized into several functional modules within src/livekit/wakeword/:

    • config.py: Pydantic models for configuration and YAML loading.
    • cli.py: Typer-based CLI for the full lifecycle (setup, generate, augment, train, export, eval, run).
    • models/: Contains feature extractors (feature_extractor.py), classifier architectures (classifier.py), and the training wrapper (pipeline.py).
    • data/: Handles data orchestration including TTS (generate.py), augmentation (augment.py), dataset management (dataset.py), and feature extraction (features.py).
    • training/: Contains the WakeWordTrainer and evaluation metrics.
    • eval/: Tools for model evaluation (DET curves, FPPH, etc.).
    • export/: Logic for ONNX export and INT8 quantization.
    • inference/: The runtime API, including WakeWordModel for predictions and WakeWordListener for async microphone detection.

  2. Generate adversarial negative phrases

    main

    The pipeline automatically generates adversarial phrases to train the model to reject near-misses (phonetically similar but incorrect phrases).

    How it works

    1. It uses the CMU Pronouncing Dictionary via NLTK.
    2. For unknown words, it attempts to split them into known subwords (e.g., "livekit" $\rightarrow$ ["live", "kit"]) to allow phoneme substitution.
    3. It replaces 1 to max_replace phonemes with a wildcard (.){1,3} and searches for similar words in the dictionary.
    4. It excludes homophones to ensure the phrases are actually adversarial.

    Customizing Negatives

    You can provide your own list of negative phrases in your configuration using the custom_negative_phrases key. These will be appended to the auto-generated adversarial phrases before synthesis.

  3. Understand the Feature Extraction Pipeline

    main

    The feature extraction pipeline converts raw audio clips into fixed-size embedding arrays suitable for wake word classification. It uses two frozen ONNX models in sequence:

    1. MelSpectrogramFrontend: Converts raw audio (16kHz, float32) into normalized mel-spectrogram features of shape (batch, time_frames, 32).
    2. SpeechEmbedding: Processes the mel-spectrogram using a sliding window to produce 96-dimensional embedding vectors of shape (batch, n_windows, 96).
    3. Timestep Selection: The sequence is normalized to exactly 16 timesteps. If there are more than 16 windows, the last 16 are kept. If there are fewer, the sequence is left-padded with zeros.

    The final output is a float32 array of shape (batch, 16, 96).

    .wav files (16kHz, ~2s)
    │
    ▼
MelSpectrogramFrontend (ONNX)
    │  (batch, time_frames, 32)
    ▼
SpeechEmbedding (ONNX)
    │  window=76, stride=8
    │  (batch, n_windows, 96)
    ▼
Take last 16 timesteps (or left-pad)
    │  (batch, 16, 96)
    ▼
Save as .npy

  4. Understand the livekit-wakeword hybrid architecture

    main

    livekit-wakeword uses a hybrid architecture combining frozen ONNX models for feature extraction and a lightweight PyTorch-trained classifier head.

    Core Components

    • Feature Extractors (Frozen ONNX): Two models handle the heavy lifting of converting raw audio into high-level features. Because they are frozen and run via ONNX Runtime, they provide fast, numpy-based inference without requiring a full PyTorch installation at detection time. These extractors are shared across all wake words.
    • Classifier Head (PyTorch/ONNX): A small, task-specific model trained for a particular wake word. It is trained in PyTorch for flexibility (using focal loss, embedding mixup, etc.) and then exported to ONNX for efficient deployment.

    Inference Pipeline

    1. Raw Audio: 16kHz mono float32 audio.
    2. MelSpectrogramFrontend: Converts audio to a 32-band mel spectrogram.
    3. SpeechEmbedding: Converts spectrograms into 96-dimensional speech embedding vectors.
    4. Classifier: Processes the embeddings to produce a sigmoid confidence score between 0 and 1.

  5. Understand the 3-Phase Training Pipeline

    main

    The training process uses an adaptive 3-phase approach to balance high recall with low false positives (FPPH).

    Phase 1: Full Training

    • Goal: Initial model convergence.
    • Schedule: Linear warmup (steps // 5) $\rightarrow$ Constant LR hold (steps // 3) $\rightarrow$ Cosine decay to 0.
    • Loss: Focal loss with linear negative weight scaling from 1.0 to max_negative_weight.

    Phase 2: Refinement

    • Goal: Reduce false positives.
    • Schedule: $0.1\times$ the original learning rate; duration is steps // 10.
    • Adaptive Logic: If the validation FPPH exceeds target_fp_per_hour, the max_negative_weight is doubled to increase the penalty on false positives.

    Phase 3: Fine-Tuning

    • Goal: Final precision tuning.
    • Schedule: $0.01\times$ the original learning rate; duration is steps // 10.
    • Adaptive Logic: If FPPH still exceeds the target, max_negative_weight is doubled again.

  6. How clip alignment works for positive and negative clips

    main

    Alignment ensures clips are positioned correctly within the target detection window (default 2.0 seconds / 32,000 samples at 16kHz). This step is only performed during Round 0.

    Positive Clips (End-Aligned)

    Positive clips (containing the wake word) are placed at the end of the window with a random jitter of up to 3200 samples (200ms at 16kHz). This simulates real-world scenarios where the wake word appears at the trailing edge of an audio buffer.

    Visual Model: [ zero padding | wake word | jitter ]

    Negative Clips (Center-Padded)

    Negative clips are centered within the target window.

    • If the clip is longer than the target, it is center-cropped.
    • If the clip is shorter than the target, it is center-padded with zeros.

  7. How timestep selection and left-padding work

    main

    To ensure the classifier receives a consistent input, every audio clip is normalized to exactly 16 embedding timesteps (defined by N_EMBEDDING_TIMESTEPS = 16 in features.py).

    Selection Logic:

    • $\ge$ 16 windows: The last 16 windows are taken (embeddings[-16:]).
    • < 16 windows: The sequence is left-padded with zeros to reach 16.

    Why Left-Padding? Left-padding places zeros at the beginning of the sequence so that real audio features occupy the end (the most recent positions). This provides:

    1. Causal alignment: The model learns patterns like "silence $\rightarrow$ wake word".
    2. Streaming compatibility: In streaming inference, the most recent audio (the potential wake word) is always at the end of the sequence.
    3. Consistent anchor point: The wake word utterance is always anchored to the end of the feature sequence regardless of clip length.

  8. Understand evaluation metrics (AUT, FPPH, Recall)

    main

    The evaluation process produces several key metrics to assess model performance:

    • AUT (Area Under the DET Curve): The primary aggregate metric. It integrates the False Negative Rate (FNR) as a function of the False Positive Rate (FPR). Lower is better (0 = perfect separation).
    • DET Curve: A plot of False Positive Rate (x-axis) vs. False Negative Rate (y-axis). A perfect model hugs the origin.
    • FPPH (False Positives Per Hour): The number of false triggers per hour of negative audio. Calculated as count(negative_scores >= threshold) / validation_hours.
    • Recall: The true positive rate at the optimal threshold, calculated as mean(positive_scores >= threshold).
    • Threshold Optimization: The system selects an optimal threshold by scanning from 0.01 to 0.99 to maximize recall while ensuring FPPH ≤ target_fp_per_hour (defined in your config).

  9. Understand the Wake Word Demo architecture

    main

    The demo follows a pipeline that moves audio from the hardware to the ML model and finally to the UI:

    1. Audio Capture: AVAudioEngine taps the microphone.
    2. Preprocessing: Audio is converted from Float32 to Int16 and stored in a 2-second Int16 ring buffer.
    3. Inference: A background queue runs WakeWordModel.predict() using the buffered audio.
    4. UI Update: The results (score and level) are published to the @MainActor to update the SwiftUI ContentView (graphs and meters).

    Note: The Swift code does not perform the ML arithmetic itself; it acts as a wrapper that calls into the LiveKitWakeWord Swift package, which uses ONNX Runtime to execute the mel-spectrogram and embedding models.

  10. How the Conv-Attention classifier works

    main

    The livekit-wakeword library uses a Conv-Attention (conv_attention) classifier as its default architecture. Unlike standard DNN heads that flatten the feature matrix, the Conv-Attention head preserves temporal structure by using:

    1. 1D Convolutions (kernel size 3): Captures local temporal patterns like syllable transitions.
    2. Multi-Head Self-Attention: Models long-range dependencies across the 16-frame embedding window.
    3. Mean Pooling: Aggregates attended features into a fixed-size vector for the final sigmoid output.

    This architecture provides better temporal awareness, allowing the model to distinguish between phonetically similar phrases that occur in different orders, resulting in significantly lower false-positive rates (FPPH) compared to flat DNN models.

  11. How the Augmentation Pipeline works

    main

    The pipeline applies transformations in a specific order to create progressively more degraded audio. It uses a 'stacking' mechanism across multiple rounds.

    Processing Order

    For each WAV file:

    1. Pre-processing: Read audio, convert to float32, and take the first channel if stereo.
    2. Per-sample augmentations: Applies SevenBandParametricEQ (25% prob) and TanhDistortion (25% prob).
    3. RIR Convolution: Convolves audio with a random Room Impulse Response (50% prob).
    4. Background Mixing: Mixes audio with random background noise at a random SNR.
    5. Alignment (Round 0 only): Aligns clips to a target window (e.g., 2.0s).
    6. Output: Writes to a new file with the suffix _r{round}.wav.

    Augmentation Rounds and Stacking

    The pipeline runs config.augmentation.rounds passes. Rounds are stacked:

    • Round 0 reads the clean original TTS clips.
    • Round 1 reads the output of Round 0.
    • Round N reads the output of Round N-1.

    This compounding effect creates increasingly complex audio environments. Original files are never modified and are preserved in the output directory.

  12. Install livekit-wakeword for training

    main

    To train custom wake words, install the package with the necessary extras for training, evaluation, and exporting. You must also install system dependencies for audio processing.

    System Dependencies

    macOS:

    brew install espeak-ng ffmpeg portaudio

    Ubuntu/Debian:

    sudo apt install espeak-ng libsndfile1 ffmpeg sox portaudio19-dev

    Python Installation

    Using pip:

    pip install livekit-wakeword[train,eval,export]

    Using uv:

    uv tool install livekit-wakeword[train,eval,export]

    Installation from source

    # Install uv (if you don't have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone and install
git clone https://github.com/livekit/livekit-wakeword
cd livekit-wakeword
uv sync --all-extras
    pip install livekit-wakeword[train,eval,export]