DOCSEARCHbyAlgolia
Submit Docs

WhisperX: Time-Accurate Automatic Speech Recognition

repository·main·Indexed 10 days ago

https://github.com/m-bain/whisperx

A high-speed ASR system providing word-level timestamps and speaker diarization. It utilizes a faster-whisper backend for batched inference and forced phoneme alignment for improved accuracy. Version 3.8.7rc1 supports transcription, translation, and diarization via a CLI and Python API, with optimized performance for GPU acceleration using CUDA toolkit 12.8.

Tokens
12.5K
Snippets
45
Records
57
Agent score
95%

What's inside WhisperX

  1. Understand WhisperX transcription differences from OpenAI Whisper

    main

    WhisperX implements several architectural differences from the original OpenAI Whisper to improve performance and accuracy:

    • Single-pass batching: To enable efficient batching, WhisperX performs inference with --without_timestamps True. This ensures one forward pass per sample in a batch, though it may cause slight discrepancies compared to default Whisper output.
    • VAD-based segment transcription: Unlike OpenAI's buffered transcription, WhisperX uses Voice Activity Detection (VAD) to segment audio. This reduces Word Error Rate (WER) and enables accurate batched inference.
    • Reduced Hallucination: The --condition_on_prev_text option is set to False by default, which helps reduce hallucinations in the transcription.

  2. Enable Speaker Diarization

    main

    To use speaker diarization (labeling segments with speaker IDs), you must:

    1. Generate a Hugging Face read access token.
    2. Accept the user agreement for the pyannote/speaker-diarization-community-1 model on Hugging Face.
    3. Provide the token using the --hf_token argument in the CLI or the token parameter in the Python DiarizationPipeline.

  3. Install WhisperX via PyPI or uvx

    main

    The recommended way to install WhisperX is through PyPI using pip or via uvx for running tools directly.

    Prerequisites If you intend to use GPU acceleration, you must install the CUDA toolkit 12.8 before installing WhisperX. For CPU-only usage, this step can be skipped.

    # Recommended installation
pip install whisperx

# Or using uvx
uvx whisperx

  4. Transcribe non-English audio with WhisperX

    main

    For Automatic Speech Recognition (ASR) in languages other than English, it is recommended to use the large Whisper model. WhisperX automatically selects the appropriate alignment models based on the language provided.

    Currently, the system includes tested default models for the following language codes:

    • en (English)
    • fr (French)
    • de (German)
    • es (Spanish)
    • it (Italian)
    • ja (Japanese)
    • zh (Chinese)
    • nl (Dutch)

    If the detected language is not in this supported list, you must manually find and use a phoneme-based ASR model from the Hugging Face model hub.

  5. Reduce GPU memory requirements in WhisperX

    main

    If you encounter GPU memory issues, you can reduce the footprint using the following methods:

    1. Reduce batch size: Use the --batch_size flag (e.g., --batch_size 4).
    2. Use a smaller ASR model: Switch to a lighter model using the --model flag (e.g., --model base).
    3. Use a lighter compute type: Use quantized computation by setting --compute_type int8.

    Note that using smaller models or different compute types may affect transcription quality.

    # Example of reducing memory via batch size and compute type
whisperx audio.wav --batch_size 4 --compute_type int8

  6. Advanced Installation Options for WhisperX

    main

    Developers or users with specific requirements can use alternative installation methods:

    • Install from GitHub: Use uvx to install directly from the repository.
    • Developer Installation: Clone the repository and use uv sync to set up a development environment with all extras.

    Note: You may also need to install ffmpeg and rust following OpenAI's Whisper setup instructions.

    # Install from GitHub
uvx git+https://github.com/m-bain/whisperX.git

# Developer Installation
git clone https://github.com/m-bain/whisperX.git
cd whisperX
uv sync --all-extras --dev

  7. How WhisperModel provides batched inference

    main

    The WhisperModel class extends faster_whisper.WhisperModel to provide batched inference capabilities. This is achieved through the generate_segment_batched method, which allows processing multiple audio features simultaneously.

    Note: Currently, batched inference in WhisperModel is optimized for non-timestamp mode and assumes a fixed prompt for all samples in a batch. It uses the encode method to generate encoder outputs and then calls the underlying model's generate method with a list of prompts corresponding to the batch size.

  8. How IntervalTree works for fast speaker assignment

    main

    The IntervalTree is a utility class used to optimize the process of matching transcription timestamps to speaker segments. Instead of a linear scan ($O(n)$), it uses a sorted array and binary search to provide $O(\log n)$ query time. This is critical for performance in long-form content like podcasts.

    • query(start, end): Returns a list of (speaker, intersection_duration) tuples for all segments overlapping the range [start, end].
    • find_nearest(time): Returns the speaker ID of the segment whose midpoint is closest to the provided time point.
    from whisperx.diarize import IntervalTree

# Initialize with (start, end, speaker) tuples
intervals = [(0.0, 5.0, "SPEAKER_00"), (6.0, 10.0, "SPEAKER_01")]
tree = IntervalTree(intervals)

# Find overlaps
overlaps = tree.query(4.0, 7.0) # Returns [("SPEAKER_00", 1.0), ("SPEAKER_01", 1.0)]

# Find nearest speaker
nearest = tree.find_nearest(5.5) # Returns "SPEAKER_00"

  9. Resolve 'Unable to load any of {libcudnn...}' errors

    main

    If WhisperX fails with an error stating it is unable to load specific libcudnn libraries, it means the libraries are installed but not in the system's dynamic linker path.

    Solution 1: Update LD_LIBRARY_PATH in Python (Recommended)

    You can append the cuDNN library path to the LD_LIBRARY_PATH environment variable at the start of your script. Ensure you adjust the path to match your specific Python version.

    Solution 2: Symlink libraries to an existing LD_LIBRARY_PATH directory

    If the Python environment variable update fails, you can manually symlink the libraries to a directory already recognized by your system.

    1. Identify an existing path in your environment: echo "$LD_LIBRARY_PATH".
    2. Create symlinks for the libcudnn files into that directory.
    import os

# Get current LD_LIBRARY_PATH
original = os.environ.get("LD_LIBRARY_PATH", "")

# Adjust the python version (e.g., python3.12) to match your environment
cudnn_path = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
os.environ['LD_LIBRARY_PATH'] = original + ":" + cudnn_path

  10. Resolve cuDNN version incompatibility errors

    main

    If you encounter a RuntimeError: cuDNN version incompatibility (e.g., PyTorch was compiled against one version but found another), it is usually because a conflicting cuDNN version in your LD_LIBRARY_PATH is taking precedence over the version bundled with PyTorch.

    Option 1: Let PyTorch use its bundled cuDNN

    Clear the LD_LIBRARY_PATH environment variable to allow PyTorch to fall back to its own internal libraries.

    Option 2: Point specifically to the correct cuDNN version

    Explicitly set LD_LIBRARY_PATH to the directory containing the cuDNN version that matches your requirements (typically the one installed via your Python packages).

    import os

# Option 1: Clear LD_LIBRARY_PATH to use PyTorch's bundled cuDNN
os.environ.pop('LD_LIBRARY_PATH', None)

# Option 2: Point only to the specific cuDNN version required
# Adjust the python version to match your environment
os.environ['LD_LIBRARY_PATH'] = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"