DOCSEARCHbyAlgolia
Submit Docs

stable-ts

repository·main·Indexed 25 days ago

https://github.com/jianfch/stable-ts

A library that modifies OpenAI's Whisper to produce more reliable timestamps and extends transcription control. It provides advanced preprocessing like voice isolation and noise removal, postprocessing via VAD-based timestamp adjustment, and support for multiple backends including faster-whisper, Hugging Face Transformers, and MLX Whisper for Apple Silicon. It includes a Python API and CLI for transcribing audio and exporting results to SRT, VTT, ASS, TSV, TXT, and JSON formats.

Tokens
16.4K
Snippets
31
Records
61
Agent score
71%

What's inside stable-ts

  1. How silence suppression works

    main

    Silence suppression adjusts Whisper's predicted timestamps to prevent them from starting too early or ending too late relative to the actual speech. It is enabled by default via suppress_silence=True.

    Methods of detection:

    1. Relative Loudness (Default): Determines non-speech timestamps based on how loud a section is relative to its neighbors. Most effective when speech is significantly louder than background noise.
    2. Silero VAD: Uses the Silero Voice Activity Detector for more robust detection. Enable this by setting vad=True.

    Note on Versions:

    • In 2.X, silence suppression is a post-inference timestamp adjustment, allowing it to work with other ASR models.
    • The older feature of suppressing timestamp tokens during inference is disabled by default but can be enabled using suppress_ts_tokens=True.

  2. Regroup words into segments using presets or custom algorithms

    main

    Stable-ts allows regrouping words into segments with more natural boundaries. By default, model.transcribe() uses a preset regrouping algorithm (regroup=True).

    You can customize this by either chaining specific regrouping methods or by providing a compact string representation of an algorithm.

    Chaining Methods

    You can chain methods like .split_by_punctuation(), .split_by_gap(), and .split_by_length() directly on the result object.

    String Representation

    All chainable methods are recorded in result.regroup_history as a string. This string can be used to reproduce the same operations via result.regroup(history).

    Example of equivalent operations:

    # Using chaining
result1 = model.transcribe('audio.mp3', regroup=False).ignore_special_periods().clamp_max().split_by_punctuation([('.', ' '), '。', '?', '？']).split_by_gap(.5).split_by_punctuation([(',', ' '), '，'], min_chars=50).split_by_length(70).clamp_max()

# Using a single string
result2 = model.transcribe('audio.mp3', regroup='isp_cm_sp=.* /。/?/？_sg=.5_sp=,* /，++++50_sl=70_cm')
    # The following results are all functionally equivalent:
result0 = model.transcribe('audio.mp3', regroup=True) # regroup is True by default
result1 = model.transcribe('audio.mp3', regroup=False)
(
    result1
    .ignore_special_periods()
    .clamp_max()
    .split_by_punctuation([('.', ' '), '。', '?', '？'])
    .split_by_gap(.5)
    .split_by_punctuation([(',', ' '), '，'], min_chars=50)
    .split_by_length(70)
    .clamp_max()
)
result2 = model.transcribe('audio.mp3', regroup='isp_cm_sp=.* /。/?/？_sg=.5_sp=,* /，++++50_sl=70_cm')

  3. Migrate from stable-ts 1.x to 2.x

    main

    Version 2.0.0 introduces significant changes to how timestamps are handled and how results are exported.

    Key Improvements:

    • Uses Whisper's more reliable word-level timestamps.
    • Allows regrouping words into segments with more natural boundaries.
    • Supports silence suppression using Silero VAD (requires PyTorch 1.12.0+).
    • Improved non-VAD silence suppression.

    Breaking Changes in API:

    • Segment stabilization is no longer required after inference because segments are stabilized during the inference process itself.
    • The transcribe() method now returns a WhisperResult object instead of a dictionary. To get a dictionary, use the .to_dict() method.

  4. Use Stable-ts with other ASR models

    main
    Stable-ts features can be used to improve the results of most Automatic Speech Recognition (ASR) models or APIs, not just Whisper. For a guide on how to implement this with non-Whisper models, refer to the official notebook example.

  5. Use MLX Whisper on Apple Silicon

    main

    To achieve faster transcription on Apple devices, you can use the MLX Whisper implementation.

    1. Install the MLX extension:
    pip install -U stable-ts[mlx]
    1. Use load_mlx_whisper in your code:
    import stable_whisper

model = stable_whisper.load_mlx_whisper('base')
result = model.transcribe('audio.mp3')

    CLI Usage:

    stable-ts audio.mp3 -o audio.srt -mlx

  6. Best practices and tips for stable-ts

    main

    Transcription & Accuracy

    • Word Timestamps: Do not disable them with word_timestamps=False if you want reliable segment timestamps.
    • VAD: Use vad=True for more accurate non-speech detection, especially in less clean audio.
    • Denoising: Use denoiser="demucs" to isolate vocals. This is highly effective for music or audio with background noise. Combine with vad=True for music.
    • Determinism: When using denoiser="demucs", set a constant seed (e.g., random.seed(0)) to ensure deterministic outputs.

    Performance & CLI

    • CPU Inference: To enable dynamic quantization on CPU, use the CLI flag --dq true or pass dq=True to stable_whisper.load_model.
    • CLI Persistence: Use the --persist or -p flag to keep the CLI running without reloading the model between commands.

  7. Install FFmpeg prerequisite

    main

    The library requires FFmpeg to be available in your system's PATH. Install it using your operating system's package manager:

    Ubuntu or Debian:

    sudo apt update && sudo apt install ffmpeg

    Arch Linux:

    sudo pacman -S ffmpeg

    MacOS (Homebrew):

    brew install ffmpeg

    Windows (Chocolatey):

    choco install ffmpeg

    Windows (Scoop):

    scoop install ffmpeg

  8. Install stable-ts with Hugging Face Transformers support

    main

    To transcribe up to 9x faster using Hugging Face Transformers (e.g., whisper-large-v3), install the hf extra:

    pip install -U stable-ts[hf]
    pip install -U stable-ts[hf]

  9. Install the whisperless version of stable-ts

    main

    If you want to install stable-ts without having whisper as a dependency, use the stable-ts-whisperless package:

    pip install -U stable-ts-whisperless

    To install the latest development commit of the whisperless version:

    pip install -U git+https://github.com/jianfch/stable-ts.git@whisperless

  11. Install stable-ts

    main

    Install the latest stable version of stable-ts using pip:

    pip install -U stable-ts

    If you need the latest development version from the GitHub repository, use:

    pip install -U git+https://github.com/jianfch/stable-ts.git

  12. Transcribe audio with stable-ts

    main

    You can transcribe audio files using either the Python API or the Command Line Interface (CLI). The Python API provides a transcribe method that includes advanced preprocessing (voice isolation, noise removal, filtering) and postprocessing (VAD-based timestamp adjustment and segment regrouping).

    Python API Usage

    import stable_whisper
model = stable_whisper.load_model('base')
result = model.transcribe('audio.mp3')
result.to_srt_vtt('audio.srt')

    CLI Usage

    stable-ts audio.mp3 -o audio.srt
    import stable_whisper
model = stable_whisper.load_model('base')
result = model.transcribe('audio.mp3')
result.to_srt_vtt('audio.srt')