Insanely Fast Whisper

repository·main·Indexed 12 days ago

https://github.com/vaibhavs10/insanely-fast-whisper

An opinionated CLI and Python integration for high-speed, on-device audio transcription using OpenAI's Whisper models. It leverages Transformers, Optimum, and Flash Attention 2 to achieve higher throughput than standard implementations. Version 0.0.15 supports high-speed transcription, speaker diarization via Pyannote.audio, and optimized inference on CUDA and Apple Silicon (MPS) devices.

Tokens: 5.9K
Snippets: 20
Records: 21
Agent score: 96%

What's inside Insanely Fast Whisper

Install insanely-fast-whisper
main
You can install the CLI using pipx for an isolated environment.
Standard installation:
```
pipx install insanely-fast-whisper
```
Troubleshooting Python 3.11+ installation: If pipx incorrectly installs an old version (e.g., 0.0.8) due to Python 3.11 version parsing, use the following command to force the latest version:
```
pipx install insanely-fast-whisper --force --pip-args="--ignore-requires-python"
```
Alternative installation via pip:
```
pip install insanely-fast-whisper --ignore-requires-python
```
https://github.com/vaibhavs10/insanely-fast-whisper/blob/main/README.md
Run transcription via CLI
main
Use the insanely-fast-whisper command to transcribe audio files from a local path or a URL.
Basic usage:
```
insanely-fast-whisper --file-name <filename or URL>
```
macOS (Apple Silicon) usage: You must specify the mps device:
```
insanely-fast-whisper --file-name <filename or URL> --device-id mps
```
Using Flash Attention 2: To enable high-speed transcription with Whisper-large-v3 using Flash Attention 2:
```
insanely-fast-whisper --file-name <filename or URL> --flash True
```
Using Distil-Whisper models:
```
insanely-fast-whisper --model-name distil-whisper/large-v2 --file-name <filename or URL>
```
Running without installation:
```
pipx run insanely-fast-whisper --file-name <filename or URL>
```
https://github.com/vaibhavs10/insanely-fast-whisper/blob/main/README.md
Run Insanely Fast Whisper in Google Colab
main
To use Insanely Fast Whisper in a Google Colab environment, ensure you are using a GPU runtime. The process involves installing pipx and python3.10-venv, then executing the transcription via pipx run.
```
# 1. Install dependencies
!pip install -q pipx && apt install python3.10-venv

# 2. Run transcription
!pipx run insanely-fast-whisper --file-name https://huggingface.co/datasets/reach-vb/random-audios/resolve/main/ted_60.wav

# 3. Inspect the output
!head output.json
```
https://github.com/vaibhavs10/insanely-fast-whisper/blob/main/insanely_fast_whisper_colab.ipynb
Install faster-whisper and autotime
main
To use the faster-whisper implementation for Automatic Speech Recognition, install the package and ipython-autotime for benchmarking via pip.
```
!pip install --upgrade -q faster-whisper ipython-autotime
```
https://github.com/vaibhavs10/insanely-fast-whisper/blob/main/notebooks/infer_faster_whisper_large_v2.ipynb
Install dependencies for Whisper inference
main
To run optimized Whisper inference using Hugging Face pipeline, you need to install torch, transformers, accelerate, and optimum.
```
!pip install -q --upgrade torch torchvision torchaudio
!pip install -q git+https://github.com/huggingface/transformers
!pip install -q accelerate optimum
```
https://github.com/vaibhavs10/insanely-fast-whisper/blob/main/notebooks/infer_transformers_whisper_large_v2.ipynb
Use the Insanely Fast Whisper CLI
main
The CLI allows you to perform high-speed Automatic Speech Recognition (ASR) and speaker diarization on audio files. You can specify models, devices, batch sizes, and diarization parameters via command-line flags.
Basic usage involves providing a --file-name and an optional --transcript-path to save the results as a JSON file.
```
python -m insanely_fast_whisper --file-name "path/to/audio.mp3" --transcript-path "output.json"
```
https://github.com/vaibhavs10/insanely-fast-whisper/blob/main/src/insanely_fast_whisper/cli.py
Troubleshooting common issues
main
Installing Flash Attention 2
To ensure flash-attn works correctly with insanely-fast-whisper, install it using pipx runpip:
```
pipx runpip insanely-fast-whisper install flash-attn --no-build-isolation
```
Windows: AssertionError: Torch not compiled with CUDA enabled
If you encounter this error on Windows, manually install torch with CUDA support in your environment:
```
python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
```
macOS: Out-Of-Memory (OOM) exceptions
The mps backend is more memory-intensive than CUDA. To avoid OOMs on Mac, reduce the batch size (e.g., to 4) and ensure you use the --device-id mps flag:
```
insanely-fast-whisper --file-name <filename> --device-id mps --batch-size 4
```
https://github.com/vaibhavs10/insanely-fast-whisper/blob/main/README.md

How to use Whisper without a CLI

main

If you want to integrate the optimized Whisper pipeline directly into your Python code, use the following pattern with transformers, optimum, and accelerate:

Install dependencies:

pip install --upgrade transformers optimum accelerate

Use the pipeline API with optimized model_kwargs for Flash Attention 2.

import torch
from transformers import pipeline
from transformers.utils import is_flash_attn_2_available

pipe = pipeline(
    "automatic-speech-recognition",
    model="openai/whisper-large-v3", # select checkpoint from https://huggingface.co/openai/whisper-large-v3#model-details
    torch_dtype=torch.float16,
    device="cuda:0", # or mps for Mac devices
    model_kwargs={"attn_implementation": "flash_attention_2"} if is_flash_attn_2_available() else {"attn_implementation": "sdpa"},
)

outputs = pipe(
    "<FILE_NAME>",
    chunk_length_s=30,
    batch_size=24,
    return_timestamps=True,
)

outputs

Reference: CLI Options

main

The insanely-fast-whisper CLI supports the following arguments to control transcription behavior, hardware acceleration, and diarization:

Option	Description
`--file-name FILE_NAME`	Path or URL to the audio file.
`--device-id DEVICE_ID`	Device ID for GPU. Use a number for CUDA, or `mps` for Mac. (default: `0`)
`--transcript-path TRANSCRIPT_PATH`	Path to save output JSON. (default: `output.json`)
`--model-name MODEL_NAME`	Pretrained model/checkpoint. (default: `openai/whisper-large-v3`)
`--task {transcribe,translate}`	Task to perform. (default: `transcribe`)
`--language LANGUAGE`	Input audio language. (default: `None` for auto-detect)
`--batch-size BATCH_SIZE`	Parallel batches. Reduce if facing OOM. (default: `24`)
`--flash FLASH`	Use Flash Attention 2. (default: `False`)
`--timestamp {chunk,word}`	Timestamp granularity. (default: `chunk`)
`--hf-token HF_TOKEN`	Hugging Face token for Pyannote.audio diarization.
`--diarization_model DIARIZATION_MODEL`	Model for diarization. (default: `pyannote/speaker-diarization`)
`--num-speakers NUM_SPEAKERS`	Exact number of speakers.
`--min-speakers MIN_SPEAKERS`	Minimum number of speakers.
`--max-speakers MAX_SPEAKERS`	Maximum number of speakers.

  -h, --help            show this help message and exit
  --file-name FILE_NAME                        Path or URL to the audio file to be transcribed.
  --device-id DEVICE_ID                        Device ID for your GPU. Just pass the device number when using CUDA, or "mps" for Macs with Apple Silicon. (default: "0")
  --transcript-path TRANSCRIPT_PATH            Path to save the transcription output. (default: output.json)
  --model-name MODEL_NAME                      Name of the pretrained model/ checkpoint to perform ASR. (default: openai/whisper-large-v3)
  --task {transcribe,translate}                Task to perform: transcribe or translate to another language. (default: transcribe)
  --language LANGUAGE                          Language of the input audio. (default: "None" (Whisper auto-detects the language))
  --batch-size BATCH_SIZE                      Number of parallel batches you want to compute. Reduce if you face OOMs. (default: 24)
  --flash FLASH                                Use Flash Attention 2. Read the FAQs to see how to install FA2 correctly. (default: False)
  --timestamp {chunk,word}                     Whisper supports both chunked as well as word level timestamps. (default: chunk)
  --hf-token HF_TOKEN                          Provide a hf.co/settings/token for Pyannote.audio to diarise the audio clips
  --diarization_model DIARIZATION_MODEL        Name of the pretrained model/ checkpoint to perform diarization. (default: pyannote/speaker-diarization)
  --num-speakers NUM_SPEAKERS                  Specifies the exact number of speakers present in the audio file. Must be at least 1. Cannot be used together with --min-speakers or --max-speakers. (default: None)
  --min-speakers MIN_SPEAKERS                  Sets the minimum number of speakers that the system should consider during diarization. Must be at least 1. Cannot be used together with --num-speakers. Must be less than or equal to --max-speakers if both are specified. (default: None)
  --max-speakers MAX_SPEAKERS                  Defines the maximum number of speakers that the system should consider in diarization. Must be at least 1. Must be greater than or equal to --min-speakers if both are specified. (default: None)

Use BetterTransformer for optimized inference

main

You can further optimize the model by converting it to a BetterTransformer. This utilizes specialized kernels to accelerate the attention mechanism in the transformer model.

pipe = pipeline("automatic-speech-recognition",
                "openai/whisper-large-v2",
                torch_dtype=torch.float16,
                device="cuda:0")

# Convert the underlying model to BetterTransformer
pipe.model = pipe.model.to_bettertransformer()

outputs = pipe("sam_altman_lex_podcast_367.flac",
               chunk_length_s=30,
               batch_size=16,
               return_timestamps=True)

Run transcription on GPU with 8-bit quantization

main

To reduce memory footprint and potentially increase speed, you can run the model using 8-bit quantization. Set compute_type="int8_float16" when initializing the WhisperModel on a CUDA device.

from faster_whisper import WhisperModel

model_size = "large-v2"
# Run on GPU with 8-bit
model = WhisperModel(model_size, device="cuda", compute_type="int8_float16")

segments, info = model.transcribe("audio_file.flac", beam_size=1)

print("Detected language '%s' with probability %f" % (info.language, info.language_probability))

for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

Speed up transcription with Half-Precision (FP16)
main
Using torch.float16 instead of the default float32 reduces memory usage and increases speed on compatible GPUs. Combine this with batch_size and return_timestamps=True for maximum efficiency.
```
pipe = pipeline("automatic-speech-recognition",
                "openai/whisper-large-v2",
                torch_dtype=torch.float16,
                device="cuda:0")

outputs = pipe("sam_altman_lex_podcast_367.flac",
               chunk_length_s=30,
               batch_size=16,
               return_timestamps=True)
```
https://github.com/vaibhavs10/insanely-fast-whisper/blob/main/notebooks/infer_transformers_whisper_large_v2.ipynb

Insanely Fast Whisper

What's inside Insanely Fast Whisper

Installing Flash Attention 2

Windows: AssertionError: Torch not compiled with CUDA enabled

macOS: Out-Of-Memory (OOM) exceptions