Whisper: Robust Speech Recognition via Large-Scale Weak Supervision

You can install the latest release of Whisper via pip. The codebase is compatible with Python 3.8-3.11 and recent PyTorch versions.

To install the latest release:

pip install -U openai-whisper

To install directly from the GitHub repository:

pip install git+https://github.com/openai/whisper.git

To update to the latest commit from the repository:

pip install --upgrade --no-deps --force-reinstall git+https://github.com/openai/whisper.git

Note: If you encounter installation errors related to tiktoken, you may need to install the rust development environment and ensure it is in your PATH. If you see No module named 'setuptools_rust', run:

pip install setuptools-rust

To create the long-form transcription dataset used in the Whisper evaluation from the TED-LIUM3 dataset, slice the audio between the beginning of the first labeled segment and the end of the last labeled segment of each talk. Use the concatenated text as the label.

Timestamps for slicing the 11 TED talks in the test split:

If you are using the CallHome or Switchboard corpora (LDC2002S09 and LDC2002T43) and have a wav.scp file, you can convert the entries into actual WAV files using the following bash command. This assumes the command in the wav.scp file can be executed to produce the audio stream.

mkdir -p wav
while read name cmd; do
    echo $name
    echo ${cmd/\|/} wav/$name.wav | bash
done < wav.scp

Whisper requires the ffmpeg command-line tool to be installed on your system. Use your platform's package manager to install it:

• Ubuntu/Debian: sudo apt update && sudo apt install ffmpeg
• Arch Linux: sudo pacman -S ffmpeg
• MacOS (Homebrew): brew install ffmpeg
• Windows (Chocolatey): choco install ffmpeg
• Windows (Scoop): scoop install ffmpeg

# Example for Ubuntu/Debian
sudo apt update && sudo apt install ffmpeg

To use Whisper models and evaluate transcription results (e.g., calculating Word Error Rate), install the Whisper package directly from the GitHub repository and the jiwer library.

! pip install git+https://github.com/openai/whisper.git
! pip install jiwer

Whisper offers several model sizes with different trade-offs between speed and accuracy. There are multilingual models and English-only (.en) versions for certain sizes.

Notes:

• .en models generally perform better for English-only tasks.
• The turbo model is an optimized version of large-v3 that is faster with minimal accuracy loss, but it is not trained for translation tasks.

You can transcribe audio files directly from the terminal.

Basic transcription (using turbo model):

whisper audio.flac audio.mp3 audio.wav --model turbo

Transcribing specific languages:

whisper japanese.wav --language Japanese

Translating non-English speech into English: To translate, you must use a multilingual model (e.g., medium or large) instead of turbo. The turbo model will return the original language even if --task translate is specified.

whisper japanese.wav --model medium --language Japanese --task translate

Run whisper --help to see all available options.

To ensure accurate Word Error Rate (WER) calculations, use EnglishTextNormalizer from whisper.normalizers to standardize both the hypothesis (transcription) and the reference (ground truth) text.

from whisper.normalizers import EnglishTextNormalizer

normalizer = EnglishTextNormalizer()
clean_text = normalizer("Your transcription text here")

To transcribe audio, pass log-mel spectrograms and a DecodingOptions object to the model.decode() method. The method returns a list of results containing the transcribed text.

# Assuming 'mels' is a batch of log-mel spectrograms and 'options' is a DecodingOptions object
results = model.decode(mels, options)

# Extracting text from results
for result in results:
    print(result.text)