WhisperS2T Documentation

You can provide a custom Voice Activity Detection (VAD) model by wrapping it in the whisper_s2t.speech_segmenter.VADBaseClass.

Your custom class must implement a __call__ method that:

1. Accepts an audio_signal as input.
2. Returns a numpy array of shape T x 3 (where T is frame length).
3. Each row must contain [speech_prob, frame_start_time, frame_end_time].

Pass the instance to load_model via the vad_model parameter.

# Assuming CustomVAD inherits from whisper_s2t.speech_segmenter.VADBaseClass
vad_model = CustomVAD()
model = whisper_s2t.load_model(model_identifier="large-v2", backend='CTranslate2', vad_model=vad_model)

You can use prebuilt images or build your own. When building, you can control the version and whether to include TensorRT-LLM support via build arguments.

# Pull prebuilt dev-trtllm image
docker pull shashikg/whisper_s2t:dev-trtllm

# Build from main branch (skipping TensorRT-LLM)
docker build --build-arg WHISPER_S2T_VER=main --build-arg SKIP_TENSORRT_LLM=1 -t whisper_s2t:main .

# Build from specific release
git checkout v1.3.0
docker build --build-arg WHISPER_S2T_VER=v1.3.0 --build-arg SKIP_TENSORRT_LLM=1 -t whisper_s2t:1.3.0 .

# Build with TensorRT-LLM support
docker build --build-arg WHISPER_S2T_VER=main -t whisper_s2t:main-trtllm .

If VAD performance is poor for specific languages, you can bypass the segmentation step by calling model.transcribe() instead of model.transcribe_with_vad().

You can also tune VAD parameters (like eos_thresh and bos_thresh) by passing a speech_segmenter_options dictionary to load_model.

# Run without VAD
out = model.transcribe(files,
                       lang_codes=lang_codes,
                       tasks=tasks,
                       initial_prompts=initial_prompts,
                       batch_size=24)

# Tweak VAD parameters during model loading
speech_segmenter_options = {
    'eos_thresh': 0.1,
    'bos_thresh': 0.1,
}
model = whisper_s2t.load_model(speech_segmenter_options=speech_segmenter_options)

If you want to install the latest commit from the GitHub repository instead of the PyPI release, use the following command:

pip install -U git+https://github.com/shashikg/WhisperS2T.git

To use the TensorRT-LLM backend, you must have TensorRT and TensorRT-LLM installed. You can attempt to use the provided installation script for Debian-based systems:

```bash
bash <repo_dir>/install_tensorrt.sh

If the script fails, follow the official NVIDIA TensorRT-LLM instructions.

To install the latest released version of WhisperS2T, use pip. You must first ensure that audio packages for resampling and loading (like ffmpeg and libsndfile1) are installed on your system.

# Install audio dependencies first
# For Ubuntu:
apt-get install -y libsndfile1 ffmpeg
# For MAC:
brew install ffmpeg
# Using Conda:
conda install conda-forge::ffmpeg

# Install WhisperS2T
pip install -U whisper-s2t

Custom configurations can be passed as keyword arguments to load_model or applied to an existing model using model.update_params().

Common configuration keys include:

• compute_type: e.g., 'int8' (supported only for CTranslate2) or 'float16' (supported for other backends).
• asr_options: A dictionary for ASR-specific settings (e.g., BEST_ASR_CONFIG).

import whisper_s2t
from whisper_s2t.backends.ctranslate2.model import BEST_ASR_CONFIG

model_kwargs = {
    'compute_type': 'int8',
    'asr_options': BEST_ASR_CONFIG
}

# Pass during loading
model = whisper_s2t.load_model(model_identifier="large-v2", backend='CTranslate2', **model_kwargs)

# OR update after loading
model.update_params(model_kwargs)

To use the TensorRT-LLM backend, you must install system dependencies (libsndfile1, ffmpeg), the WhisperS2T package, and run the provided TensorRT installation script.

Follow these steps in order:

1. Install system dependencies via apt-get.
2. Install the latest WhisperS2T from GitHub.
3. Download and execute install_tensorrt.sh to set up the TensorRT environment.

# Install system dependencies
!apt-get update && apt-get install -y libsndfile1 ffmpeg

# Install WhisperS2T
!pip install -U git+https://github.com/shashikg/WhisperS2T.git

# Install TensorRT environment
!wget https://github.com/shashikg/WhisperS2T/raw/main/install_tensorrt.sh
!bash install_tensorrt.sh

To use WhisperS2T, you must install the required system libraries (libsndfile1 and ffmpeg) and the Python package directly from the GitHub repository.

apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install -U git+https://github.com/shashikg/WhisperS2T.git

!apt-get update && apt-get install -y libsndfile1 ffmpeg
!pip install -U git+https://github.com/shashikg/WhisperS2T.git

If your CUDNN and CUBLAS installations were performed using pip wheels, you may need to add the library paths to your LD_LIBRARY_PATH to ensure the inference engines can find them:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`

The CTranslate2 backend is highly optimized and supports features like word alignment and batching multiple languages/tasks. Use whisper_s2t.load_model with backend='CTranslate2' and then call transcribe_with_vad.

import whisper_s2t

# Load model with CTranslate2 backend
model = whisper_s2t.load_model(model_identifier="large-v2", backend='CTranslate2')

files = ['data/KINCAID46/audio/1.wav']
lang_codes = ['en']
tasks = ['transcribe']
initial_prompts = [None]

# Transcribe with VAD and batching
out = model.transcribe_with_vad(files,
                                lang_codes=lang_codes,
                                tasks=tasks,
                                initial_prompts=initial_prompts,
                                batch_size=32)

print(out[0][0])