WhisperPlus Documentation

Automatically generate and burn captions into a video file.

System Dependencies:

apt install imagemagick libmagick++-dev
cat /etc/ImageMagick-6/policy.xml | sed 's/none/read,write/g'> /etc/ImageMagick-6/policy.xml

Python Dependencies:

pip install moviepy

Usage: Use WhisperAutoCaptionPipeline with a video path.

from whisperplus.pipelines.whisper_autocaption import WhisperAutoCaptionPipeline
from whisperplus import download_youtube_to_mp4

video_path = download_youtube_to_mp4(
    "https://www.youtube.com/watch?v=di3rHkEZuUw",
    output_dir="downloads",
    filename="test",
)  # Optional

caption = WhisperAutoCaptionPipeline(model_id="openai/whisper-large-v3")
caption(video_path=video_path, output_path="output.mp4", language="english")

To use WhisperPlus, install the package via pip. If you are working in a Jupyter Notebook, you may also need to apply nest_asyncio to handle asynchronous loops.

pip install -U whisperplus
# Additional dependencies for advanced features
pip install git+https://github.com/huggingface/transformers
pip install flash-attn --no-build-isolation

!pip install -U whisperplus
!pip install whisperplus git+https://github.com/huggingface/transformers
!pip install flash-attn --no-build-isolation

Use LightningWhisperMLX for a specialized MLX implementation that supports batch processing.

from whisperplus.pipelines.lightning_whisper_mlx import LightningWhisperMLX
from whisperplus import download_youtube_to_mp3

url = "https://www.youtube.com/watch?v=1__CAdTJ5JU"
audio_path = download_youtube_to_mp3(url)

whisper = LightningWhisperMLX(model="distil-large-v3", batch_size=12, quant=None)
output = whisper.transcribe(audio_path=audio_path)["text"]

Use TextSummarizationPipeline to generate summaries from a transcript using models like facebook/bart-large-cnn.

from whisperplus.pipelines.summarization import TextSummarizationPipeline

summarizer = TextSummarizationPipeline(model_id="facebook/bart-large-cnn")
summary = summarizer.summarize(transcript)
print(summary[0]["summary_text"])

For optimized performance on Apple Silicon, use the mlx_whisper pipeline. This requires a model compatible with MLX (e.g., from the mlx-community on HuggingFace).

from whisperplus.pipelines import mlx_whisper
from whisperplus import download_youtube_to_mp3

url = "https://www.youtube.com/watch?v=1__CAdTJ5JU"
audio_path = download_youtube_to_mp3(url)

text = mlx_whisper.transcribe(
    audio_path, path_or_hf_repo="mlx-community/whisper-large-v3-mlx"
)["text"]
print(text)

For transcripts that exceed standard model context windows, use LongTextSummarizationPipeline.

from whisperplus.pipelines.long_text_summarization import LongTextSummarizationPipeline

summarizer = LongTextSummarizationPipeline(model_id="facebook/bart-large-cnn")
summary_text = summarizer.summarize(transcript)
print(summary_text)

Use TextToSpeechPipeline to convert text into audio using models like suno/bark.

from whisperplus.pipelines.text2speech import TextToSpeechPipeline

tts = TextToSpeechPipeline(model_id="suno/bark")
audio = tts(text="Hello World", voice_preset="v2/en_speaker_6")

Performance evaluations for Whisper models in this repository are based on the Mozilla-Foundation/Common-Voice-17-0 dataset. The benchmarks compare different quantization methods for the distil-whisper/distil-large-v3 model, specifically looking at performance metrics (likely WER/CER or similar, though the specific metric name is not defined in the table) using Hqq and bitsandbytes (Bnb) quantization.

| Model                                | Metric Value |
| ------------------------------------ | ------------ |
| distil-whisper/distil-large-v3 + Hqq | 120.88       |
| distil-whisper/distil-large-v3       | 120.48       |
| distil-whisper/distil-large-v3 + Bnb | 120.14       |

For users on Apple Silicon, you can use the mlx_whisper pipeline to leverage MLX for optimized transcription.

from whisperplus.pipelines import mlx_whisper
from whisperplus import download_youtube_to_mp3

url = "https://www.youtube.com/watch?v=1__CAdTJ5JU"
audio_path = download_youtube_to_mp3(url)

text = mlx_whisper.transcribe(
    audio_path, path_or_hf_repo="mlx-community/whisper-large-v3-mlx"
)["text"]

Downloads a YouTube video and uses WhisperAutoCaptionPipeline to generate and embed captions into the video file.

Parameters:

• url (str): The YouTube URL.
• language (str): The language for captioning.
• model_id (str): The Whisper model ID (defaults to openai/whisper-large-v3).

Returns:

• output: The path to the captioned video file.