WhisperKitTools

To use whisperkittools, follow these steps to set up a Python environment and install the package in editable mode:

1. Fork the repository.
2. Create and activate a Python 3.11 virtual environment (e.g., using Conda):

   conda create -n whisperkit python=3.11 -y && conda activate whisperkit

3. Install the base package as editable from the root directory:

   cd WHISPERKIT_ROOT_DIR && pip install -e .

To evaluate models on your own dataset, follow these steps:

1. Publish a dataset to the Hugging Face Hub with a structure containing audio files and a metadata.json file.
2. Set the following environment variables:
   • CUSTOM_EVAL_DATASET: The name of your dataset on the Hub.
   • DATASET_REPO_OWNER: Your Hugging Face username or organization.
   • MODEL_REPO_ID: (Optional) The repository ID of the model you are evaluating.
3. Run the evaluation command:

export CUSTOM_EVAL_DATASET="my-dataset-name-on-hub"
export DATASET_REPO_OWNER="my-user-or-org-name-on-hub"
export MODEL_REPO_ID="my-org/my-whisper-repo-name"
whisperkit-evaluate-model --model-version <model-version> --output-dir <output-dir> --evaluation-dataset my-dataset-name-on-hub

To regenerate the support models used in the WhisperKitAndroid pipeline (input/output preprocessing and postprocessing), follow these steps:

1. Create an account at aihub.qualcomm.com.
2. Configure your Qualcomm AI Hub API key: qai-hub configure --api_token.
3. Install the required Android dependencies (Note: requires python<3.11):

   pip install -e '.[android]'

4. Execute the generation script:

   python tests/test_aihub.py --persistent-cache-dir <output-path>

You can publish your converted Whisper models to your own Hugging Face Hub repository. This allows them to be over-the-air deployable to apps using WhisperKit.

1. Ensure you have write access to your Hugging Face user or organization and are logged in via huggingface-cli.
2. Set the MODEL_REPO_ID environment variable to your target repository (e.g., my-org/my-whisper-repo-name).
3. Run the generation command specifying the source PyTorch repository in --model-version.

Example:

MODEL_REPO_ID=my-org/my-whisper-repo-name whisperkit-generate-model --model-version distil-whisper/distil-small.en --output-dir <output-dir>

Your model will be published to hf.co/<MODEL_REPO_ID>/<source-repo-name>.

If you encounter the error: xcrun: error: unable to find utility "coremlcompiler", not a developer tool or in PATH

Ensure Xcode is installed on your Mac and point the command line tools to the correct path by running:

sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer

The whisperkit.pipelines module provides a unified wrapper for several on-device and server-side Whisper frameworks, including WhisperKit, WhisperCpp, WhisperMLX, and WhisperOpenAIAPI.

First, install the pipeline dependencies:

pip install -e '.[pipelines]'

Usage Example

from whisperkit.pipelines import WhisperKit, WhisperCpp, WhisperMLX, WhisperOpenAIAPI

# Initialize the WhisperKit pipeline
pipe = WhisperKit(whisper_version="openai/whisper-large-v3", out_dir="/path/to/out/dir")

# Run inference on audio files
print(pipe("audio.{wav,flac,mp3}"))

Requirements

• WhisperCpp: Requires ffmpeg installed (e.g., brew install ffmpeg).
• WhisperOpenAIAPI: Requires the OPENAI_API_KEY environment variable to be set.

Evaluate Argmax-hosted or developer-published models on standard speech recognition datasets using the whisperkit-evaluate-model CLI.

First, install the evaluation dependencies:

pip install -e '.[evals,pipelines]'

Then run the evaluation:

whisperkit-evaluate-model --model-version <model-version> --output-dir <output-dir> --evaluation-dataset {librispeech-debug,librispeech,earnings22}

Supported datasets include librispeech-debug, librispeech, and earnings22.

whisperkit-evaluate-model --model-version <model-version> --output-dir <output-dir> --evaluation-dataset librispeech

Use the whisperkit-generate-model CLI tool to convert Hugging Face Whisper models (PyTorch) into the WhisperKit (Core ML) format for Apple devices.

whisperkit-generate-model --model-version <model-version> --output-dir <output-dir>

Run the command with -h to see optional arguments for model optimizations.