transcribe-anything

transcribe-anything

transcribe-anything is a Python CLI tool and library that provides a unified interface for transcribing audio/video content using multiple Whisper AI backends. It supports GPU acceleration, speaker diarization, and multi-platform optimization (Mac/Linux/Win).

Key Features:

• Multiple Whisper backends: CPU, CUDA, "insane" (GPU-accelerated), WhisperX, and MLX (Mac Apple Silicon).
• Speaker diarization: Generates speaker.json output on supported backends.
• Flexible Inputs: Supports local files and online URLs (YouTube, Rumble, etc.).
• Subtitle Support: Embedding subtitles into video files.
• Customization: Custom vocabulary support via initial prompts.
• Deployment: Docker containerization with GPU support and isolated environment management.

When using the --live mode, you can select a latency profile to balance speed and accuracy.

To achieve the lowest latency, use beam_size=1 (greedy decoding) and smaller models like tiny or base.

To manage complex and conflicting AI dependencies (like different versions of PyTorch or Whisper), transcribe-anything uses an isolated environment pattern via uv-iso-env:

1. Parent Environment: Contains the main CLI and coordination logic.
2. Backend Environments: Each Whisper backend (e.g., whisper, whisper_mlx) resides in its own isolated Python environment.
3. Communication: The parent environment communicates with backend environments using subprocess-based IPC.

This ensures that installing a new backend or updating a dependency for one backend does not break the others.

The application uses a single CLI and a router mechanism to manage different transcription backends. When you run a command, the --device flag (or auto-detection) routes the task to the appropriate backend.

To prevent dependency conflicts (like CUDA or Torch version mismatches), each backend runs in its own isolated environment (using uv virtual environments) which is built automatically on the first use.

Workflow:

1. Input: Accepts local files, YouTube links, or any URL.
2. Fetch: Uses yt-dlp and static-ffmpeg to normalize audio to 16-bit WAV.
3. Router: Selects the backend via --device.
4. Execution: Runs the transcription in an isolated environment.
5. Output: Generates out.txt, out.srt, out.vtt, out.json, and speaker.json (containing speaker partitions).

Choose a backend based on your hardware and specific needs (alignment, diarization, or speed).

• MLX (--device mlx): Optimized for Mac Apple Silicon. Focused feature set.
• Insanely Fast (--device insane): Best for Windows/Linux GPUs. Uses a transformer-based architecture.
• Insane Flash (--device insane-flash): For CUDA GPUs with verified FlashAttention2. Use this instead of manually passing --flash True to ensure correct setup.
• WhisperX (--device whisperx): Best for alignment, diarization, and word-level timing. An additive backend, not a replacement for insane.
• SenseVoice (--device sensevoice): Multilingual (zh/en/yue/ja/ko) with built-in VAD and emotion detection. Uses FunASR. Use --hub hf to use HuggingFace instead of ModelScope.
• CPU (--device cpu): Universal compatibility; supports full OpenAI Whisper arguments.

Live transcription follows a producer-consumer pattern to ensure real-time performance. The flow is as follows:

1. Hotkey Press: Triggers the process.
2. Permission Check: Verifies microphone access.
3. Audio Capture: A dedicated thread captures audio from the microphone into an Audio Buffer.
4. VAD Detection: A Voice Activity Detector (Silero VAD) monitors the buffer. If speech is detected, audio is moved to an Audio Queue.
5. Transcription: A transcription thread consumes the queue, sending audio to the Whisper Backend (e.g., faster-whisper).
6. Output: The resulting text is sent to stdout or a file.

The project uses a unique architecture where complex AI model dependencies are siloed into separate environments using uv-iso-env pyproject.toml files for each backend.

Key architectural constraints:

• Communication: The parent environment and backend environments communicate exclusively via subprocess.Popen. You cannot call functions in a sub-environment directly from the parent environment.
• Backend Isolation: Each backend (e.g., WhisperX, Insanely Fast Whisper) maintains its own virtual environment (e.g., venv/insanely_fast_whisper).
• FlashAttention: The --device insane-flash backend uses a specific environment (venv/insanely_fast_whisper_flash) that forces and verifies FlashAttention2. This dependency is pinned via direct wheel URL and sha256 in src/transcribe_anything/flash_attention_wheels.py to ensure stability.

To ensure consistent environment generation and avoid repeated 2.2GB+ torch downloads, transcribe-anything uses a caching mechanism for NVIDIA detection.

How it works:

1. The system generates a fingerprint based on the platform (Windows, Linux, Darwin), machine architecture (AMD64, x86_64, arm64), platform version, and the existence of the nvidia-smi executable.
2. The result of the has_nvidia_smi() check is stored in ~/.transcribe_anything_nvidia_cache.json mapped to that fingerprint.
3. Subsequent runs use the cached result if the fingerprint matches, ensuring the generated pyproject.toml remains identical and avoids triggering a uv-iso-env reinstall.

Cache File Format:

{
  "Windows-AMD64-10.0.19041-nvidia_smi:true": true,
  "Linux-x86_64-5.4.0-nvidia_smi:false": false
}

To speed up processing, increase the --batch_size (if GPU memory permits), enable Flash Attention 2 for insane mode, use smaller models, or use distilled models.

# Increase batch size
transcribe-anything video.mp4 --device mlx --batch_size 24
transcribe-anything video.mp4 --device insane --batch-size 16
transcribe-anything video.mp4 --device whisperx --batch_size 16

# Enable Flash Attention 2 for insane mode
transcribe-anything video.mp4 --device insane --flash True --batch-size 16

# Use smaller model for speed
transcribe-anything video.mp4 --device insane --model small

# Use distilled models for even faster processing
transcribe-anything video.mp4 --device insane --model distil-whisper/large-v2 --flash True