const-me/whisper

The core of this project is a C++ DLL that performs the heavy lifting for Whisper implementations. It provides the following capabilities:

• ML model implementation.
• Multimedia file handling via Media Foundation.
• Audio capture via Media Foundation.
• Voice Activity Detection (VAD) using custom CPU-based code.

The library is designed for 64-bit Windows and requires the following hardware/software support:

Platform & OS:

• 64-bit Windows (Windows 8.1 or newer recommended; tested on Windows 10).

GPU Requirements:

• Must support Direct3D 11.0 (most hardware from 2011 onwards, e.g., Intel Sandy Bridge or newer).
• Uses GPGPU via DirectCompute (compute shaders in Direct3D 11).

CPU Requirements:

• Must support AVX1 and F16C instructions.

The CompressShaders project is a C# console application used as a code generator for Whisper.dll and WhisperNet.dll. It automates the creation of C++ and C# source files from raw assets.

It performs two primary functions:

1. Shader Compression: Compresses compiled DXBC shaders into byte blobs and outputs them as std::array definitions within shaderData-Release.inl and shaderData-Debug.inl C++ files.
2. Language Data Generation: Parses the languageCodez.tsv file to generate corresponding C++ and C# code containing the language data.

Important Note on File Paths: This tool relies on relative paths across its source files. You must not move the tool's source code or its source data files, as doing so will break the path resolution and cause the generation process to fail.

The library includes integration for the RenderDoc GPU debugger.

Usage:

• Launch your program through RenderDoc.
• Press F12 to capture compute calls.
• Note: For better debugging experience with HLSL shaders, use the debug build of the DLL, which includes debug versions of the shaders.

To use the pre-built WhisperDesktop application:

1. Download WhisperDesktop.zip from the GitHub "Releases" section.
2. Unpack the ZIP file.
3. Run WhisperDesktop.exe.

Workflow:

• Model Loading: On the first screen, download a model. ggml-medium.bin (1.42GB) is recommended for testing.
• Transcription: Use the transcription screen to process existing audio files.
• Live Capture: Use the capture screen to transcribe or translate live audio from a microphone.

To compare the GPU and CPU implementations, you must enable debug tracing. By default, tracing is disabled via a preprocessor macro in stdafx.h of the main Whisper project to prevent large files from being generated.

Once enabled, the Whisper project generates two trace files at the following locations:

• GPU Implementation Trace: C:\Temp\2remove\Whisper\gpu.bin
• CPU (Reference) Implementation Trace: C:\Temp\2remove\Whisper\ref.bin

You can then use the compareTraces C++ console tool to compare these files. Note that the comparison tool is optimized for development speed and has the following requirements and behaviors:

• Hardware Requirement: Requires an AVX2 compatible CPU.
• Performance: Uses memory-mapped IO instead of formal parsing for speed.
• Error Handling: Performs minimal error checking.

The project provides two versions of many compute shaders to ensure compatibility across different hardware architectures:

• something64.hlsl: Use this version for AMD GPUs.
• something.hlsl (no 64 suffix): Use this version for nVidia and Intel GPUs.

To use the library, you must first load an ML model from a binary file using the Whisper.Library.loadModel function.

Models are available for download from the whisper.cpp Hugging Face repository.

Note on model files:

• ggml-medium.bin is the recommended model for testing.
• Compressed models in ZIP format (containing mlmodelc in the filename) are not supported. Use the raw .bin files.

To use WhisperNet, your environment must meet the following hardware and software requirements:

• OS: 64-bit Windows.
• Process: Must run within a 64-bit process.
• CPU: 64-bit CPU supporting AVX1 and F16C extensions.
• GPU: A hardware GPU that supports Direct3D 11.0.

To use the Whisper.dll library in your C++ application, you must include exactly one of the provided API headers depending on your target platform and compiler requirements.

For Windows Applications

Include whisperWindows.h. This provides a traditional Win32 COM projection of the API, which is compatible with Visual Studio's CComPtr<T> and related utilities.

For Cross-Platform or Non-MSVC Compilers

If you are porting to other operating systems (using GCC or Clang) or already using the ComLight support library, include whisperComLight.h.

Note: When using the whisperComLight.h header, you must also include the ComLightLib dependency. While the internal implementation uses the ComLight flavor of the interfaces, they remain binary compatible with the Windows projection.