whisper.rn

The RingBuffer is a fixed-size circular buffer designed for managing audio data in real-time transcription scenarios. It is particularly useful for pre-recording audio, allowing you to maintain a fixed memory footprint while keeping only the most recent $N$ seconds of audio before speech is detected.

Key characteristics:

• Fixed Memory: Allocates a set amount of memory upfront and does not grow unbounded.
• O(1) Writes: Writing data is highly efficient.
• Overwrite Behavior: When the buffer reaches its capacity, the oldest data is automatically overwritten by new incoming data, ensuring you always have the most recent audio available.

// Example: Creating a buffer for 5 seconds of audio (assuming 16kHz, 16-bit mono)
// 16000 samples/sec * 2 bytes/sample * 5 seconds = 160,000 bytes
const ringBuffer = new RingBuffer(160000);

// Writing data
ringBuffer.write(audioChunk);

// Reading the most recent data in order
const audioToProcess = ringBuffer.read();

The RealtimeTranscriber provides enhanced realtime transcription using VAD, auto-slicing, and memory management. It requires an audio stream adapter and a filesystem module.

Dependencies:

• @fugood/react-native-audio-pcm-stream (for AudioPcmStreamAdapter)
• A compatible filesystem module (e.g., react-native-fs)

import { RealtimeTranscriber } from 'whisper.rn/realtime-transcription'
import { AudioPcmStreamAdapter } from 'whisper.rn/realtime-transcription/adapters'
import RNFS from 'react-native-fs'

const whisperContext = await initWhisper({ /* ... */ })
const vadContext = await initWhisperVad({ /* ... */ })
const audioStream = new AudioPcmStreamAdapter()

const transcriber = new RealtimeTranscriber(
  { whisperContext, vadContext, audioStream, fs: RNFS },
  {
    audioSliceSec: 30,
    vadPreset: 'default',
    autoSliceOnSpeechEnd: true,
    transcribeOptions: { language: 'en' },
  },
  {
    onTranscribe: (event) => console.log('Transcription:', event.data?.result),
    onVad: (event) => console.log('VAD:', event.type, event.confidence),
    onStatusChange: (isActive) => console.log('Status:', isActive),
    onError: (error) => console.error('Error:', error),
  },
)

await transcriber.start()
await transcriber.stop()

Using Parakeet for Realtime: Provide parakeetContext instead of whisperContext. Note that initialPrompt and promptPreviousSlices are Whisper-only and will be ignored. Parakeet audio must be mono, 16 kHz, signed 16-bit PCM.

To use Core ML on iOS (iOS 15.0+, tvOS 15.0+), you must provide both the .ggml model file and the corresponding .mlmodelc Core ML model files. The .mlmodelc is a directory containing specific files required for the model to run.

Important Notes:

• The .ggml model is still required as a fallback for the decoder or encoder.
• The .mlmodelc directory typically contains 3 required files: model.mil, coremldata.bin, and weights/weight.bin.
• Core ML models can be downloaded from Hugging Face.
• If downloading at runtime, you must unzip the archive to access the .mlmodelc directory (e.g., using react-native-zip-archive).
• To avoid bloating the Android bundle, it is recommended to move Core ML asset configurations into a platform-specific file like context-opts.ios.js.

const whisperContext = await initWhisper({
  filePath: require('../assets/ggml-tiny.en.bin'),
  coreMLModelAsset: Platform.OS === 'ios' ? {
    filename: 'ggml-tiny.en-encoder.mlmodelc',
    assets: [
      require('../assets/ggml-tiny.en-encoder.mlmodelc/weights/weight.bin'),
      require('../assets/ggml-tiny.en-encoder.mlmodelc/model.mil'),
      require('../assets/ggml-tiny.en-encoder.mlmodelc/coremldata.bin'),
    ],
  } : undefined,
})

Install the package using npm:

npm install whisper.rn

iOS Setup

• Run npx pod-install after installation.
• By default, it uses a pre-built rnwhisper.xcframework. To build from source, set RNWHISPER_BUILD_FROM_SOURCE=1 in your Podfile.
• For medium or large models, it is recommended to enable the Extended Virtual Addressing capability in your iOS project.

Android Setup

• If Proguard is enabled, add the following rule to android/app/proguard-rules.pro:

# whisper.rn
-keep class com.rnwhisper.** { *; }

• It is recommended to use ndkVersion = "24.0.8215888" (or above) in your root project build configuration, especially for Apple Silicon Macs.

Expo Setup

• You must prebuild the project before using this library.

When integrating whisper.rn into a project, you may need to prevent multiple versions of peer dependencies from being loaded by Metro. This is achieved by:

1. Adding the peer dependency paths to the resolver.blockList to block them at the project root.
2. Adding the same dependencies to resolver.extraNodeModules to alias them to the versions located in the local node_modules.

This pattern ensures that only one version of a dependency is loaded, preventing runtime conflicts.

// Example pattern for resolving peer dependencies
const modules = Object.keys({ ...pak.peerDependencies });

const config = {
  resolver: {
    blockList: metroExclusionList(
      modules.map(
        (m) => new RegExp(`^${escape(path.join(root, 'node_modules', m))}\\/.*$`),
      ),
    ),
    extraNodeModules: modules.reduce((acc, name) => {
      acc[name] = path.join(__dirname, 'node_modules', name);
      return acc;
    }, {}),
  },
};