Moonshine Voice

Moonshine Voice uses an event-driven architecture to handle speech processing. Key concepts include:

• Transcriber: The primary object for converting audio input into text. Requires a path to downloaded models.
• MicTranscriber: A specialized helper class that automatically connects to the system's default microphone.
• Stream: A handler for audio input, allowing a single transcriber to process multiple audio sources simultaneously without duplicating model resources.
• TranscriptLine: A data structure representing a segment of speech (a phrase) identified by a pause. It includes state (started, speaking, or complete), start time, and duration.
• Transcript: A time-ordered list of TranscriptLine objects representing the recognized text of a session.
• TranscriptEvent: Contains information about changes to the transcript, such as a new line starting, text updating, or a line completing.
• TranscriptEventListener: A protocol/interface used to attach application logic to transcription events.
• TextToSpeech: An object used to synthesize audio for playback.
• DialogFlow: A specialized TranscriptEventListener that manages conversations and can trigger callbacks based on registered phrases (voice commands).
• Dialog: An object representing a single conversational exchange between an agent and a user.

Moonshine's language bindings are designed to provide a fluent, opinionated API focused on building voice interfaces. Developers should expect the following patterns when using the library:

• Opinionated Defaults: Common voice interface tasks are designed to be terse, often requiring only one or two function calls. Implementation details are abstracted away to simplify standard workflows.
• Builder Pattern: Configuration is typically handled via a series of method calls (builder pattern) rather than a single large options object.
• Platform Idioms: APIs are designed to follow the native idioms and patterns of the specific platform (e.g., Pythonic patterns in Python, idiomatic Java in Android).
• Asynchronous Resource Loading: High-latency operations like model loading follow native asynchronous patterns to prevent blocking.
• High-Level Abstractions: Complex workflows like DialogFlow and voice cloning are exposed as single high-level concepts. For example, DialogFlow manages the orchestration of STT, intent, and TTS models internally, while voice cloning handles audio extraction and transcription automatically.

Moonshine is optimized for live voice interfaces and streaming applications, whereas Whisper is better suited for bulk batch processing in the cloud.

Key advantages of Moonshine for live speech:

• Flexible Input Windows: Unlike Whisper's fixed 30-second window, Moonshine processes only the audio provided (recommended < 30s), avoiding zero-padding latency.
• Streaming Caching: Moonshine supports incremental audio addition by caching input encoding and decoder states, significantly reducing redundant computation.
• Language-Specific Accuracy: Moonshine offers specialized models for languages like Arabic, Japanese, Korean, Spanish, Ukrainian, Vietnamese, and Chinese, providing higher accuracy than multilingual models of similar size.
• Edge & Cross-Platform Support: Designed with a portable C++ core and OnnxRuntime, Moonshine provides consistent APIs across Linux, MacOS, Windows, iOS, Android, Python, Swift, Java, and C++.
• Efficiency: Moonshine models (e.g., Medium Streaming) can achieve lower Word Error Rates (WER) than Whisper Large v3 while using significantly fewer parameters (250M vs 1.5B), making them ideal for edge deployment.

The Moonshine Voice system uses a two-stage pipeline for on-device speech processing:

1. Always-on Listening (TinyVadCNN): Processes 16 kHz PCM audio using 32 ms hops. It consumes a log-mel stream to detect voice activity. When speech is detected, a segmenter identifies the speech boundaries.
2. On-speech Classification (SpellingCNN): Once speech ends, a ~1 second clip is captured and processed. This stage converts the audio into a log-mel batch, which is then classified by SpellingCNN to produce logits for spoken tokens (letters, digits, or commands).

Note: Both models consume normalised log-mel features. The .tflite and .onnx files provided contain only the classifier heads; the mel front-end (16 kHz PCM → Slaney filterbank → normalisation) must be implemented separately.

flowchart LR
  subgraph listen["Always-on (32 ms hops)"]
    PCM1["16 kHz PCM"] --> MelS["Log-mel stream"]
    MelS --> VAD["TinyVadCNN"]
    VAD --> Seg["Segmenter"]
  end

  subgraph classify["On speech end (~1 s clip)"]
    Seg --> Clip["1 s int16 clip"]
    Clip --> MelB["Log-mel batch"]
    MelB --> STT["SpellingCNN"]
    STT --> Logits["51 logits"]
    Logits --> TTS["TTS"]
  end

  TTS --> Spk["spoken reply"]

To transcribe audio, initialize a Transcriber with your model path and architecture, then attach a TranscriptEventListener to handle events. You can feed audio to the transcriber using add_audio() in a loop to simulate live streaming, or use transcribe_without_streaming() for processing existing audio arrays.

Note: The transcriber analyzes speech at a default interval of 500ms, which can be adjusted via the update_interval argument in the constructor. Always call start() to begin a session and stop() to end it.

# Initialize the transcriber
transcriber = Transcriber(model_path=model_path, model_arch=model_arch)

# Define a listener for events
class TestListener(TranscriptEventListener):
    def on_line_started(self, event):
        print(f"Line started: {event.line.text}")

def on_line_text_changed(self, event):
        print(f"Line text changed: {event.line.text}")

def on_line_completed(self, event):
        print(f"Line completed: {event.line.text}")

# Attach listener and start processing
transcriber.add_listener(listener)
transcriber.start()

# Example: Feeding audio chunks from a wav file
# (Assuming load_wav_file is available from the library)
audio_data, sample_rate = load_wav_file(wav_path)
chunk_duration = 0.1
chunk_size = int(chunk_duration * sample_rate)

for i in range(0, len(audio_data), chunk_size):
    chunk = audio_data[i: i + chunk_size]
    transcriber.add_audio(chunk, sample_rate)

transcriber.stop()