DOCSEARCHbyAlgolia
Submit Docs

FluidAudio

repository·main·Indexed 25 days ago

https://github.com/fluidinference/fluidaudio

A Swift SDK for high-performance, local audio AI on Apple platforms (macOS and iOS). It optimizes CoreML models to run on the Apple Neural Engine (ANE) to provide Automatic Speech Recognition (ASR), Text-to-Speech (TTS) via Kokoro and PocketTTS, speaker diarization, speaker embedding extraction, and Voice Activity Detection (VAD). It includes official wrappers for React Native, Expo, and Rust/Tauri.

Tokens
121.7K
Snippets
253
Records
495
Agent score
82%

What's inside FluidAudio

  1. Overview of Sortformer Streaming Speaker Diarization

    main

    Sortformer is an end-to-end neural speaker diarization model designed for real-time "who spoke when" identification. It outputs frame-level speaker probabilities for 4 fixed speaker slots, eliminating the need for separate VAD, segmentation, or clustering stages.

    Key Technical Specifications:

    • Speaker Limit: Exactly 4 fixed speaker slots. It cannot handle 5 or more speakers (it will merge or miss them).
    • Resolution: ~80ms frame resolution (derived from 8x subsampling of 10ms mel frames).
    • Optimization: CoreML-optimized for Apple Silicon.
    • License: NVIDIA Open Model License (no restrictions).

    Known Limitations:

    • No Speaker Persistence: It does not remember speakers across different recordings; it has no persistent speaker embeddings.
    • Sensitivity: It is trained to ignore background conversations, which means it may miss quiet or distant speech.

  2. Overview of FluidAudio capabilities

    main

    FluidAudio is a Swift SDK designed for fully local, low-latency audio AI on Apple devices (macOS and iOS). It leverages the Apple Neural Engine (ANE) to offload inference, which minimizes CPU usage, reduces memory footprint, and avoids the use of GPU/MPS.

    Key capabilities include:

    • Automatic Speech Recognition (ASR): Supports batch transcription (e.g., Parakeet TDT v3) and streaming transcription with end-of-utterance detection (e.g., Parakeet EOU).
    • Text-to-Speech (TTS): Includes Kokoro for parallel synthesis with SSML support and PocketTTS for streaming TTS with voice cloning.
    • Speaker Diarization: Provides both online (streaming) and offline (batch) pipelines for speaker separation and identification.
    • Speaker Embedding Extraction: Generates embeddings for voice comparison and clustering.
    • Voice Activity Detection (VAD): Uses Silero models to detect active speech.
    • Inverse Text Normalization (ITN): Post-processes ASR output to convert spoken forms to written forms (e.g., "two hundred" to "200").

  3. Overview of LS-EEND Streaming Speaker Diarization

    main

    LS-EEND (Long-Form Streaming End-to-End Neural Diarization) is a real-time speaker diarization system that identifies "who spoke when" without requiring separate Voice Activity Detection (VAD), segmentation, or clustering. It uses a causal Conformer encoder with a retention mechanism and an online attractor decoder to track speaker identities frame by frame.

    Key Specifications:

    • Speaker Capacity: Supports 4–10 simultaneous speakers depending on the selected model variant.
    • Resolution: ~100ms frame resolution (10 Hz output) at the default step size.
    • Duration: Capable of handling recordings up to one hour.
    • Audio Requirements: Input is automatically resampled to 8000 Hz.
    • Streaming Model: Uses a chunk-in-chunk-out approach where each CoreML call emits one step of committed frames.
    • Optimization: CoreML-optimized for Apple Silicon (CPU-only execution is recommended for maximum speed).

    Limitations:

    • Fidelity: The 8000 Hz sample rate provides lower audio fidelity compared to 16 kHz models.
    • Identity Persistence: Speaker identity is local to the specific recording; persistent speaker enrollment across different sessions may be unreliable.
    • Domain Sensitivity: Accuracy is highly dependent on choosing the correct model variant for your specific audio domain.

  4. Use Nemotron Speech Streaming Multilingual 0.6B for ASR

    main

    FluidAudio supports NVIDIA's nemotron-asr-streaming-multilingual-0.6b for real-time streaming Automatic Speech Recognition (ASR) across approximately 40 languages on Apple Silicon.

    Key Features

    • Multilingual Support: Supports ~40 languages (e.g., en, es, de, fr, it, pt, ar, ja, ko, zh-CN, ru, hi, vi).
    • Language Detection: The model emits a leading <xx-XX> language-tag token which is automatically filtered from the transcript and can be retrieved via detectedLanguage().
    • Latency Tiers: Available in three distinct CoreML builds optimized for different latency/accuracy trade-offs: 560 ms, 1120 ms, and 2240 ms.
    • Hardware Optimization: Specifically targeted for Apple Silicon (ANE) using an int8 encoder.

    Model Requirements

    • Hardware: Apple Silicon only.
    • Audio Input: 16 kHz mono audio.
    • Model Files: Requires a directory containing preprocessor.mlmodelc (or .mlpackage), encoder.mlmodelc, decoder.mlmodelc, joint.mlmodelc, metadata.json, and tokenizer.json.

  5. Use Kokoro ANE for high-performance TTS

    main

    The KokoroAneManager implements a 7-stage Kokoro 82M model optimized for Apple Silicon. It splits the model into stages so that ANE-friendly layers (Albert, PostAlbert, Alignment, and Vocoder) stay on the Neural Engine, while Prosody, Noise, and Tail run on CPU/GPU. This architecture yields 3-11× Real-Time Factor (RTFx) on Apple Silicon.

    Key Constraints

    • Languages: English (.english) and Mandarin (.mandarin).
    • Input Length: Maximum of 510 IPA or Bopomofo phonemes per utterance. There is no built-in chunker; split long text upstream.
    • Voices: Only one voice is supported per variant (af_heart for English, zf_001 for Mandarin).
    • Features: No SSML, no custom lexicon, and no Markdown overrides.
    • Compute: 4 stages on ANE, 3 on GPU/CPU.
    // English (default)
let english = KokoroAneManager(variant: .english)

// Mandarin
let mandarin = KokoroAneManager(variant: .mandarin)

  6. What is SpeakerManager and how to configure it

    main

    The SpeakerManager tracks and manages speaker identities across audio chunks for streaming diarization. It maintains an in-memory database of speakers and their voice embeddings to ensure consistent IDs across audio segments.

    Compatibility Note: SpeakerManager is only compatible with DiarizerManager (the streaming pipeline). It is not supported with OfflineDiarizerManager, which uses VBx clustering instead.

    To initialize SpeakerManager, you can configure thresholds for matching and updating speaker profiles:

    let speakerManager = SpeakerManager(
    speakerThreshold: 0.65,           // Max cosine distance for speaker match
    embeddingThreshold: 0.45,         // Max distance for embedding updates
    minSpeechDuration: 1.0,           // Min seconds to create new speaker
    minEmbeddingUpdateDuration: 2.0   // Min seconds to update embeddings
)

  7. What is Text Processing in FluidAudio?

    main

    FluidAudio uses the text-processing-rs library (a Rust port of NVIDIA NeMo Text Processing) to provide two types of text transformation:

    1. Inverse Text Normalization (ITN): Converts spoken-form ASR output into written form (e.g., converting "two hundred" to "200"). This is used for post-processing ASR transcriptions.
    2. Text Normalization (TN): Converts written-form text into spoken form (e.g., converting "$5.50" to "five dollars and fifty cents"). This is used for preprocessing text for Text-to-Speech (TTS).

    Supported languages include EN, DE, ES, FR, HI, JA, and ZH. The feature is optional; if the native library is not linked, the normalize() method will return the input string unchanged.

  8. Important: Model Parameters are Static

    main

    Because Sortformer uses CoreML, the input tensor dimensions for the FIFO queue, speaker cache, and audio chunks are fixed at the time of model conversion.

    You cannot change the following parameters at runtime:

    • fifoLen
    • spkcacheLen
    • chunkLeftContext (LC)
    • chunkRightContext (RC)

    To use a different configuration (e.g., switching from 'Default' to 'High Context'), you must load a different .mlpackage or .mlmodel file that was specifically converted with those dimensions.

    # During model conversion (Python)
fifo_len = 40        # Fixed in model
spkcache_len = 188   # Fixed in model
chunk_mel_frames = (chunk_len + lc + rc) * 8  # Fixed in model

  9. Paraformer Model Architecture and Precision

    main

    Paraformer uses a non-autoregressive architecture consisting of a SANM encoder, a CIF (Continuous Integrate-and-Fire) predictor, and a parallel decoder. The process is split across CPU and the Apple Neural Engine (ANE).

    Pipeline Flow

    waveform → Preprocessor (FP32/CPU) → Encoder (FP16 or INT8/ANE) → CifAlphas (FP16/ANE) → Host Integrate-and-Fire Loop → Decoder (FP16 or INT8/ANE) → CharTokenizer

    Model Components

    ComponentPrecisionCompute UnitRole
    ParaformerPreprocessor.mlmodelcFP32CPUwaveform → 560-d LFR features
    ParaformerEncoder.mlmodelc / _int8FP16 / INT8ANESANM encoder
    ParaformerCifAlphas.mlmodelcFP16ANEenc_out → per-frame alphas
    ParaformerDecoder.mlmodelc / _int8FP16 / INT8ANEparallel decoder → token logits
    vocab.json——8404 CharTokenizer tokens

    Note: The preprocessor must run on the CPU in FP32 because power-spectrum and log operations exceed the FP16 range.

  10. Choose the right speaker diarization model

    main

    FluidAudio provides several diarization models depending on your environment, speaker count, and whether you are processing audio in real-time or in batches.

    Model Comparison

    ModelBest Use CaseMax SpeakersKey Characteristics
    LS-EENDNoisy environments, overlapping speech, whispers10Lightweight (runs on CPU); prone to false alarms; streaming updates every 100ms.
    SortformerClean/silent rooms, high stability, pre-enrolled speakers4Extremely stable speaker identities; struggles with many loud voices or quiet speech; streaming updates every 480ms.
    DiarizerManagerLegacy online diarizationNo maxComputationally heavy; struggles with noise, overlap, and short utterances; prone to incorrect labeling.
    Offline VBx pipelineHigh-quality batch processingN/ABest for complete files; uses segmentation, embedding extraction, and clustering.

    Speaker Enrollment Considerations

    If your workflow requires pre-enrolling known speakers before live audio starts:

    • Use Sortformer: It is the stronger choice for enrollment. It auto-maps speakers with high confidence using a speaker cache and handles similar voices effectively.
    • Avoid LS-EEND for strict enrollment: It is an end-to-end model and lacks an API for per-slot similarity or explicit slot-lock assignment. It may fail to enroll speakers if their voices are too similar to existing ones (collisions).

  11. Use Sliding-Window ASR for near real-time transcription

    main

    For long-form audio processing that requires near real-time results, use the SlidingWindowAsrManager. This manager processes audio by chunking it (approximately 15s segments with overlap) and then stitching the resulting transcripts together.

    Supported models for this pipeline include:

    • Parakeet TDT v3: The default multilingual ASR model (25 European languages).
    • Parakeet TDT-CTC-110M: A smaller, faster, iOS-compatible alternative with a reduced memory footprint.
    • Parakeet TDT Japanese: Optimized for Japanese speech-to-text.
    • Cohere Transcribe: A batch encoder-decoder model supporting 14 languages. Note that language must be passed explicitly via the conditioned prompt and there is a 35s audio cap per call.
    • SenseVoiceSmall (FunASR): A fast, non-autoregressive multilingual model (50+ languages) managed by SenseVoiceManager. Language is auto-detected by default.
    • Paraformer-large (zh): A non-autoregressive model specifically for Mandarin Chinese, managed by ParaformerManager.

  12. Fixing seam word duplicates and casing issues

    main

    When chunks are split mid-sentence, decoders may incorrectly capitalize the first word of a new chunk, leading to duplicates like ...the meeting Meeting was.... FluidAudio uses two mechanisms to resolve this during the merge process:

    • Case-folded overlap matching: Uses caseVariantCanonicalIds to map case-only token twins to a single canonical ID. This allows tokenIdsMatch to align them, causing the duplicate to collapse into the contextually correct casing of the left window.
    • collapseSeamWordDuplicates: Reconstructs SentencePiece words from the token stream and drops adjacent case-only duplicates within the overlap window. This is designed to handle small subword vocabularies where a single word spans multiple tokens.