DOCSEARCHbyAlgolia
Submit Docs

TTS Audio Suite

repository·main·Indexed 22 days ago

https://github.com/diodiogod/tts-audio-suite

A universal multi-engine TTS extension for ComfyUI (version 5.6.3) providing a unified architecture for Text-to-Speech, Voice Conversion, and Audio Editing. It supports engines including IndexTTS-2, ChatterBox, Chatterbox Multilingual TTS, F5-TTS, Higgs Audio 2, VibeVoice, and RVC. Key features include character voice management, SRT subtitle TTS support, and audio processing capabilities.

Tokens
153.3K
Snippets
320
Records
649
Agent score
77%

What's inside tts-audio-suite

  1. Overview of TTS Audio Suite

    main

    TTS Audio Suite is a universal multi-engine Text-to-Speech (TTS) extension for ComfyUI. It provides a unified interface for various audio tasks including:

    • Text-to-Speech (TTS): Support for numerous engines (e.g., F5-TTS, ChatterBox, Higgs Audio, MOSS-TTS, Fish Audio S2 Pro).
    • Voice Conversion (VC): Real-time voice conversion and integrated RVC (Real-time Voice Conversion) model training.
    • Audio Editing: Tools for speech editing, emotion control, and sound effect generation.
    • Subtitle Workflows: Capabilities to transcribe to SRT, rebuild subtitles from edited transcripts, or estimate SRT timing from plain text using advanced readability rules.
    • ASR (Automatic Speech Recognition): Integrated ASR capabilities (e.g., Qwen3-TTS, Granite ASR) for transcription and speaker attribution.

    The suite features a modular architecture designed for extensibility and runtime isolation to support diverse and potentially conflicting engine requirements.

  2. Overview of Qwen3-ASR models

    main

    Qwen3-ASR is a family of speech recognition models that support language identification and Automatic Speech Recognition (ASR) for 52 languages and dialects. It also includes a specialized forced-alignment model.

    Key Models

    • Qwen3-ASR-1.7B & Qwen3-ASR-0.6B: All-in-one models supporting 30 languages and 22 Chinese dialects (including Anhui, Dongbei, Fujian, etc.). They support both offline and streaming inference modes and can handle speech, singing voices, and songs with background music (BGM).
    • Qwen3-ForcedAligner-0.6B: A non-autoregressive (NAR) model designed for timestamp prediction of arbitrary units in up to 5 minutes of speech across 11 languages (Chinese, English, Cantonese, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish).

    Core Features

    • Unified Inference: Single models support both streaming and offline transcription.
    • High Throughput: The 0.6B version is optimized for efficiency, reaching up to 2000x throughput at a concurrency of 128.
    • Comprehensive Toolkit: Supports vLLM-based batch inference, asynchronous serving, and timestamp prediction.

  3. Use the Web Interface components

    main

    The project includes several modular JavaScript components for web-based interaction:

    • TTS Tag Editor: A modular editor (string_multiline_tag_editor.js) with support for syntax highlighting, font controls, and tag utilities.
    • Audio Analyzer: A suite of tools (audio_analyzer_*.js) for visualization, region selection, and node integration.
    • Voice Capture: A microphone recording UI (chatterbox_voice_capture.js).
    • Emotion Radar: An IndexTTS-2 radar chart visualization (index_tts_emotion_radar.js).
    • Model-Specific Widgets: Specialized widgets for Qwen3-TTS (qwen3_tts_widgets.js) and ASR SRT preset locking (asr_srt_preset_widgets.js).

  5. Compare TTS and Audio Engines

    main

    The tts-audio-suite provides a wide range of engines for Text-to-Speech (TTS), Speech-to-Text (ASR), Voice Conversion (VC), and Sound Effects generation. Use the following capability matrix to select an engine based on your requirements:

    Capability Key

    • TTS: Text-to-Speech
    • SRT: Subtitle/Timestamp generation
    • VC: Voice Conversion
    • ASR: Automatic Speech Recognition
    • Sound Effects: Text-to-Audio generation
    • Training: Integrated training workflows

    Engine Selection Guide

    EnginePrimary CapabilitiesKey FeaturesLicense
    F5-TTSTTS, SRTTargeted Word/Speech Editing, Speed controlCC-BY-NC-4.0
    ChatterBoxTTS, SRT, VCExpressiveness sliderMIT
    VibeVoiceTTS, SRT90-min long-form, 4-bit quantizationMIT (research-only)
    Higgs Audio v3TTS, SRTZero-shot voice cloning, 100+ languages, Inline emotion/style tagsResearch/Non-Commercial
    Qwen3-TTSTTS, SRT, ASRVoice design, ASR capabilitiesApache-2.0
    Granite ASRSRT, ASRDiarization, Word-level timestamps, Speech translationApache-2.0
    Fish Audio S2 ProTTS, SRTMulti-turn dialogue, Zero-shot cloning, Sub-word emotion tagsFish Audio Research
    DramaBoxTTS, SRTExpressive scene prompting, Duration targeting, Long-form chunkingLTX-2 Community
    OmniVoiceTTS, SRT600+ language support, Reference-free voice designApache-2.0
    MOSS-TTSTTS, SRT, ASR, SFXPrompt-only sound effects, Multi-speaker dialogue, LoRA trainingApache-2.0
    RVCVC, TrainingReal-time VC, Pitch shift (±14), Integrated trainingMIT (framework)

  7. Compare RVC features: Replay vs ComfyUI Custom Node

    main

    This document provides a feature comparison between the Replay RVC implementation and the ComfyUI RVC custom nodes. It is used to identify functional gaps when integrating RVC into the TTS suite.

    Key Feature Differences

    Available in both (or equivalent):

    • Audio Input: Select audio or use AUDIO/VHS_AUDIO inputs.
    • Voice Model: Model selection via RVC_MODEL input.
    • Pitch Shift: f0_up_key in ComfyUI (semitone steps) vs Pitch (+/-) in Replay.
    • Pitch Detection: crepe and mangio-crepe are available in both. rmvpe is available in both.
    • Advanced Settings: index_rate (0.0-1.0), protect (0.0-0.5), and rms_mix_rate (0.0-1.0) are functionally equivalent across both.
    • Audio Processing: De-echo/Reverb and Stemming are available in ComfyUI via a separate UVR5Node.

    ComfyUI Exclusives (Bonus Features):

    • rmvpe+ pitch extraction.
    • f0_autotune (boolean).
    • resample_sr options.
    • crepe_hop_length (16-512).
    • use_cache (boolean) for performance.

    Missing in ComfyUI (High Priority Gaps):

    • Instrumental Pitch Control: Ability to shift the backing track pitch independently during recombination.
    • Additional Pitch Detection Methods: PM, Harvest, FCPE, and Crepe Tiny.

  8. Status and Performance of Higgs Audio Stateless Wrapper

    main

    The Higgs Audio Stateless Wrapper implementation is designed to provide a stateless interface for audio models while managing VRAM and device consistency.

    Confirmed Capabilities

    • Voice cloning functionality: Full feature preservation is maintained.
    • Memory management: Successfully frees approximately 11GB of VRAM during unloading.
    • Device consistency: Prevents device mismatch crashes by ensuring tensors and models are synchronized across the pipeline.
    • Performance: Achieves high-speed inference at approximately 54.92 tok/s.

    Known Limitations and Risks

    • CUDA Graph cleanup: There is a known risk of crashes during CUDA Graph cleanup; a hybrid solution is currently implemented but requires verification.
    • Performance vs. Safety: There is an inherent tradeoff between maximizing performance (via CUDA Graphs) and ensuring safe model unloading/cleanup.

  9. Compare TTS and Audio Engine features

    main

    The tts-audio-suite includes a wide variety of engines with different capabilities. Use the following feature matrix to select an engine based on your requirements for Text-to-Speech (TTS), Speech-to-Text (ASR), Voice Conversion, or Sound Effects.

    Key Capability Summary

    • Text-to-Speech (TTS): Supported by most engines (F5-TTS, ChatterBox, VibeVoice, Higgs Audio, etc.).
    • Speech-to-Text (ASR/Transcribe): Supported by Qwen3-TTS and Granite ASR.
    • Voice Conversion: Supported by ChatterBox, ChatterBox 23L, CosyVoice3, and RVC.
    • Sound Effects: Supported by MOSS-TTS, MOSS-SoundEffect v2.
    • Voice Cloning Requirements:
      • F5-TTS: Requires a reference transcript.
      • Step Audio EditX: Requires a reference transcript.
      • Fish Audio S2 Pro: Requires reference audio plus an exact transcript.
      • Dots TTS: Requires reference audio.
      • OmniVoice: Reference audio is optional.
      • MOSS-TTS: Reference audio is optional.
    • Emotion Control:
      • Higgs Audio v3: Supports 8 emotions natively via inline tags.
      • Step Audio EditX: Supports 14 emotions.
      • Fish Audio S2 Pro: Supports free-form inline natural-language tags.
      • OmniVoice: Supports natural-language scene prompts and stage directions.
    • Long-form Audio:
      • Fish Audio S2 Pro: Configurable 4K-32K native context.
      • Dots TTS: Uses official duration-aware quote-group chunking (~37s target / 45s cap).
      • MOSS-TTS: Uses audio_chunk_duration / audio_chunk_threshold orchestration.

  10. Overview of Migrated TTS, VC, and SRT Nodes

    main

    The core functionality of the suite is provided through three main node types. While the internal architecture has been refactored to use ModelManager, ImportManager, and BaseTTSNode, the user-facing capabilities remain the same:

    • ChatterboxTTSNode (nodes/tts_node.py): Handles text-to-speech. It includes features for text chunking (via ImprovedChatterBoxChunker), audio combination, and caching.
    • ChatterboxVCNode (nodes/vc_node.py): Handles voice conversion. It features improved error handling and temporary file management.
    • ChatterboxSRTTTSNode (nodes/srt_tts_node.py): Handles SRT-based text-to-speech. It supports complex SRT timing, smart/natural modes, and caching.

  11. What is FACodec and how does it work?

    main

    FACodec is a speech codec used in NaturalSpeech 3 that performs attribute factorization. It decomposes complex speech waveforms into disentangled subspaces representing specific speech attributes: content, prosody, timbre, and acoustic details.

    This decomposition allows researchers to build various TTS architectures, such as:

    • Non-autoregressive discrete diffusion models (e.g., NaturalSpeech 3) using the latents after quantization.
    • Autoregressive models (e.g., VALL-E) by generating discrete tokens (prosody codes followed by content/acoustic codes).

    Technical Specifications:

    • Supported Sample Rate: 16kHz.
    • Hop Size: 200 samples.
    • Code Generation: Generates 6 codes per frame (based on the total number of codebooks).

  12. Overview of IndexTTS-2 Engine

    main
    IndexTTS-2 is an autoregressive zero-shot TTS engine designed for high emotional expressivity and precise duration control. It features emotion disentanglement, allowing the separation of speaker identity from emotional expression, and supports multi-modal emotion control via audio references, text descriptions, or manual vectors. It also supports zero-shot voice cloning using a single reference audio file.