FlashTTS Documentation

repository·master·Indexed 20 days ago

https://github.com/huiresearch/flashtts

A text-to-speech (TTS) project supporting multiple models including Mega, Orpheus, and Spark. FlashTTS provides local command-line inference via `flashtts infer` and server-based API deployment using `flashtts serve`. It supports various high-performance inference engines such as vLLM, sglang, llama-cpp, mlx-lm, tensorrt-llm, and torch across Linux, Windows, and macOS.

Tokens
18.6K
Snippets
45
Records
77
Agent score
69%

What's inside FlashTTS

  1. Overview of FlashTTS documentation structure

    master

    FlashTTS documentation is organized into three primary functional areas:

    1. Getting Started (get_started): Covers installation, dependency management, and basic setup.
    2. Model Inference (inference): Provides detailed guides for specific TTS models (Mega TTS, Orpheus TTS, Spark TTS) and the automatic inference engine.
    3. Server Deployment (server): Instructions for deploying the service, managing the runtime environment, and using the client to interact with model APIs.
  2. Understand differences between Spark, Orpheus, and Mega engines

    master

    While AutoEngine unifies the interface, the underlying engines have different capabilities:

    FeatureSparkOrpheusMega
    Voice CloningSupportedNot SupportedSupported (requires WaveVAE)
    StreamingFull supportSupportedNot explicitly mentioned
    Multi-speakerSupportedSupportedSupported
    Special FeaturesAcoustic token reuse, pitch/speed controlLong-form synthesis with emotional tagsMulti-speaker dialogue focus

    Performance Tip: Adjust llm_device, backend, torch_dtype, and memory usage parameters to optimize for your specific hardware.

  3. Reuse acoustic tokens

    master

    Acoustic tokens can be generated, saved, and reused to optimize synthesis workflows.

    1. Generate and Save: Call speak_async with return_acoustic_tokens=True to get the tokens, then use tokens.save("filename.txt").
    2. Load and Reuse: Use SparkAcousticTokens.load("filename.txt") to retrieve the tokens, then pass them to speak_async via the acoustic_tokens parameter.
    # 1. Generate and save
    wav, tokens = asyncio.run(
        engine.speak_async(..., return_acoustic_tokens=True)
    )
    tokens.save("acoustic_tokens.txt")
    
    # 2. Reuse
    from flashtts import SparkAcousticTokens
    tokens = SparkAcousticTokens.load("acoustic_tokens.txt")
    wav2 = asyncio.run(
        engine.speak_async(..., acoustic_tokens=tokens)
    )
  4. Install Flash-TTS

    master

    Flash-TTS requires Python 3.10 or above and a supported operating system (Linux x86_64, macOS, or Windows via WSL2).

    1. Install PyTorch

    Install PyTorch and torchaudio compatible with your CUDA version. For CUDA 12.4:

    pip install torch==2.6.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124

    2. Install the Package

    You can install via pip or from source:

    Via pip:

    pip install flashtts

    From source:

    git clone https://github.com/HuiResearch/FlashTTS.git
    cd FlashTTS
    pip install .

    Windows Troubleshooting

    If you encounter compilation errors with WeTextProcessing, use Conda to install dependencies first:

    conda install -c conda-forge pynini==2.1.6
    pip install WeTextProcessing==1.0.4.1
  5. Initialize and use AutoEngine for unified TTS

    master

    Use AutoEngine to automatically select and manage between SparkTTS, OrpheusTTS, or MegaTTS based on your model directory structure. It provides a unified interface for speech synthesis, voice cloning, and multi-speaker dialogue across different model architectures.

    Engine Auto-Detection Logic

    AutoEngine identifies the engine by checking for specific subdirectories in your model_path:

    • Spark: Requires LLM, BiCodec, and wav2vec2-large-xlsr-53 directories.
    • Mega: Requires aligner_lm, diffusion_transformer, duration_lm, g2p, and wavvae directories.
    • Orpheus: Detected if a snac directory exists or if snac_path is explicitly provided.

    If none of these patterns match, a RuntimeError: No engine found is raised.

    import asyncio
    from flashtts import AutoEngine
    
    # AutoEngine automatically detects if this is Spark, Orpheus, or Mega
    engine = AutoEngine(
        model_path="checkpoints/YourModelDir",
        snac_path=None,  # Required for Orpheus if `snac` subdirectory is not present
        lang="mandarin",  # Only used by Orpheus
        llm_device="cuda",
        tokenizer_device="cuda",
        detokenizer_device="cuda",
        backend="vllm",
        torch_dtype="float32",
        batch_size=1,
        llm_batch_size=256,
        wait_timeout=0.01,
        seed=42
    )
    
    async def main():
        # Basic synthesis example
        wav = await engine.speak_async(
            text="Hello, world!",
            name="female",
            pitch="moderate",
            speed="moderate",
            temperature=0.9,
            top_k=50,
            top_p=0.95
        )
        engine.write_audio(wav, "output.wav")
    
    if __name__ == '__main__':
        asyncio.run(main())
  6. Install dependencies for Flash-TTS Python Client

    master

    To use the Flash-TTS service via Python, install the following required libraries:

    pip install requests pyaudio openai
    pip install requests pyaudio openai
  7. Initialize the AsyncSparkEngine

    master

    To use Spark-TTS, instantiate the AsyncSparkEngine class. You must specify the model_path and the devices for the LLM, tokenizer, and detokenizer.

    Important Note on Data Types: Spark-TTS does not support float16. You must use torch_dtype="bfloat16" or torch_dtype="float32".

    Supported backend options include: torch, vllm, sglang, llama-cpp, and mlx-lm.

    from flashtts import AsyncSparkEngine
    
    engine = AsyncSparkEngine(
        model_path="checkpoints/Spark-TTS-0.5B",
        llm_device="cuda",
        tokenizer_device="cuda",
        detokenizer_device="cuda",
        backend="vllm",  # Can be torch, vllm, sglang, llama-cpp, mlx-lm
        torch_dtype="bfloat16"  # Note: Spark-TTS does not support float16, use float32 or bfloat16
    )
  8. Deploy FlashTTS as a FastAPI server

    master

    Deploy the service using flashtts serve to expose a Web interface and API. The service is built on FastAPI and supports various hardware acceleration backends and device assignments.

    ```bash
    flashtts serve \
     --model_path Spark-TTS-0.5B \\
     --backend vllm \\
     --role_dir data/roles \\
     --llm_device cuda \\
     --tokenizer_device cuda \\
     --detokenizer_device cuda \\
     --wav2vec_attn_implementation sdpa \\
     --llm_attn_implementation sdpa \\
     --torch_dtype "bfloat16" \\
     --max_length 32768 \\
     --llm_gpu_memory_utilization 0.6 \\
     --fix_voice \\
     --host 0.0.0.0 \\
     --port 8000
    • Web Interface: http://localhost:8000
    • API Documentation: http://localhost:8000/docs
  9. Download Flash-TTS Model Weights

    master

    Flash-TTS supports several models. You can download them from Hugging Face, ModelScope, or use GGUF formats where available.

    ModelHugging FaceModelScopeGGUF
    Spark-TTSSparkAudio/Spark-TTS-0.5BSparkAudio/Spark-TTS-0.5BSparkTTS-LLM-GGUF
    Orpheus-TTScanopylabs/orpheus-3b-0.1-ft & hubertsiuzdak/snac_24khzcanopylabs/orpheus-3b-0.1-ftorpheus-gguf
    Orpheus-TTS (Multilingual)orpheus-multilingual-research-release & hubertsiuzdak/snac_24khz--
    MegaTTS3ByteDance/MegaTTS3--
  10. Start the Flash-TTS Server

    master

    You can start the Flash-TTS backend using the flashtts serve command. The command varies depending on which model you are using (Spark-TTS, MegaTTS3, or Orpheus).

    Common backend options: vllm, sglang, torch, llama-cpp, mlx-lm, tensorrt-llm.

    Accessing the server:

    • Web Interface: http://localhost:8000
    • API Documentation (Swagger): http://localhost:8000/docs
    # Example for Spark-TTS
    flashtts serve \
    --model_path Spark-TTS-0.5B \
    --backend vllm \
    --llm_device cuda \
    --tokenizer_device cuda \
    --detokenizer_device cuda \
    --wav2vec_attn_implementation sdpa \
    --llm_attn_implementation sdpa \
    --torch_dtype "bfloat16" \
    --max_length 32768 \
    --llm_gpu_memory_utilization 0.6 \
    --fix_voice \
    --host 0.0.0.0 \
    --port 8000
  11. Deploy the FlashTTS service

    master

    Deploy FlashTTS as a web service using the flashtts serve command. This exposes a FastAPI-based server with a web interface and API documentation.

    Commonly used flags include:

    • --model_path: Path to the model weights.
    • --backend: Inference engine (e.g., vllm).
    • --role_dir: Directory containing role data.
    • --llm_device / --tokenizer_device / --detokenizer_device: Hardware device for specific components (e.g., cuda).
    • --torch_dtype: Precision type (e.g., bfloat16).
    • --fix_voice: If set, fixes the built-in Spark-TTS voices (female and male).
    • --host / --port: Network binding settings.

    Once running, the web interface is available at http://localhost:8000 and the API documentation at http://localhost:8000/docs.

    flashtts serve \
     --model_path Spark-TTS-0.5B \
     --backend vllm \
     --role_dir data/roles \
     --llm_device cuda \
     --tokenizer_device cuda \
     --detokenizer_device cuda \
     --wav2vec_attn_implementation sdpa \
     --llm_attn_implementation sdpa \
     --torch_dtype "bfloat16" \
     --max_length 32768 \
     --llm_gpu_memory_utilization 0.6 \
     --fix_voice \
     --host 0.0.0.0 \
     --port 8000