DOCSEARCHbyAlgolia
Submit Docs

kokoro-fastapi

repository·master·Indexed 26 days ago

https://github.com/remsky/kokoro-fastapi

FastKoko is a Dockerized FastAPI wrapper for the Kokoro-82M text-to-speech model. It provides an OpenAI-compatible API for multi-language speech synthesis, supporting features such as voice mixing, weighted voice ratios, timestamped captions, and streaming audio. It supports deployment across CPU, NVIDIA GPU (CUDA), Apple Silicon (MPS), and AMD GPU (ROCm) hardware.

Tokens
10.8K
Snippets
27
Records
66
Agent score
88%

What's inside kokoro-fastapi

  1. Install FastKoko via Docker

    master

    FastKoko provides pre-built multi-arch images with models included. Choose the image based on your hardware:

    • No GPU (CPU-only): kokoro-fastapi-cpu:latest
    • Apple Silicon (M1/M2/M3): Use kokoro-fastapi-cpu:latest in Docker, or run natively via ./start-gpu_mac.sh for MPS acceleration.
    • NVIDIA GPU (x86_64):
      • GTX 9xx, 10xx, 20xx, 30xx, 40xx: kokoro-fastapi-gpu:latest-cu126 or kokoro-fastapi-gpu:latest
      • RTX 50-series / Blackwell: kokoro-fastapi-gpu:latest-cu128
    • NVIDIA on arm64 (Jetson, GH200): kokoro-fastapi-gpu:latest
    • AMD GPU (experimental, x86_64 only): kokoro-fastapi-rocm:latest

    Configuration can be managed via environment variables (refer to core/config.py).

    # CPU
docker run -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-cpu:latest

# NVIDIA (x86_64 or arm64)
docker run --gpus all -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-gpu:latest

# NVIDIA Blackwell / RTX 50-series
docker run --gpus all -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-gpu:latest-cu128

# AMD
docker run --device=/dev/kfd --device=/dev/dri -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-rocm:latest

  2. Install FastKoko via Docker Compose

    master

    To run the full setup including the Web UI using Docker Compose:

    1. Clone the repository:
      git clone https://github.com/remsky/Kokoro-FastAPI.git
cd Kokoro-FastAPI
    2. Navigate to the appropriate directory for your hardware:
      • NVIDIA: cd docker/gpu
      • CPU: cd docker/cpu
      • AMD: cd docker/rocm
    3. Start the service:
      docker compose up --build

    Note for Apple Silicon users: Use docker/cpu for Docker, or run natively via UV for MPS acceleration.

    git clone https://github.com/remsky/Kokoro-FastAPI.git
cd Kokoro-FastAPI

# For NVIDIA GPU support
cd docker/gpu
docker compose up --build

  4. Fix 'invalid state' error in KPipeline

    master

    The error Error generating speech: The object is in an invalid state is caused by device mismatches when creating pipelines directly. To avoid this, do not instantiate KPipeline manually. Instead, use the backend's internal pipeline management to ensure the pipeline is correctly initialized for the target language and device.

    # Instead of creating pipelines directly:
# pipeline = KPipeline(...)

# Use backend's pipeline management:
pipeline = backend._get_pipeline(pipeline_lang_code)

  5. Deploy Kokoro-FastAPI via Docker

    master

    Depending on your hardware, use the corresponding Docker directory to build and run the service.

    • NVIDIA GPU: Requires NVIDIA driver with CUDA 12.6+ support.
    • CPU: Uses PyTorch CPU inference.
    • AMD GPU: Uses ROCm 6.4 (experimental, amd64 only).
    # GPU: Requires NVIDIA driver with CUDA 12.6+ support
cd docker/gpu
docker compose up --build

# CPU: PyTorch CPU inference
cd docker/cpu
docker compose up --build

# AMD GPU: ROCm 6.4 (experimental, amd64 only)
cd docker/rocm
docker compose up --build

  6. Run FastKoko directly via UV

    master

    For native execution with hot-reload support, use uv.

    Prerequisites:

    • Install astral-uv
    • Install espeak-ng on your system (recommended fallback for unknown sounds).

    Steps:

    1. Clone the repository and enter the directory.
    2. Run the model download script if models are not present: python docker/scripts/download_model.py --output api/src/models/v1_0
    3. Start the service:

    Linux and macOS:

    • ./start-cpu.sh (CPU)
    • ./start-gpu.sh (GPU)

    Windows:

    • .\start-cpu.ps1 (CPU)
    • .\start-gpu.ps1 (GPU)
    git clone https://github.com/remsky/Kokoro-FastAPI.git
cd Kokoro-FastAPI

# Start directly via UV (Linux/macOS)
./start-gpu.sh

  7. Fix ESpeak-NG data path errors

    master

    If you encounter errors regarding missing espeak-ng-data/phontab files, ensure you are not using a direct phonemizer dependency that conflicts with misaki. Instead, allow misaki to manage phonemizer-fork and espeakng-loader.

    To resolve path issues in a containerized environment, set the following environment variables to point to your espeak-ng data directory:

    ENV PHONEMIZER_ESPEAK_PATH=/usr/bin \
    PHONEMIZER_ESPEAK_DATA=/usr/share/espeak-ng-data \
    ESPEAK_DATA_PATH=/usr/share/espeak-ng-data

  8. Configure debug and development endpoints

    master

    By default, /dev/* and /debug/* routes are disabled for security. To use them, set the following environment variables:

    • Enable Debug Routes: Set ENABLE_DEBUG_ENDPOINTS=true to access /debug/threads, /debug/storage, and /debug/system.
    • Enable Model Unloading: Set ALLOW_DEV_UNLOAD=true to enable POST /dev/unload, which frees the model from VRAM and reloads it lazily on the next request.

  9. Configure API logging level

    master

    The global API logging level can be controlled via the API_LOG_LEVEL environment variable. The default is DEBUG. Supported levels follow loguru logging levels.

    # Docker
docker run --env 'API_LOG_LEVEL=WARNING' ...

# Linux/macOS (Direct via UV)
export API_LOG_LEVEL=WARNING
./start-cpu.sh

# Windows (Direct via UV)
$env:API_LOG_LEVEL = 'WARNING'
.\start-cpu.ps1

  10. Deploy kokoro-fastapi with ROCm support via Docker Compose

    master

    Use the docker-compose.yml file in the docker/rocm directory to deploy the Kokoro FastAPI service with AMD GPU (ROCm) acceleration. This configuration requires access to /dev/dri and /dev/kfd devices and specific group IDs for video, input, and render to function correctly.

    Key Configuration Details:

    • Ports: The service is exposed on port 8880.
    • GPU Acceleration: Requires USE_GPU=true and TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 environment variables.
    • Persistence: MIOpen configuration and cache are persisted using named volumes (miopen_config and miopen_cache) to avoid expensive per-process kernel searches on every run.
    • Group IDs: You may need to adjust the group_add values (currently 44, 993, 996) to match your host system's group IDs for video, input, and render. You can find these by running: getent group $GROUP_NAME | cut -d: -f3.
    name: kokoro-fastapi-rocm
services:
  kokoro-tts:
      build:
        context: ../..
        dockerfile: docker/rocm/Dockerfile
      devices:
        - /dev/dri
        - /dev/kfd
      group_add:
        - 44
        - 993
        - 996
      restart: 'always'
      volumes:
        - miopen_config:/home/appuser/.config/miopen
        - miopen_cache:/home/appuser/.cache/miopen
      ports:
        - 8880:8880
      environment:
        - USE_GPU=true
        - TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1

volumes:
  miopen_config:
  miopen_cache:

  12. Optimize MIOpen performance for RDNA 2+ GPUs

    master

    To achieve best-case latency on RDNA 2+ hardware, you can pre-populate the MIOpen cache. By default, MIOPEN_FIND_MODE=2 is used to reuse the on-disk database. To perform an exhaustive search and populate the cache, follow these steps:

    1. Override Environment Variables: Set the following in your environment:
      • MIOPEN_FIND_MODE=3
      • MIOPEN_FIND_ENFORCE=3
    2. Generate Workload: Run docker/rocm/warmup_miopen.py inside the container, or generate audio for varied input lengths (e.g., several paragraphs of text).
    3. Revert: Remove the overrides. The default FIND_MODE=2 will then reuse the newly populated cache stored in the miopen_cache volume.