KokoClone

repository·main·Indexed 41 days ago

https://github.com/ashish-patnaik/kokoclone

A fast, real-time compatible multilingual voice cloning system built on Kokoro-ONNX. Version 0.1.0 supports Text-to-Speech (TTS) cloning and Audio-to-Audio voice conversion across multiple languages including English, Hindi, French, Japanese, Chinese, Italian, Spanish, and Portuguese. It features a Python API, a CLI, and a Gradio-based web interface, utilizing VRAM-aware chunked voice conversion to manage memory for long recordings.

Tokens: 2.4K
Snippets: 8
Records: 12
Agent score: 53%

What's inside KokoClone

How VRAM-aware chunked voice conversion works
main
When performing audio-to-audio conversion, KokoClone uses chunked_voice_conversion in core/chunked_convert.py to manage memory for long recordings:
- VRAM Budget: On CUDA, chunks are sized so each forward pass uses at most 50% of total GPU memory (configurable via the vram_fraction parameter).
- RoPE Ceiling: The Kanade mel_decoder Transformer has positional embeddings precomputed for 1,024 mel frames. Chunk windows are hard-capped below this limit (≈ 8.9s of source audio per chunk) with a 10% safety margin to prevent recomputation and quality degradation.
- Overlap Smoothing: Each chunk includes a 0.5s overlap on both sides to suppress boundary artifacts.
- Single-Pass Vocoding: The full reassembled mel spectrogram is passed to the vocoder in one shot for clean waveform reconstruction.
https://github.com/ashish-patnaik/kokoclone/blob/main/README.md

Install KokoClone via Conda or uv

main

Follow these steps to set up the environment. You can use Conda (recommended) or uv for fast package management.

1. Clone the Repository

git clone https://github.com/Ashish-Patnaik/kokoclone.git
cd kokoclone

2. Set Up Environment

Using Conda (Recommended)

conda create -n kokoclone python=3.12.12 -y
conda activate kokoclone

For CPU Users (Mac / Standard Laptops):

pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

For GPU Users (Nvidia):

pip install -r requirements.txt
pip install kokoro-onnx[gpu]

Using `uv`

# For CPU Users
uv sync

# For GPU Users (Nvidia)
uv sync --extra gpu

# Activate the environment
source .venv/bin/activate  # Linux/macOS
.venv\Scripts\activate     # Windows

Launch the KokoClone Web Interface
main
KokoClone includes a Gradio-based web application with two main tabs:
1. Text → Clone: Enter text, select a language, upload a reference voice, and generate speech.
2. Audio → Clone: Upload source audio and a reference voice to get re-voiced audio.
To launch the interface, run:
```
python app.py
```
https://github.com/ashish-patnaik/kokoclone/blob/main/README.md
Tips for high-quality voice cloning and conversion
main
To achieve the best results with KokoClone, follow these guidelines:
For Text → Cloned Speech
- Clean Audio: Use a reference audio clip without background noise or music.
- Length: A reference clip of 3 to 10 seconds is the ideal length.
- Language Match: Ensure the selected language code matches the text you are synthesizing.
For Audio → Re-voiced Speech
- Clean Inputs: Use noise-free audio for both the source and the reference clips.
- Reference Length: 3–10 seconds of target speaker audio provides the best voice transfer.
https://github.com/ashish-patnaik/kokoclone/blob/main/app.py

Integrate KokoClone via Python API

main

You can use the KokoClone class and helper functions to integrate voice cloning into your applications.

Text to Cloned Speech

Use KokoClone.generate() to create speech from text using a reference audio sample.

from core.cloner import KokoClone

cloner = KokoClone()
cloner.generate(
    text="This voice is cloned using KokoClone.",
    lang="en",
    reference_audio="reference.wav",
    output_path="output.wav"
)

Audio-to-Audio Voice Conversion

For re-voicing existing audio, use chunked_voice_conversion to handle long recordings with VRAM-aware chunking. This method uses the Kanade voice-conversion model and skips the TTS pipeline.

import soundfile as sf
from kanade_tokenizer import load_audio
from core.cloner import KokoClone
from core.chunked_convert import chunked_voice_conversion

cloner = KokoClone()

# Load audio tensors
source_wav = load_audio("source_speech.wav", sample_rate=cloner.sample_rate).to(cloner.device)
ref_wav = load_audio("target_voice.wav", sample_rate=cloner.sample_rate).to(cloner.device)

# Convert using VRAM-aware chunking
converted = chunked_voice_conversion(
    kanade=cloner.kanade,
    vocoder_model=cloner.vocoder,
    source_wav=source_wav,
    ref_wav=ref_wav,
    sample_rate=cloner.sample_rate,
)

sf.write("revoiced_output.wav", converted.numpy(), cloner.sample_rate)

Use the KokoClone CLI

main

The CLI allows you to generate speech or convert audio directly from the terminal. It supports two modes: tts (Text-to-Speech) and convert (Audio-to-Audio voice conversion).

Text to Cloned Speech (Default Mode)

Use this to synthesize speech from text using a reference voice.

python cli.py --text "Hello from KokoClone" --lang en --ref reference.wav --out output.wav

Audio to Re-voiced Speech

Use this to re-voice an existing recording to sound like a target speaker.

python cli.py --mode convert --source original_speech.wav --ref target_voice.wav --out revoiced.wav

CLI Arguments Reference

Argument	Default	Description
`--mode`	`tts`	`tts` (text → speech) or `convert` (audio → re-voiced audio)
`--text`	—	Text to synthesize (required for `tts` mode)
`--lang`	`en`	Language code: `en hi fr ja zh it es pt`
`--source`	—	Path to source audio (required for `convert` mode)
`--ref`	—	Path to reference voice audio (always required)
`--out`	`output.wav`	Output file path

Generate cloned speech from text using KokoClone.generate()
main
Use the generate method of the KokoClone class to perform text-to-speech cloning. You provide the text to be spoken, the target language, a reference audio file containing the target voice, and a path where the output should be saved.
Parameters:
- text (str): The text you want to synthesize.
- lang (str): The language code (e.g., 'en', 'hi', 'fr').
- reference_audio (str): Path to the reference audio file used for voice cloning.
- output_path (str): Path where the resulting .wav file will be saved.
```
from core.cloner import KokoClone

cloner = KokoClone()
cloner.generate(
    text="Hello, this is a cloned voice.",
    lang="en",
    reference_audio="path/to/reference.wav",
    output_path="output.wav"
)
```
https://github.com/ashish-patnaik/kokoclone/blob/main/app.py
Convert source audio to a target voice using KokoClone.convert()
main
Use the convert method of the KokoClone class for audio-to-audio voice conversion (re-voicing). This allows you to take an existing speech recording and change the speaker's identity to match a reference audio clip without needing a transcription.
Parameters:
- source_audio (str): Path to the audio file containing the speech you want to re-voice.
- reference_audio (str): Path to the audio file containing the target voice characteristics.
- output_path (str): Path where the converted .wav file will be saved.
```
from core.cloner import KokoClone

cloner = KokoClone()
cloner.convert(
    source_audio="path/to/source_speech.wav",
    reference_audio="path/to/target_voice.wav",
    output_path="converted_output.wav"
)
```
https://github.com/ashish-patnaik/kokoclone/blob/main/app.py

KokoClone CLI Reference: Arguments and Flags

main

The KokoClone CLI supports the following arguments:

Flag	Mode	Description
`--mode`	Both	Operation mode: `tts` (text → cloned speech) or `convert` (audio → re-voiced speech). Default: `tts`
`--text`	`tts`	Text to synthesize
`--lang`	`tts`	Language code (`en`, `hi`, `fr`, `ja`, `zh`, `it`, `pt`, `es`). Default: `en`
`--source`	`convert`	Path to source audio file to re-voice (`.wav`)
`--ref`	Both	Path to reference voice audio file (`.wav`) (Required)
`--out`	Both	Output file path (`.wav`). Default: `output.wav`

Supported languages for KokoClone
main
When using KokoClone.generate(), the following language codes are supported:
Language Code
English en
Hindi hi
French fr
Japanese ja
Chinese zh
Italian it
Spanish es
Portuguese pt
https://github.com/ashish-patnaik/kokoclone/blob/main/app.py
Use KokoClone CLI for Audio-to-Audio Voice Conversion
main
Use the convert mode to re-voice an existing audio file using a different reference voice. This performs audio-to-audio voice conversion.
Required Arguments:
- --ref: Path to the reference voice audio file (.wav) that you want to clone.
- --source: Path to the source audio file (.wav) that you want to re-voice.
Optional Arguments:
- --out: Output file path (.wav). Defaults to output.wav.
```
python cli.py --mode convert --source source_audio.wav --ref reference_voice.wav --out converted_output.wav
```
https://github.com/ashish-patnaik/kokoclone/blob/main/cli.py
Use KokoClone CLI for Text-to-Speech (TTS)
main
Use the tts mode to synthesize speech from text using a reference voice. You must provide the text to be spoken, a reference audio file for the voice cloning, and an output path.
Required Arguments:
- --ref: Path to the reference voice audio file (.wav).
- --text: The text you want to synthesize.
Optional Arguments:
- --lang: Language code. Supported values: en, hi, fr, ja, zh, it, pt, es. Defaults to en.
- --out: Output file path (.wav). Defaults to output.wav.
```
python cli.py --mode tts --text "Hello, how are you?" --ref reference.wav --out output.wav
```
https://github.com/ashish-patnaik/kokoclone/blob/main/cli.py

Language	Code
English	`en`
Hindi	`hi`
French	`fr`
Japanese	`ja`
Chinese	`zh`
Italian	`it`
Spanish	`es`
Portuguese	`pt`

KokoClone

What's inside KokoClone

1. Clone the Repository

2. Set Up Environment

Using Conda (Recommended)

Using uv

For Text → Cloned Speech

For Audio → Re-voiced Speech

Text to Cloned Speech

Audio-to-Audio Voice Conversion

Text to Cloned Speech (Default Mode)

Audio to Re-voiced Speech

CLI Arguments Reference

Using `uv`