Dia Text-to-Speech Model

Dia is integrated into the transformers library. You must install the main branch of transformers to use it. Use AutoProcessor and DiaForConditionalGeneration to load the model and generate audio.

from transformers import AutoProcessor, DiaForConditionalGeneration

torch_device = "cuda"
model_checkpoint = "nari-labs/Dia-1.6B-0626"

text = [
    "[S1] Dia is an open weights text to dialogue model. [S2] You get full control over scripts and voices. [S1] Wow. Amazing. (laughs) [S2] Try it now on Git hub or Hugging Face."
]
processor = AutoProcessor.from_pretrained(model_checkpoint)
inputs = processor(text=text, padding=True, return_tensors="pt").to(torch_device)

model = DiaForConditionalGeneration.from_pretrained(model_checkpoint).to(torch_device)
outputs = model.generate(
    **inputs, max_new_tokens=3072, guidance_scale=3.0, temperature=1.8, top_p=0.90, top_k=45
)

outputs = processor.batch_decode(outputs)
processor.save_audio(outputs, "example.mp3")

To achieve high-quality, realistic dialogue with Dia, follow these guidelines:

• Input Length: Aim for moderate lengths. Inputs under 5 seconds may sound unnatural; inputs over 20 seconds may result in unnaturally fast speech.
• Speaker Tags: Always begin input text with [S1]. Alternate between [S1] and [S2] (e.g., [S1]... [S2]... [S1]...). Do not repeat the same tag consecutively (e.g., [S1]... [S1]... is incorrect).
• Non-verbal Tags: Use non-verbal tags (like (laughs)) sparingly. Overuse or using unlisted tags can cause artifacts.
• Voice Cloning (Audio Prompts):
  • Provide the transcript of the audio you wish to clone before the generation text.
  • Use [S1] and [S2] tags correctly in the transcript.
  • For best results, the cloning audio should be 5–10 seconds long (approx. 430–860 tokens).
• Audio Quality: To improve quality at the end of a generation, place the tag for the second-to-last speaker (e.g., [S1] or [S2]) at the very end of the audio.

The run_inference function is the primary programmatic entrypoint for generating speech. It handles text input, optional audio prompting (for voice cloning/style transfer), and various sampling parameters.

Parameters:

• text_input (str): The text to be synthesized.
• audio_prompt_text_input (str): The transcript of the provided audio prompt. Required if audio_prompt_input is used.
• audio_prompt_input (Optional[Tuple[int, np.ndarray]]): A tuple containing the sample rate (int) and the audio data (numpy array). If provided, the text in audio_prompt_text_input is prepended to text_input to guide the model.
• max_new_tokens (int): Controls the maximum length of the generated audio.
• cfg_scale (float): Guidance strength. Higher values increase adherence to the text prompt.
• temperature (float): Randomness. Lower is more deterministic, higher is more random.
• top_p (float): Nucleus sampling probability.
• cfg_filter_top_k (int): Top-k filter for CFG guidance.
• speed_factor (float): Adjusts the speed of the output (e.g., 1.0 is original, 0.8 is slower).
• seed (Optional[int]): Seed for reproducibility. Use -1 or None for a random seed.

from app import run_inference
import numpy as np

# Example call
audio, seed, logs = run_inference(
    text_input="Hello, this is a test.",
    audio_prompt_text_input="",
    audio_prompt_input=None,
    max_new_tokens=3072,
    cfg_scale=3.0,
    temperature=1.8,
    top_p=0.95,
    cfg_filter_top_k=45,
    speed_factor=1.0,
    seed=42
)

The following arguments are available for the Dia CLI:

Positional Arguments

• text: Input text for speech generation.

Required Arguments

• --output <path>: Path to save the generated audio file (e.g., output.wav).

Model Loading

• --repo-id <id>: Hugging Face repository ID (default: nari-labs/Dia-1.6B-0626).
• --local-paths: Flag to load model from local config and checkpoint files.
• --config <path>: Path to local config.json file (required if --local-paths is set).
• --checkpoint <path>: Path to local model checkpoint .pth file (required if --local-paths is set).

Generation Parameters

• --max-tokens <int>: Maximum number of audio tokens to generate (defaults to config value).
• --cfg-scale <float>: Classifier-Free Guidance scale (default: 3.0).
• --temperature <float>: Sampling temperature (higher is more random, default: 1.3).
• --top-p <float>: Nucleus sampling probability (default: 0.95).
• --audio-prompt <path>: Path to an optional audio prompt WAV file for voice cloning.

Infrastructure

• --seed <int>: Random seed for reproducibility.
• --device <string>: Device to run inference on (e.g., cuda, cpu; default is auto-detected).