DramaBox Documentation

When defining a speaker, use a generic noun phrase combined with at most one optional adjective. Do not use professional roles, as the model may attempt to speak the role name itself.

To ensure reliable delivery, use a multi-segment pattern that combines stage directions with quoted dialogue. The model uses unquoted text as stage directions to shape delivery and quoted text as the literal spoken content.

The Pattern: A <speaker> <verb>, "<dialogue>" <pronoun> <verb>, "<dialogue>"

Key Rules:

• Quoted text is spoken literally: This includes non-verbal sounds like Hahaha, Mmmm-mmm, or Woooo.
• Unquoted text is stage direction: This shapes the delivery but is not spoken.
• End at the last closing quote: Any description following the final quote may be ignored or read incorrectly.
• Use multi-segment beats: For high emotional contrast, use a calm setup followed by an emotional shift rather than a single expressive verb.

Example:

A man speaks evenly, "I gave you one job."
His voice rises with fury, "AND YOU MESSED IT UP!"

The REUSEUpsampler class requires optional dependencies for speech enhancement and bandwidth extension. If these are not installed, the class will fail to initialize. To enable denoising, install the required packages using the provided requirements file:

pip install -r requirements-reuse.txt

Note: mamba-ssm and causal-conv1d do not have pre-built wheels for macOS or Windows and require matching CUDA and nvcc on Linux. If you cannot install these, you can run the pipeline without RE-USE by setting denoise_ref=False in the server configuration or unticking 'Denoise voice reference' in the Gradio UI.

To improve voice cloning quality, you can opt-in to nvidia/RE-USE denoising. This requires additional dependencies that are not included in the base installation to avoid compatibility issues on macOS/Windows.

1. Install dependencies:

pip install -r requirements-reuse.txt

2. Use in code: Set denoise_ref=True in your generate_to_file call. If dependencies are missing, the server will log a warning and fallback to standard generation without denoising.

server.generate_to_file(
    prompt='A woman speaks warmly, "Hello."',
    output="out.wav",
    voice_ref="ref.wav",
    denoise_ref=True,
)

All audio generated via inference.py or TTSServer.generate_to_file is automatically watermarked using Resemble Perth. This watermark is imperceptible and survives compression/editing.

To detect the watermark, use the perth library:

```python
import perth, librosa
wav, sr = librosa.load("output.wav", sr=None, mono=True)
detector = perth.PerthImplicitWatermarker()
print(detector.get_watermark(wav, sample_rate=sr))   # confidence ≈ 1.0

To disable watermarking for debugging, pass --no-watermark to the CLI or watermark=False to generate_to_file.

DramaBox uses a prompt-driven approach where the text controls emotion, delivery, and non-verbal sounds.

Structure: <speaker description>, "<dialogue>" <action direction> "<more dialogue>"

Guidelines:

• Inside Quotes (Actual Sounds): Use for laughs ("Hahaha", "Hehehe") and sounds ("Mmmmm", "Ugh", "Ahhh"). Note: Laughs must be a single word.
• Outside Quotes (Stage Directions): Use for emotional cues or physical actions (e.g., She sighs deeply., He gulps nervously., A long pause.).
• Avoid inside quotes: Do not write words like Ahem, Sigh, or Cough inside quotes, as the model may attempt to speak them literally.
• Consistency: Match the gender/age in the speaker description to your voice_ref.
• Termination: End the prompt at the last closing quote mark; do not add trailing descriptions.

When initializing REUSEUpsampler, you can tune several parameters to balance VRAM usage and processing quality:

• target_sr (int): The target sampling rate for enhancement/extension (default: 48000).
• config_path (str, optional): Path to a specific YAML configuration file. If not provided, it defaults to a standard recipe from the RE-USE repository.
• chunk_size_s (float): The duration of each processing chunk in seconds. Smaller values reduce VRAM usage but may affect continuity. (default: 1.0).
• hop_portion (float): The portion of the chunk size used for the hop length in the overlap-add process. (default: 0.5).
• device (str | torch.device): The device to run inference on (default: "cuda").

up = REUSEUpsampler(
    target_sr=48000,
    config_path="/path/to/custom_config.yaml",
    chunk_size_s=2.5,
    hop_portion=0.5,
    device="cuda"
)

For prompts that result in estimated durations longer than the model's stable regime (typically >45s), DramaBox uses a chunking mechanism to maintain quality.

• Max chunk duration (max_chunk_duration): The hard cap for a single audio chunk. Chunks exceeding this are rare and may exhibit degradation.
• Target chunk duration (target_chunk_duration): A soft cap. The system stops adding new sentences to a chunk once it approaches this duration, providing a buffer (headroom) before the hard cap.
• Crossfade (crossfade_ms): The amount of equal-power overlap between adjacent chunks. A value of 50.0 ms is generally inaudible and provides smooth transitions. 0.0 ms results in a hard cut.