Amphion Toolkit

Amphion is an open-source toolkit designed for Audio, Music, and Speech Generation research and development. It aims to provide a platform for converting various inputs into audio and supports reproducible research through model visualizations.

Key supported generation tasks include:

• TTS: Text to Speech
• SVS: Singing Voice Synthesis
• VC: Voice Conversion
• AC: Accent Conversion
• SVC: Singing Voice Conversion
• TTA: Text to Audio
• TTM: Text to Music (currently under development)

The toolkit also provides vocoders for high-quality audio signal production and evaluation metrics for consistent performance measurement in generation tasks.

The Amphion GAN-based Vocoder recipe allows you to train and use various GAN-based vocoder architectures. The process follows four main stages: Data Preparation, Feature Extraction, Training, and Inference.

Supported Generators:

• MelGAN
• HiFi-GAN
• NSF-HiFiGAN
• BigVGAN
• APNet

Supported Discriminators:

• Multi-Scale Discriminator (msd)
• Multi-Period Discriminator (mpd)
• Multi-Resolution Discriminator
• Multi-Scale Short-Time Fourier Transform Discriminator (msstftd)
• Multi-Scale Constant-Q Transform Discriminator (mssbcqtd)

Emilia is a large-scale, multilingual speech generation dataset. Emilia-Large contains over 215k hours of speech data across English, Chinese, German, French, Japanese, and Korean.

Emilia-Pipe is an open-source preprocessing pipeline designed to transform raw, in-the-wild speech data into high-quality, annotated training data for speech generation. The pipeline follows these steps:

1. Standardization: Audio normalization.
2. Source Separation: Separating speech from background music (BGM).
3. Speaker Diarization: Extracting single-speaker speech.
4. Fine-grained Segmentation (VAD): Creating 3-30s single-speaker segments.
5. ASR: Generating transcriptions.
6. Filtering: Final dataset refinement.

MultipleContentsSVC is an implementation for Singing Voice Conversion (SVC) that leverages diverse semantic-based audio pretrained models (Whisper and ContentVec) to extract content features. The acoustic model uses DiffWaveNetSVC (a Bidirectional Non-Causal Dilated CNN), and the vocoder uses the BigVGAN architecture.

To use this recipe, all commands must be executed from the Amphion root directory.

SingVisio is a visual analytics system designed to analyze the diffusion models used in Singing Voice Conversion (SVC). It consists of a web-based front-end (built with D3.js) and a back-end generative model.

Core Components

• Web Interface: The front-end code is located in ../../../visualization/SingVisio/webpage/.
• Generative Model: Uses MultipleContentsSVC, a diffusion-based model. The recipe for this model can be found in the Amphion repository under ../../svc/MultipleContentsSVC/.

Online Experience

You can experience the online system via OpenXLab.

Amphion provides several models for voice conversion:

• Vevo: Zero-shot imitation. Use Vevo-Timbre for style-preserved conversion and Vevo-Voice for style-converted conversion.
• FACodec: Decomposes speech into content, prosody, and timbre subspaces for zero-shot VC.
• Noro: A noise-robust zero-shot VC system using dual-branch reference encoding and noise-agnostic contrastive speaker loss.

Amphion supports several state-of-the-art TTS architectures:

• FastSpeech2: Non-autoregressive Transformer-based.
• VITS: End-to-end with conditional VAE and adversarial learning.
• VALL-E: Zero-shot neural codec language model.
• NaturalSpeech2: Latent diffusion model.
• Jets: Jointly trained FastSpeech2 and HiFi-GAN.
• MaskGCT: Fully non-autoregressive without explicit alignment.
• Vevo-TTS: Zero-shot with controllable timbre and style (autoregressive + flow-matching).
• DualCodec-VALLE: VALL-E model using 12.5Hz DualCodec tokens.

Amphion supports various neural vocoders:

• GAN-based: MelGAN, HiFi-GAN, NSF-HiFiGAN, BigVGAN, APNet.
• Flow-based: WaveGlow.
• Diffusion-based: Diffwave.
• Auto-regressive: WaveNet, WaveRNN.
• Enhancement: Supports the Multi-Scale Constant-Q Transform Discriminator to enhance GAN-based vocoders during training without affecting inference.

Amphion implements high-efficiency audio codecs:

• DualCodec: A low-frame-rate (12.5Hz or 25Hz) semantically-enhanced codec designed for efficient speech generation.
• FACodec: Decomposes speech into subspaces (content, prosody, timbre) for attribute manipulation.

Amphion currently supports the NORO architecture for noise-robust Voice Conversion. Key features of this architecture include:

1. Noise-Robust Voice Conversion: Uses a dual-branch reference encoding module and noise-agnostic contrastive speaker loss to ensure high-quality conversion even in noisy environments.
2. One-shot Voice Conversion: Enables timbre conversion using only a single reference speech sample.
3. Speaker Representation Learning: Leverages the reference encoder as a self-supervised speaker encoder.