X-Codec-2.0 Documentation

X-Codec-2.0 is a high-performance speech codec designed for high-quality speech reconstruction and multilingual semantic support. It features a large codebook size (65536) using Finite Scalar Quantization, achieving high codebook usage and fast inference (50x tokens per second). It is built using a Transformer + Vocos Decoder architecture and is optimized for 16kHz speech.

Key features include:

• Single Vector Quantization: 65536 codebook size with 99% usage.
• Multilingual Support: Uses Wav2Vec2-BERT semantic encoder trained on 150k hours of multilingual data.
• High-Quality Reconstruction: Utilizes a BigCodec encoder and a specialized Spec discriminator.

For processing large datasets, use the specialized scripts for batch inference and code extraction. Code extraction results are saved in an output folder maintaining the same subfolder structure as the original audio files.

• Batch Inference: Use inference_save_code.py.
• Large-scale Code Extraction: Use the Slurm script large_scale_save_code.sh.

# Batch inference
python inference_save_code.py

# Large-scale code extraction (Slurm)
Sbatch large_scale_save_code.sh

Training requires data preparation followed by running the training script.

1. Prepare Data: Generate a TSV file list using get_tsv.py.
2. Execute Training: Run train.py and provide a log_dir argument.

For large-scale training on Slurm clusters, use the provided shell script.

# 1. Prepare data
python get_tsv.py

# 2. Train with default settings
python train.py log_dir=/path/to/log_dir

# For Slurm-based large-scale training
Sbatch train_slurm.sh

The code is tested on python3.9. Follow these steps to set up your environment:

1. Clone the repository.
2. Create and activate a conda environment with Python 3.9.
3. Install dependencies via requirements.txt.
4. Download the pretrained checkpoint.

Note on xcodec2 version: For codec inference and LLaSA fine-tuning, use xcodec2==0.1.5. For maximum stability during codec training, xcodec2==0.1.3 is recommended.

conda create --name xcodec2 python=3.9
conda activate xcodec2
pip install -r requirements.txt
# Download checkpoint from: https://huggingface.co/HKUST-Audio/xcodec2/blob/main/ckpt/epoch%3D4-step%3D1400000.ckpt

To perform standard inference using the provided script, run inference.py.

python inference.py