TeleSpeech-ASR

TeleSpeech-ASR is a large-scale multi-dialect speech recognition model pre-trained on 300,000 hours of unlabeled multi-dialect speech. It supports approximately 30 dialects, including Cantonese, Shanghainese, Sichuanese, and Wenzhou dialect.

Available Models:

Usage Note: Pre-trained models require supervised fine-tuning on labeled data before they can perform ASR tasks. You can use either the fairseq-based fine-tuning framework or the wenet-based representation extraction framework.

The model requires 40-dimensional MFCC features extracted from 16K sampling rate audio, not raw audio.

1. Feature Extraction

Use Kaldi to extract 40-dim MFCC features. You can run the prepare_kaldi_feats.sh script by placing it in a Kaldi egs directory alongside a configuration file like mfcc_hires.conf.

Note: If you already have extracted features, you can replace kaldiio.load_mat with the method from wenet/kaldi_io.py to avoid requiring a Kaldi installation during runtime.

2. Create Data Lists

Prepare a data.list file (or .tsv for fine-tuning) where each line is tab-separated (\t) following this format:

utt:<utt_id>\tfeat:<path_to_ark>:<offset>\tfeat_shape:<frames>,<dim>\ttext:<transcription>\ttoken:<tokens>\ttokenid:[TOKENID]\ttoken_shape:<len>,<vocab_size>

Example train/data.list:

utt:X0000000000_100638174_S00037	feat:/data/raw_nnaudio.test.1.ark:2983479385	feat_shape:363,40	text:不惜在这种试验中毁灭包括自己在内的一切	token:不 惜 在 这 种 试 验 中 毁 灭 包 括 自 己 在 内 的 一 切	tokenid:[TOKENID]	token_shape:19,5537

Discrete unit training involves two main steps: extracting features via K-Means and then training the discrete ASR model.

1. Feature Extraction (K-Means)

Use the script discrete_token/dump_feat.sh with the following parameters:

• train_km_set: The dataset used to train the K-Means model.
• percent: The fraction of data to use for training (e.g., 0.1 for 10%). Use -1 to use the entire dataset.
• input_type: Supports original audio wav.scp files or Wenet data.list files.
• feat_save_type: The format for saving representations (npy or kaldi).

Configure K-Means via discrete_token/kmeans_d2v.yaml:

• reader_conf: Settings should be largely consistent with frontend_conf in the continuous training config.
  • max_chunk: If set, audio is truncated by max_chunk during representation extraction. Useful for saving GPU memory.
  • weights: Layer weights used when multilayer_feature is true. Weights can be obtained from continuous representation training.
• kmeans_conf: Parameters for MiniBatchKMeans.
  • n_clusters: The number of clusters (vocabulary size).

2. Discrete ASR Training

Use conf/train_d2v2_discrete_conformer.yaml for training:

• frontend_conf:
  • input_size: Must match the n_clusters value used during K-Means.
  • padding_idx: The padding index used during training.
• Other settings are identical to the continuous representation training configuration (conf/train_d2v2_ark_conformer.yaml).

Continuous representation training is configured via YAML files, such as conf/train_d2v2_ark_conformer.yaml. Key configuration sections include:

frontend_conf (Pre-trained Model Settings)

• finetune_model: Set to true if the model in moder_dir is a fine-tuned model; set to false to use an unsupervised pre-trained model.
• multilayer_feature: Boolean. If true, uses weighted calculation across multiple layers.
• layer: Determines which layers to use. Combined with multilayer_feature, it supports four modes:
  1. layer: [-1] and multilayer_feature: true: Uses the weighted sum of all layers.
  2. layer: [-1] and multilayer_feature: false: Uses only the last layer's representation.
  3. layer: [specific_values] and multilayer_feature: true: Calculates the weighted sum of only the specified layers.
  4. layer: [specific_values] and multilayer_feature: false: Calculates the average of the specified layers.
• num_layer: Total number of layers in the model. Note: For unsupervised pre-trained models, this is layers + 1; for fine-tuned models, it is layers.

preencoder_conf (Dimensionality Reduction)

• Comment this section out to disable dimensionality reduction.

spec_aug_after_conf (Representation Dropout)

• scale: Determines if the mask length should be scaled according to the dimension. Comment this section out to disable.

encoder_conf (Encoder Settings)

• input_layer: Use conv2d2 to enable 2x downsampling.

dataset_conf (Dataset Settings)

• cmvn: Normalization settings for data sent to the pre-trained model. This must match the settings used during pre-training.
• max_length: Set based on positional encoding (default 5000). For data2vec models with 4x downsampling, this can be up to 20000. If input_layer is conv2d2, use max_length * 2.
• Note: speed_perturb and spec_aug are not used for MFCC feature inputs.

# Example configuration structure (conceptual)
frontend_conf:
  finetune_model: true
  multilayer_feature: true
  layer: [-1]
  num_layer: 12
preencoder_conf: {}
spec_aug_after_conf:
  scale: true
encoder_conf:
  input_layer: "conv2d2"
dataset_conf:
  cmvn: true
  max_length: 10000

The train.py script is the primary CLI entrypoint for training or fine-tuning ASR models. It supports single-GPU, multi-GPU (DDP), and CPU training. It can initialize models from a configuration file, load existing checkpoints, or initialize specific encoder modules from pre-trained models.

Key capabilities include:

• Distributed Training: Supports nccl or gloo backends via DDP.
• Mixed Precision: Supports automatic mixed precision (--use_amp) and FP16 gradient synchronization for DDP (--fp16_grad_sync).
• Fine-tuning: Allows freezing specific parameters via the freeze_param key in the YAML config or initializing the encoder with --enc_init.
• Logging: Integrates with TensorBoard for tracking loss, learning rate, and CER.

python wenet/bin/train.py \
    --config your_config.yaml \
    --train_data train_data_file \
    --cv_data cv_data_file \
    --model_dir output_model_dir \
    --symbol_table symbol_table_file