DOCSEARCHbyAlgolia
Submit Docs

SLAM-LLM

repository·main·Indexed 21 days ago

https://github.com/x-lance/slam-llm

A modular, open-source deep learning toolkit for training and deploying multimodal large language models (MLLMs) specialized in Speech, Language, Audio, and Music processing. It provides recipes for training custom MLLMs, high-performance checkpoints for inference, and support for encoders like WavLM, Whisper, and EAT, alongside LLMs such as Qwen2.5-7B-Instruct and Vicuna-7b-v1.5.

Tokens
16.9K
Snippets
73
Records
99
Agent score
77%

What's inside slam-llm

  1. Overview of SLAM-AAC

    main

    SLAM-AAC is an LLM-based framework designed for Automated Audio Captioning (AAC). It improves captioning quality through two primary methods:

    1. Paraphrasing Augmentation: Used during training to enhance dataset diversity.
    2. CLAP-Refine Strategy: A plug-and-play decoding strategy used during inference to refine multiple beam search candidates using a pre-trained CLAP model.

    Model Architecture:

    • Audio Encoder: EAT
    • LLM Decoder: Vicuna-7B
    • Trainable Modules: During training, only the Linear Projector and LoRA modules are updated.

  2. Overview of MaLa-ASR_SLIDESPEECH

    main

    MaLa-ASR is an LLM-based Automatic Speech Recognition (ASR) model designed to improve recognition accuracy for conference content by integrating textual keywords extracted from presentation slides.

    Model Architecture:

    • Speech Encoder: Official WavLM-Large model.
    • LLM Decoder: Public Vicuna 7B.
    • Projector: A simple linear projector consisting of a 1-D convolution layer and two linear layers.

    In this specific recipe, only the linear projector is trained.

  3. Overview of SLAM-LLM features

    main

    SLAM-LLM is a deep learning toolkit for training custom multimodal large language models (MLLM) focusing on Speech, Language, Audio, and Music.

    Key features include:

    • Extensibility: Easily extend to new models and tasks.
    • Mixed Precision Training: Faster training with reduced GPU memory on NVIDIA tensor cores.
    • Multi-GPU Support: Supports DDP, FSDP, and DeepSpeed for data and model parallelism.
    • Flexible Configuration: Uses Hydra and Python dataclasses for hierarchical configuration.

  4. Overview of AISPEECH_ASR Example

    main

    The aispeech_asr example is optimized for large-scale industrial training (e.g., 100,000+ hours of data). Key features include:

    • Multi-task training: Supports tasks like ASR (Automatic Speech Recognition) and ST (Speech Translation) via a unified data format.
    • Dynamic Prompting: Randomly selects prompts from a provided list.
    • Iterative Dataset: Reduces startup latency for massive datasets.
    • DeepSpeed Integration: Reduces GPU memory usage during training.
    • Distributed Inference: Supports multi-machine, multi-GPU decoding to speed up evaluation.
    • Dynamic Frame Batching: Combines frames based on audio size rather than fixed batch sizes, significantly reducing training/evaluation time (up to 75% reduction for 100k hours).

  5. Speech Emotion Caption (SECap) Overview

    main

    The Speech Emotion Caption recipe generates high-quality, human-like descriptions of emotions present in speech. The architecture utilizes a q-former projector to bridge a speech encoder and a Large Language Model (LLM).

    Model Components:

    • Encoder: emotion2vec_base
    • Projector: Q-Former (the only component trained in this recipe)
    • LLM: vicuna-7b-v1.5

  6. Understand SLAM-LLM configuration priority

    main

    SLAM-LLM uses a hierarchical configuration system. When multiple configuration sources are provided, they are applied in the following order of precedence (highest priority first):

    1. command-line (shell file): Arguments passed via the CLI or shell scripts.
    2. Hydra configuration (yaml file): Settings defined in .yaml files.
    3. dataclass configuration (Python file): Default values defined in Python dataclasses.

  9. How CTC-Assisted LLM-Based Contextual ASR works

    main

    This model implements contextual Automatic Speech Recognition (ASR) by combining CTC (Connectionist Temporal Classification) decoding with a Large Language Model (LLM).

    The process follows two main steps:

    1. Filtering: The model uses CTC decoding results to filter potential relevant hotwords from a pre-defined hotwords list.
    2. Incorporation: These filtered hotwords are incorporated into the LLM prompt input to improve the recognition accuracy of specific terms.

    Model Components:

    • Speech Encoder: WavLM-Large (pre-trained on 94,000 hours, fine-tuned on 960h Librispeech with CTC loss).
    • Adapter: A linear projector consisting of a 1-D convolution layer and two linear layers.
    • LLM Decoder: Vicuna 7B.

  10. Train a New Model using AISPEECH_ASR Scripts

    main

    You can train models by modifying scripts/finetune_deepspeed.sh (recommended) or scripts/finetune_torchrun.sh.

    Training Workflow

    1. Projector Training: Set use_peft=false to train only the projector.
    2. LoRA Fine-tuning: Set use_peft=true and provide the ckpt_path pointing to the projector weights saved in the previous step.

    Configuration Variables

    Modify these variables in your shell script:

    • run_dir: Directory to save the model.
    • train_scp_file_path: Path to training data.
    • dev_scp_file_path: Path to validation data.
    • multitask_prompt_path: Path to multitask.jsonl.
    • projector: Type of projector (e.g., linear).
    • encoder_name: Name of the encoder (e.g., whisper).
    • llm_name: Name of the LLM (e.g., Qwen2.5-7B-Instruct).
    • use_peft: Boolean (true/false) for LoRA training.
    • use_fp16: Boolean (true/false) for half-precision training.
    • freeze_encoder: Boolean (true/false) to freeze the encoder.
    • deepspeed_config: Path to DeepSpeed configuration file.
    run_dir=  # Directory to save the model
train_scp_file_path=  # Path to training data
dev_scp_file_path=  # Path to validation data
train_max_frame_length=1500
eval_max_frame_length=1000
multitask_prompt_path=  # Path to multitask.jsonl
projector=linear
encoder_name=whisper
llm_name=Qwen2.5-7B-Instruct
use_peft=false
use_fp16=true
freeze_encoder=true
pad_or_trim=true
deepspeed_config=  # Path to DeepSpeed configuration file

  11. Set up the SLAM-Omni environment

    main

    You can set up the SLAM-Omni environment either by installing dependencies via pip or by using a pre-configured Docker image. Ensure the SLAM-LLM environment is prepared before proceeding.

    # Option 1: Pip installation
pip install -r ./examples/s2s/requirements.txt

# Option 2: Docker installation
docker pull worstchan/slam-omni:v0
docker run -it --gpus all --name slam-omni worstchan/slam-omni:v0 /bin/bash

  12. Train or fine-tune a new VALL-E-X AR model

    main

    Once the dataset has been pretreated into binary files, you can start training or fine-tuning the Autoregressive (AR) model.

    1. Configure Data Path: In the training script, set train_data_path to the directory containing your processed binary files.
    2. Execute Training: Run the provided shell script to begin the process.
    # After setting train_data_path in the script
bash examples\vallex\scripts\vallex.sh