SALMONN Multi-modal LLM Family

SALMONN is a suite of advanced multi-modal large language models (LLMs) designed for various sensory modalities including vision, speech, text, and action. Because the repository acts as a central hub, specific implementations and models are located in dedicated branches.

Key models in the family include:

• ELLSA: An end-to-end model unifying vision, speech, text, and action in a streaming full-duplex framework.
• video-SALMONN 2: An audio-visual LLM for high-quality audio-visual video captioning and general video QA.
• video-SALMONN-o1: A reasoning-enhanced audio-visual LLM.
• SALMONN for speech quality assessment: Specialized for automatic speech quality evaluation.
• video-SALMONN: Speech-enhanced audio-visual LLM.
• SALMONN (Original): Focused on generic hearing abilities for LLMs.

The SALMONN project is organized into different branches for each specific model or research direction. To use a specific model, you must navigate to its corresponding branch in the repository:

• SALMONN 2: https://github.com/bytedance/SALMONN/tree/salmonn2
• ELLSA: https://github.com/bytedance/SALMONN/tree/ELLSA
• video-SALMONN 2: https://github.com/bytedance/video-SALMONN-2
• F-16: https://github.com/bytedance/F-16
• video-SALMONN-o1: https://github.com/bytedance/SALMONN/tree/video-salmonn-o1
• Speech Quality Assessment: https://github.com/bytedance/SALMONN/tree/speech_quality_assessment
• video-SALMONN: https://github.com/bytedance/SALMONN/tree/videosalmonn
• SALMONN (Original): https://github.com/bytedance/SALMONN/tree/salmonn

The QualiSpeech dataset is a speech quality assessment dataset featuring natural language reasoning and descriptions. It can be used to develop audio LLMs for speech quality assessment or to evaluate the low-level speech perception capabilities of existing audio LLMs.

You can download the dataset from Hugging Face: https://huggingface.co/datasets/tsinghua-ee/QualiSpeech

For users training their own SALMONN models, the project provides all necessary annotations for the 3-stage training process, including:

• 600k SQA/AQA data
• 50k audio-based storytelling data

Annotations can be downloaded via Google Drive: https://drive.google.com/file/d/15cQO--rtMM9JD22y-A5oXXvT3DujgE2e/view?usp=sharing