DOCSEARCHbyAlgolia
Submit Docs

Conversational Speech Model (CSM)

repository·main·Indexed 12 days ago

https://github.com/sesameailabs/csm

A speech generation model that converts text and audio inputs into RVQ audio codes using a Llama-based architecture and the Mimi audio decoder. CSM supports audio generation from text and conversational context via Segment objects to maintain speaker identity and flow. It utilizes the Llama-3.2-1B and CSM-1B models and is primarily optimized for English.

Tokens
1.5K
Snippets
3
Records
4
Agent score
47%

What's inside CSM

  1. What is CSM and its capabilities?

    main

    CSM (Conversational Speech Model) is a speech generation model that generates RVQ audio codes from text and audio inputs. It uses a Llama backbone and a Mimi audio decoder.

    Key Characteristics:

    • Base Model: It is a base generation model and does not come with pre-set specific voices; it can produce a variety of voices but is not fine-tuned on specific identities.
    • Not an LLM: CSM is an audio generation model, not a general-purpose multimodal LLM. It cannot generate text. For conversational applications, you should use a separate LLM to generate the text responses before passing them to CSM.
    • Language Support: While it has some capacity for non-English languages due to training data, it is primarily optimized for English and may not perform well in other languages.

  2. Install and Setup CSM

    main

    To set up CSM, clone the repository, create a Python 3.10 virtual environment, and install the requirements. You must have a CUDA-compatible GPU and access to the Llama-3.2-1B and CSM-1B models on Hugging Face.

    Important Configuration:

    • Disable lazy compilation: Set the environment variable NO_TORCH_COMPILE=1 to disable lazy compilation in Mimi.
    • Authentication: Run huggingface-cli login to authenticate with Hugging Face to access the required models.

    Windows Users: Standard triton cannot be installed on Windows. Use pip install triton-windows instead.

    git clone git@github.com:SesameAILabs/csm.git
cd csm
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Disable lazy compilation in Mimi
export NO_TORCH_COMPILE=1

# Authenticate to access models
huggingface-cli login

  3. Generate audio with conversational context using `Segment`

    main

    To improve audio quality and continuity, provide conversational context using a list of Segment objects. Each Segment represents a previous utterance in the conversation and requires:

    • text: The transcript of the utterance.
    • speaker: The integer ID of the speaker.
    • audio: A torch tensor of the audio (must be resampled to the generator's sample rate).

    This allows the model to maintain speaker identity and conversational flow.

    from generator import load_csm_1b, Segment
import torchaudio
import torch

generator = load_csm_1b(device="cuda")

speakers = [0, 1, 0, 0]
transcripts = [
    "Hey how are you doing.",
    "Pretty good, pretty good.",
    "I'm great.",
    "So happy to be speaking to you.",
]
audio_paths = [
    "utterance_0.wav",
    "utterance_1.wav",
    "utterance_2.wav",
    "utterance_3.wav",
]

def load_audio(audio_path):
    audio_tensor, sample_rate = torchaudio.load(audio_path)
    # Resample to match generator.sample_rate
    audio_tensor = torchaudio.functional.resample(
        audio_tensor.squeeze(0), orig_freq=sample_rate, new_freq=generator.sample_rate
    )
    return audio_tensor

segments = [
    Segment(text=transcript, speaker=speaker, audio=load_audio(audio_path))
    for transcript, speaker, audio_path in zip(transcripts, speakers, audio_paths)
]

audio = generator.generate(
    text="Me too, this is some cool stuff huh?",
    speaker=1,
    context=segments,
    max_audio_length_ms=10_000,
)

torchaudio.save("audio.wav", audio.unsqueeze(0).cpu(), generator.sample_rate)

  4. Generate audio from text with `load_csm_1b`

    main

    Use load_csm_1b to initialize the generator on a specific device (cuda, mps, or cpu). The generate method produces audio based on text input.

    When no context or specific speaker identity is provided, the model uses a random speaker identity. You can specify a speaker ID (integer) and set a max_audio_length_ms limit.

    from generator import load_csm_1b
import torchaudio
import torch

# Device selection
if torch.backends.mps.is_available():
    device = "mps"
elif torch.cuda.is_available():
    device = "cuda"
else:
    device = "cpu"

generator = load_csm_1b(device=device)

audio = generator.generate(
    text="Hello from Sesame.",
    speaker=0,
    context=[],
    max_audio_length_ms=10_000,
)

torchaudio.save("audio.wav", audio.unsqueeze(0).cpu(), generator.sample_rate)