Bark Text-to-Audio Model

Bark is a generative text-to-audio model composed of three sequential transformer models. The pipeline transforms text into audio through three distinct stages of tokenization and decoding:

1. Text to semantic tokens: Uses a causal transformer (80M parameters) to convert text (tokenized via Hugging Face's BERT tokenizer) into semantic tokens.
2. Semantic to coarse tokens: Uses a causal transformer (80M parameters) to convert semantic tokens into the first two codebooks of the EnCodec codec.
3. Coarse to fine tokens: Uses a non-causal transformer (80M parameters) to expand the first two EnCodec codebooks into the full 8 codebooks required for audio reconstruction.

This multi-stage approach allows the model to handle the complexity of language semantics before moving into the high-fidelity acoustic domain.

The prompt data is stored in .npz format (Python NumPy arrays). Each file contains three specific arrays representing different stages of the text-to-speech pipeline:

• semantic_prompt: An array of shape (n,) containing token IDs from the Hugging Face BERT tokenizer. This encodes the initial text input.
• coarse_prompt: An array of shape (2, m) containing token IDs from the first two codebooks of the Facebook EnCodec Codec. This is an intermediate representation.
• fine_prompt: An array of shape (8, p) containing 8 codebooks from the EnCodec Codec. These represent the final stage of tokenization used to generate the actual audio output.

Bark is a generative text-to-audio model that converts text directly to audio without intermediate phonemes. This allows it to interpret non-speech instructions and stylistic cues within the text prompt.

To influence the output, use the following patterns:

• Non-speech sounds: Wrap sounds in square brackets, e.g., [laughter], [laughs], [sighs], [gasps], [music], or [clears throat].
• Hesitations: Use em-dashes (—) or ellipses (...).
• Lyrics: Use the musical note symbol ♪ for song lyrics.
• Emphasis: Use CAPITALIZATION for words you want emphasized.
• Speaker Bias: Use [MAN] or [WOMAN] to bias the model toward specific genders.

Note: Because Bark is a GPT-style model, it may take creative liberties and deviate from the script in unexpected ways.

Bark is compatible with the 🤗 Transformers library (version 4.31.0+). This is useful if you want to use the model within the Transformers ecosystem.

1. Install Transformers from main: pip install git+https://github.com/huggingface/transformers.git.
2. Use AutoProcessor and BarkModel to load and run inference.

from transformers import AutoProcessor, BarkModel

processor = AutoProcessor.from_pretrained("suno/bark")
model = BarkModel.from_pretrained("suno/bark")

voice_preset = "v2/en_speaker_6"
inputs = processor("Hello, my dog is cute", voice_preset=voice_preset)

audio_array = model.generate(**inputs)
audio_array = audio_array.cpu().numpy().squeeze()

# To save with scipy:
import scipy
sample_rate = model.generation_config.sample_rate
scipy.io.wavfile.write("bark_out.wav", rate=sample_rate, data=audio_array)

The full version of Bark typically requires approximately 12GB of VRAM. If you are using a GPU with less memory (down to ~2GB), you can enable CPU offloading and use smaller models by setting specific environment variables before running your generation code.

import os
os.environ["SUNO_OFFLOAD_CPU"] = "True"
os.environ["SUNO_USE_SMALL_MODELS"] = "True"

Bark's v2 prompts are engineered to provide more consistent vocal performance. To utilize these improved prompts, include the v2/ prefix in your history_prompt argument when calling generate_audio.

Example usage:

from bark import generate_audio
text_prompt = "madam I'm adam"
# Use 'v2/' prefix to access improved voice consistency
audio_array = generate_audio(text_prompt, history_prompt="v2/en_speaker_1")

To install the official Suno Bark repository, use one of the following methods.

CRITICAL: Do NOT use pip install bark, as this installs a different, unrelated package. Use the official GitHub link instead.

# Option 1: Install directly from GitHub
pip install git+https://github.com/suno-ai/bark.git

# Option 2: Clone and install locally
git clone https://github.com/suno-ai/bark
cd bark && pip install .

Bark provides two primary mechanisms to reduce GPU memory consumption:

1. Small Models: Using smaller versions of the Bark models. This can be enabled by setting the environment variable SUNO_USE_SMALL_MODELS to "1".
2. CPU Offloading: Holding only one model on the GPU at a time and shuttling models to the CPU between generations. This can be enabled by setting the environment variable SUNO_OFFLOAD_CPU to "1".

The full version of Bark requires approximately 12GB of VRAM. If you are working with limited hardware (e.g., GPUs with <8GB VRAM), you can use a smaller version of the models by setting the following environment variable:

SUNO_USE_SMALL_MODELS=True

To run Bark with the lowest possible memory footprint, set the following environment variables before importing Bark modules:

• SUNO_USE_SMALL_MODELS="1": Uses smaller model weights.
• SUNO_OFFLOAD_CPU="1": Enables offloading models to CPU to save GPU VRAM.
• CUDA_VISIBLE_DEVICES: Specify the GPU index to use.

import os

os.environ["CUDA_VISIBLE_DEVICES"] = "0"
os.environ["SUNO_USE_SMALL_MODELS"] = "1"
os.environ["SUNO_OFFLOAD_CPU"] = "1"

from bark.generation import preload_models
from bark import generate_audio

preload_models()
audio_array = generate_audio("madam I'm adam", history_prompt="v2/en_speaker_5")

To generate long audio from a large block of text, split the text into individual sentences (e.g., using nltk.sent_tokenize). Iterate through the sentences, generating audio for each using generate_audio, and concatenate the resulting audio arrays. It is recommended to insert small segments of silence between sentences to prevent audio artifacts.

import nltk
import numpy as np
from bark import generate_audio, SAMPLE_RATE

script = "Your long text here..."
sentences = nltk.sent_tokenize(script)

SPEAKER = "v2/en_speaker_6"
silence = np.zeros(int(0.25 * SAMPLE_RATE))

pieces = []
for sentence in sentences:
    audio_array = generate_audio(sentence, history_prompt=SPEAKER)
    pieces += [audio_array, silence.copy()]

full_audio = np.concatenate(pieces)