Fish Speech

Fish Audio S2 Pro supports sub-word level control of prosody and emotion using a [tag] syntax. You can insert emotional instructions anywhere in the text to influence the voice output.

• Supported Tags: Over 15,000 unique tags.
• Free-form Descriptions: You can use natural language descriptions such as [whisper in small voice], [professional broadcast tone], or [pitch up] to achieve specific effects.

Fish Audio S2 Pro allows sub-word level control of prosody and emotion using natural language tags in the format [tag]. You can insert these anywhere in your text to guide the model's expression.

Capabilities:

• Free-form descriptions: Supports over 15,000 unique tags, including descriptive text like [whisper in small voice] or [professional broadcast tone].
• Common emotion/action tags:
  • [pause], [emphasis], [laughing], [inhale], [chuckle], [tsk], [singing], [excited], [laughing tone], [interrupting], [chuckling], [excited tone], [volume up], [echo], [angry], [low volume], [sigh], [low voice], [whisper], [screaming], [shouting], [loud], [surprised], [short pause], [exhale], [delight], [panting], [audience laughter], [with strong accent], [volume down], [clearing throat], [sad], [moaning], [shocked]

Fish Audio S2 allows for native multi-speaker generation. Instead of uploading new reference audio for every speaker change, you can use the <|speaker:i|> token to control which speaker is active. This enables mixing multiple speakers within a single generation task by providing reference audio for each speaker ID.

<|speaker:0|> Hello, I am the first speaker. <|speaker:1|> And I am the second speaker.

Fish Audio S2 Pro supports sub-word level control of prosody and emotion using a [tag] syntax. You can embed these tags anywhere in your text to provide fine-grained instructions.

Key Features:

• Free-form descriptions: Supports over 15,000 unique tags, including descriptive text like [whisper in small voice] or [professional broadcast tone].
• Common emotion/instruction tags:
  • Pauses/Breathing: [pause], [short pause], [inhale], [exhale], [clearing throat]
  • Emotions: [excited], [angry], [sad], [surprised], [delight], [shocked]
  • Vocal Styles: [laughing], [chuckle], [sigh], [whisper], [screaming], [shouting], [singing]
  • Volume/Pitch: [emphasis], [volume up], [volume down], [low volume], [low voice], [pitch up]
  • Other: [tsk], [interrupting], [echo], [panting], [audience laughter]

The S2 Pro model utilizes a Dual-Autoregressive (Dual-AR) architecture:

1. Slow AR (4B parameters): Responsible for modeling the temporal structure of the audio.
2. Fast AR (400M parameters): Responsible for reconstructing fine acoustic details.

The model also utilizes Reinforcement Learning (RL) alignment via GRPO with multidimensional reward signals to ensure high-quality, natural output.

The S2 model uses a Dual-AR architecture based on a decoder-only transformer and an RVQ-based audio codec (10 codebooks, ~21 Hz frame rate). Generation is split into two asymmetric stages to balance efficiency and fidelity:

1. Slow AR: Operates along the time axis and predicts the primary semantic codebook (4B parameters).
2. Fast AR: Generates the remaining 9 residual codebooks at each time step to reconstruct fine-grained acoustic details (400M parameters).

This design allows for high-fidelity audio while maintaining efficient inference.

S2 Pro uses a master-slave Dual-AR architecture consisting of a Decoder-only Transformer and an RVQ audio codec (10 codebooks, ~21 Hz). This design separates semantic prediction from acoustic detail restoration:

1. Slow AR (4B parameters): Operates on the time axis to predict the core semantic codebook.
2. Fast AR (400M parameters): Generates the remaining 9 residual codebooks at each time step to restore fine acoustic details.

This asymmetry allows for high audio fidelity while significantly improving inference speed.

Fish Audio S2 supports advanced multi-speaker and conversational capabilities:

Multi-speaker Generation

You can upload reference audio containing multiple speakers. The model uses <|speaker:i|> tokens to handle speaker features. You can then control which speaker is active in a single generation sequence by using the corresponding speaker ID tokens.

Multi-turn Dialogue

The model's extended context window allows it to use information from previous turns to improve the expressiveness and naturalness of subsequent generations, making it suitable for long-form dialogue.

Fast Voice Cloning

Voice cloning can be achieved using short reference samples (typically 10-30 seconds). The model captures timbre, speaking style, and emotional tendencies without requiring additional fine-tuning.

Fish Audio S2 provides advanced capabilities for complex dialogue and speaker management:

Native Multi-Speaker Generation

You can upload a single reference audio containing multiple speakers. The model identifies features for each speaker via the <|speaker:i|> token. You can then control which speaker performs which part of the text using these speaker ID tokens, allowing for multi-speaker dialogue in a single generation without needing separate reference files for every individual.

Multi-Turn Generation

The model's expanded context allows it to leverage information from previous turns in a conversation. This improves the expressiveness and naturalness of subsequent generated content by maintaining conversational flow.

S2 Pro uses a master-slave Dual-AR architecture consisting of a Decoder-only Transformer and an RVQ audio codec (10 codebooks, ~21Hz frame rate) to balance audio fidelity and inference speed.

• Slow AR (4B parameters): Operates along the time axis to predict the core semantic codebook.
• Fast AR (400M parameters): Generates the remaining 9 residual codebooks at each timestep to restore fine-grained acoustic details.