DOCSEARCHbyAlgolia
Submit Docs

Fish Audio Developer Documentation

website·Indexed 51 days ago

https://docs.fish.audio/overview/capabilities

Documentation for Fish Audio's text-to-speech and voice cloning platform. Includes SDK guides for Python and JavaScript, API references for s1, s2-pro, and s2.1-pro models, and integration guides for LiveKit and Telnyx. Covers real-time streaming, emotion control, phoneme-level fine-grained control, and migration paths from OpenAI and ElevenLabs.

Tokens
104.8K
Snippets
414
Records
693
Agent score
69%

What's inside Fish Audio

  1. Overview of Fish Audio Enterprise self-hosting options

    Fish Audio Enterprise can be deployed on infrastructure controlled by the user (cloud, on-premise, or air-gapped). The self-hosted stack uses the same speech engine as the hosted API, ensuring consistent requests, audio formats, and voice behavior. There are two primary deployment platforms:

    1. Helm (Kubernetes): Best for production clusters and elastic capacity. Supports replicas across GPUs and nodes with rolling upgrades.
    2. All-in-One (Docker): A single container on a single machine with the smallest footprint. Best for evaluation, fixed-size appliances, or strict air gaps.

  2. Overview of Fish Audio Agent configuration sections

    The Agent Builder configuration is divided into several key sections that define the agent's behavior, identity, and integrations. These sections include the system prompt, voice profiles, conversation logic, tool integrations, knowledge sources, post-call analysis, webhook delivery, and LLM settings.
    {
  "prompt": "System prompt and first-message settings",
  "voice": "Voice profile, speaking language, and expressive mode",
  "conversation": "Call duration, turn-taking, interruption behavior, timezone, and storage",
  "tools": "Attached webhook tools and system tool switches",
  "knowledge_base": "Attached knowledge sources",
  "analysis": "Post-call summary, data fields, and success criteria",
  "webhooks": "Post-call webhook delivery and the inbound call webhook",
  "llm": "Conversation model settings, including a custom LLM endpoint"
}

  3. Overview of Post-Call Analysis

    After a conversation ends, the platform automatically runs an LLM pass over the stored transcript to produce three types of results:

    1. Summary: A concise recap of the conversation (enabled by default).
    2. Data Fields: Structured values extracted from the transcript (opt-in).
    3. Success Criteria: A verdict on whether specific goals were met (opt-in).

    Analysis runs automatically for production sessions and preview calls. Results are typically ready within seconds and move through the following statuses: queued, running, completed, skipped (if disabled or no user messages), or error.

  4. Overview of Fish Agents platform

    Fish Agents is a hosted platform for building real-time voice agents. It manages speech recognition, turn-taking, and speech synthesis automatically. Developers can define an agent's system prompt, voice, knowledge base, and tools. Agents can be deployed via a web app (using SDKs), over phone numbers, or via a REST API. Note: The platform is in private beta; accounts without granted access will receive a 403 error for /v1/agent/* requests.

  5. Overview of Fish Audio Agent Webhooks

    Fish Audio agents can be configured to send webhook events to your server to notify you of session states without polling the sessions API. These events are useful for updating CRMs, ticketing systems, or data warehouses.

    Key events include:

    • call.ended: Fires when a session reaches a terminal state (every real call).
    • call.analyzed: Fires when post-call analysis settles (completed, skipped, or error).
    • phone_call.dial_finished: Fires only for outbound calls when the dial attempt resolves (answered, busy, no answer, or failed).

    Order of delivery: For answered production calls, phone_call.dial_finished fires first, followed by call.ended, and finally call.analyzed. Transcripts are excluded from payloads and must be fetched via GET /v1/agent/sessions/{session_id}.

  6. Overview of Fish Audio All-in-One appliance

    The All-in-One image is a turnkey single-node appliance that packages the entire speech stack into one container, including the edge API, model API layer, inference router and worker, vocoder, text normalizer, and Redis. Model weights are typically baked into the image unless using external-weights modes. It is designed for single-node deployment and does not support Kubernetes, autoscaling, or sharding across multiple GPUs/nodes. Note that it does not include the forced aligner, meaning it cannot return word or segment timings.

  7. Use Fish Audio S2.1-Pro for production TTS

    S2.1-Pro is the recommended production text-to-speech model. It features improved quality, latency, and throughput compared to S2-Pro, and provides TTFA (Time To First Audio) and DPA guarantees for production workloads. It supports 83 languages with automatic language detection and multi-speaker dialogue.

  8. Deploy Fish Audio agents to different channels

    Fish Audio agents can be deployed across multiple front-end surfaces. All surfaces run the same agent using the same published configuration. You must publish your agent before it can be used in any channel; creating a session against an unpublished agent will fail.

  9. Retrieve an agent session by ID

    Use the GET /v1/agent/sessions/{session_id} endpoint to retrieve detailed information about a specific agent session. This includes the session's current status, call direction, duration, a list of transcript items (messages, tool calls, and tool results), and the post-call analysis summary.
    curl --request GET \
  --url https://api.fish.audio/v1/agent/sessions/{session_id} \
  --header 'Authorization: Bearer <token>'

  10. Understand Fish Audio Agent core concepts

    The Fish Audio platform is organized around four primary entities:

    • Workspace: The top-level isolation boundary. All agents, tools, knowledge sources, and tests belong to a workspace. Resources like tools and knowledge sources are shared at the workspace level and can be attached to multiple agents.
    • Agent: A configured voice assistant consisting of a system prompt, first-message policy, voice/language settings, tools, and knowledge sources.
    • Version: An immutable snapshot of an agent's configuration created when a draft is published.
    • Session: A single conversation between a user and an agent. Sessions can originate from the Web/React SDKs, phone calls, public agent links, or Builder preview calls.

  11. Understand the Fish Audio Enterprise Kubernetes stack components

    The Fish Audio Helm chart installs a complete speech stack into a single namespace. The deployed components include:

    • Edge API
    • Model API layer
    • Inference router and worker
    • Vocoder
    • Text normalizer
    • Model warehouse (serves weights for default in-cluster source)
    • Redis

    By default, Ingress and TLS are disabled. Users must provide their own ingress controller/load balancer in front of the edge API service or enable the chart's built-in ingress management.

  12. Fish Audio API Protocol Compatibility Overview

    Fish Audio provides compatibility layers for two major protocols under https://api.fish.audio/compat:

    1. OpenAI Protocol: Available at /v1, /api/v1, and /openai/v1. Compatible with OpenAI, OpenRouter, and Groq SDKs.
    2. ElevenLabs Protocol: Available at /elevenlabs.

    Parameter Handling Logic:

    • Mapped: Parameters are converted to native equivalents (e.g., stability → temperature, speed → speech rate).
    • Ignored: Parameters that do not affect synthesis (e.g., similarity_boost, style, seed) are ignored.
    • Refused: Parameters that would change response content or structure return a 4xx error.
    • Unknown Fields: JSON fields not recognized by the protocol are ignored (matching OpenAI behavior).