Cartesia Documentation

website·Indexed 31 days ago

https://docs.cartesia.ai/get-started/overview

Technical documentation for Cartesia's AI platform, covering API references for datasets and fine-tuning, self-hosted deployment via Docker, Kubernetes, and air-gapped environments, and implementation guides for Sonic models, WebSocket streaming, and the Line agent platform.

Tokens
203.8K
Snippets
529
Records
896
Agent score
49%

What's inside Cartesia

  1. Overview of Ink Whisper STT model

    Ink Whisper is Cartesia's most affordable speech-to-text (STT) model. It is designed to provide higher accuracy and lower latency than the baseline Whisper model.

    Key capabilities include:

    • Dynamic Chunking: Gracefully handles variable-length audio chunks and interruptions.
    • Noise Robustness: Reliably transcribes speech even with background noise.
    • Audio Artifact Handling: Accurately transcribes audio containing telephony artifacts, accents, and disfluencies.
    • Domain Expertise: Excels at transcribing proper nouns and domain-specific terminology.
  2. Overview of Ink 2 streaming speech-to-text model

    Ink 2 is a high-performance streaming speech-to-text (STT) model designed for production voice agents. It features low word error rates and built-in turn detection, eliminating the need for a separate Voice Activity Detection (VAD) system.

    Key capabilities:

    • Structured Data Transcription: Accurately transcribes phone numbers, dates, and emails.
    • Built-in Turn Detection: Emits a full lifecycle of turn events to manage agent interaction states.
    • Turn Event Lifecycle: The model emits turn.start, turn.update, turn.eager_end, turn.resume, and turn.end events.

    Model Details:

    • Model ID: ink-2
    • Status: Stable
    • Supported Languages: en (English)
  3. Overview of Cartesia Line for Voice Agents

    Cartesia Line is a platform designed to bring voice capabilities to text-based agents. It manages audio orchestration, deployment, and observability, allowing developers to focus on agent reasoning.

    Key technical components:

    • Audio Orchestration: Managed runtime with auto-scaling and low-latency audio processing.
    • Speech-to-Text (STT): Powered by Ink.
    • Text-to-Speech (TTS): Powered by Sonic.

    Developers can interact with Line via two primary methods:

    1. Agent Builder: A no-code environment for prototyping and iterating on agents.
    2. SDK: A programmatic approach for writing custom reasoning logic in code.
  4. Overview of Cartesia Sonic 3.5 TTS Model

    Sonic 3.5 is a high-speed, natural text-to-speech model designed for conversational delivery and low latency (sub-90ms).

    Key capabilities include:

    • 42 Languages: Native quality support for English, Spanish, French, German, Japanese, Hindi, and many others.
    • Natural Alphanumerics: Handles order numbers, phone numbers, IDs, and emails without manual preprocessing.
    • Context-Aware Pronunciation: Correctly handles English heteronyms (e.g., 'read', 'bass', 'bow') based on surrounding text.
    • Expressive Delivery: Tuned for support agents and conversational transcripts with strong pacing and emotional range.
  5. Overview of Cartesia AI Models (Sonic and Ink)

    Cartesia provides two primary model families for building conversational AI experiences:

    Sonic Models (Text-to-Speech)

    Sonic models convert text input into streamed, ultra-realistic speech. They support voice cloning and provide control over pronunciation and accent.

    • Sonic 3.5: Optimized for real-time and conversational use cases. It can stream the first byte of audio in approximately 90ms.
    • Use cases: Real-time conversational AI, dubbing, narration, and AI avatars.

    Ink Models (Speech-to-Text)

    Ink models provide streaming speech-to-text transcription optimized for real-time voice agents.

    • Ink 2: Features native turn detection, using context to intelligently determine when a user has finished speaking and when an agent should respond.
    • Use cases: Real-time voice agents and transcription services.
  6. Overview of Cartesia Self-Hosted Deployment Options

    Cartesia models can be deployed into customer-provisioned environments, including GCP, AWS, or on-premise data centers. Self-hosting is intended for use cases requiring specific latency, isolation, or security requirements.

    Key benefits include:

    • Colocation: Reduce network latency by hosting models near your existing services.
    • Isolation: Achieve single-tenant isolation.
    • Security: Maintain a tight security posture by keeping traffic off the public internet. Self-hosted deployments only contact Cartesia servers for authentication and usage reporting (metadata like character count and voice ID, but no transcript data).
    • Air-gapped Support: Support for deployments with no cloud contact via offline licenses.
    • Sovereignty: Host in any geographic region to meet jurisdictional requirements.
  7. Overview of Cartesia and Tencent TRTC Integration

    Tencent Real-Time Communication (TRTC) and Cartesia have a public partnership to build low-latency conversational AI. This integration combines Tencent's TRTC networking stack with Cartesia's Sonic Text-to-Speech (TTS) and Ink-Whisper Speech-to-Text (STT) to power real-time voice agents for calls, live streaming, and conferencing.

    Note: Specific integration steps, SDK details, and configuration must be performed within the Tencent Cloud console and documentation.

  8. Monitor Cartesia inference cluster metrics with Prometheus

    Cartesia’s self-hosted inference cluster supports Prometheus for monitoring. Metrics are scraped every 5 seconds via a PodMonitor on port 8080 at the /metrics endpoint.
  9. Retrieve a Call Batch by ID

    Use the GET /agents/calls/batches/{batch_id} endpoint to retrieve detailed information about a specific call batch, including its status, concurrency limits, and a list of individual recipients and their call statuses.

    Request Headers

    • Authorization: Bearer token (sk_car_...).
    • Cartesia-Version: The API version (e.g., 2026-03-01).

    Path Parameters

    • batch_id (string, required): The unique identifier of the batch to retrieve.
    curl --request GET \
      --url https://api.cartesia.ai/agents/calls/batches/{batch_id} \
      --header 'Authorization: Bearer <token>' \
      --header 'Cartesia-Version: 2026-03-01'
  10. How LlmAgent processes event history

    When using the LlmAgent class, the SDK automatically converts the event history into a format compatible with LLM message structures. This ensures the LLM has full context of the conversation without manual mapping:

    • User messages: Derived from UserTextSent events.
    • Assistant messages: Derived from AgentTextSent events.
    • Tool calls: Derived from AgentToolCalled and AgentToolReturned events.

    This automatic conversion allows the LLM to reference previous tool calls and results directly.

  11. Migrate from Cartesia Line v0.1.x to v0.2

    Upgrading from v0.1.x to v0.2 involves moving from a manual event routing system (VoiceAgentSystem + Bridge) to an automated system using VoiceAgentApp with a get_agent callback.

    Key Changes Overview

    Featurev0.1.xv0.2
    Core ArchitectureVoiceAgentSystem + Bus + BridgeVoiceAgentApp with get_agent callback
    Agent LogicReasoningNode subclassesLlmAgent or custom Agent protocol
    Handler Signaturecall_handler(system, request)get_agent(env, request) -> Agent
    Event RoutingManual event routing via bridge.on()Automatic event dispatch with filters
    Processing Methodprocess_context() methodprocess(env, event) async iterable

    Migration Steps

    1. Update Imports: Replace old node and system imports with the new line.llm_agent and updated line.events modules.
    2. Replace System with get_agent: Instead of manually routing events like UserTranscriptionReceived or DTMFInputEvent using a bridge, define a get_agent function that returns an agent instance.
    3. Implement Filters: Use Run and Cancel filters to control agent triggers and interruptions instead of manual bridge.on() calls.
    # v0.2 Import Pattern
    from line.voice_agent_app import VoiceAgentApp, AgentEnv
    from line.llm_agent import LlmAgent, LlmConfig, end_call, transfer_call, loopback_tool, passthrough_tool
    from line.events import (
        AgentSendText,
        AgentEndCall,
        AgentTransferCall,
        UserTurnEnded,
        CallStarted,
    )
  12. Integrate Tavily web search into Cartesia Line voice agents

    You can add live web search and page extraction capabilities to a Cartesia Line voice agent using loopback tools backed by Tavily. This allows the agent to answer questions about current events, fresh facts, and specific URLs during a live call.

    Prerequisites

    • Python 3.10+
    • CARTESIA_API_KEY
    • TAVILY_API_KEY
    • An LLM provider key (e.g., OPENAI_API_KEY)
    • Cartesia CLI installed for local testing

    Installation

    pip install cartesia-line tavily-python

    Implementation Example

    To implement this, define a class containing loopback_tool methods that wrap the AsyncTavilyClient. Use search_depth="fast" to maintain low latency suitable for voice interactions.

    from typing import Annotated, Optional
    from tavily import AsyncTavilyClient
    from line.llm_agent import LlmAgent, LlmConfig, ToolEnv, end_call, loopback_tool
    from line.voice_agent_app import AgentEnv, CallRequest, VoiceAgentApp
    
    class TavilyTools:
        def __init__(self, api_key: str):
            # Reusing the client ensures the httpx connection pool is reused across tool calls
            self._client = AsyncTavilyClient(api_key=api_key, client_source="cartesia-line-agent")
    
        @loopback_tool
        async def web_search(
            self,
            ctx: ToolEnv,
            query: Annotated[str, "The search query. Be specific."],
            time_range: Annotated[Optional[str], "Optional recency filter: 'day', 'week', 'month', or 'year'."] = None,
        ) -> str:
            """Search the web for current information."""
            # 'fast' depth is recommended for voice latency
            kwargs: dict = {"query": query, "search_depth": "fast", "max_results": 5}
            if time_range is not None:
                kwargs["time_range"] = time_range
            
            response = await self._client.search(**kwargs)
            results = response.get("results", [])
            if not results:
                return "No relevant information found."
    
            parts = [f"Search results for: '{query}'\n"]
            for i, r in enumerate(results, start=1):
                parts.append(f"\n--- Source {i}: {r['title']} (score {r.get('score', 0):.2f}) ---\n")
                if r.get("content"):
                    parts.append(f"{r['content']}\n")
                parts.append(f"URL: {r['url']}\n")
            return "".join(parts)
    
        @loopback_tool
        async def web_extract(
            self,
            ctx: ToolEnv,
            url: Annotated[str, "The URL to extract content from."],
        ) -> str:
            """Extract the full content of a webpage given its URL."""
            response = await self._client.extract(urls=[url])
            results = response.get("results", [])
            if not results:
                return "No content could be extracted."
    
            raw = results[0].get("raw_content", "")
            # Truncate to keep LLM context tight
            EXTRACT_MAX_CHARS = 3000
            if len(raw) > EXTRACT_MAX_CHARS:
                raw = raw[:EXTRACT_MAX_CHARS] + "\n\n[Content truncated]"
            return f"Extracted content from {url}:\n\n{raw}"

    Running the Agent

    1. Start the application: python main.py
    2. In a separate terminal, connect via CLI: cartesia chat 8000