DOCSEARCHbyAlgolia
Submit Docs

Ultravox Documentation

website·Indexed 34 days ago

https://docs.ultravox.ai

Developer documentation for building and managing voice AI experiences. Includes guides on creating reusable agents, managing call stages, handling telephony credentials, and using the Ultravox API for call management, concurrency usage, and scheduled call batches.

Tokens
56.5K
Snippets
289
Records
422
Agent score
68%

What's inside Ultravox

  1. Overview of Ultravox Voice Activity Detection (VAD)

    Ultravox uses a multi-layered VAD system to determine when a user is speaking and when a turn has ended. It combines a traditional VAD model (analyzing 32ms audio frames for 'speechiness'), a Neural VAD model (predicting conversation state and turn-taking patterns), and audio processing models for noise cancellation, echo cancellation, and false positive reduction.

  2. Overview of Inline Instructions for Agent Guidance

    Inline instructions are used to guide an agent through a conversation step-by-step using tool responses and deferred messages. This approach is recommended when 'mono prompting' (a single system prompt) fails to ensure the agent completes complex sequences or follows specific steps (e.g., a 10-question intake form). Unlike Call Stages, which are for non-overlapping phases, inline instructions layer guidance into the existing prompt to maintain a natural conversational flow while keeping the agent 'on rails'.

  3. Overview of Ultravox Call Stages

    Call Stages allow you to segment a conversation into distinct phases. Each stage can have its own system prompt, set of tools, voice, and conversation history. This enables the agent to adapt its focus and behavior as the conversation progresses. New stages inherit all properties from the previous stage unless explicitly overridden.

  4. Overview of Outbound Call Scheduler

    The Outbound Call Scheduler manages batches of outbound calls with automatic concurrency control and retry handling. It eliminates the need for manual rate-limit management (429 errors) and allows developers to define specific time windows for call execution.

    Key features include:

    • Automatic Concurrency Management: Prevents rate-limit errors.
    • Flexible Scheduling: Define start and end times for campaigns.
    • Capacity Reservation: Ensures room for high-priority or incoming calls.
    • Batch Tracking: Monitor progress and individual call outcomes.

    Important Constraint: Call delivery is not guaranteed if a windowEnd is specified. If capacity limits prevent a call from being made before the window closes, the call status will be marked as EXPIRED.

  5. Overview of Ultravox Threads

    Threads allow developers to spawn independent, parallel conversations alongside a main call. Each thread forks the parent's conversation history, system prompt, and tool state. They are primarily used for background processing, delegated tasks, or monitoring (Observer pattern) to ensure the main 'UI' thread remains responsive.

    Key constraints:

    • Only the UI thread (ID: UI) can receive audio input, produce audio output, end the call, or start a stage change.
    • Background threads can influence the UI thread by passing messages to it.
    • All side threads are canceled when the call ends or a stage change occurs.

  6. Overview of supported telephony platforms

    Ultravox Realtime integrates with several major telephony providers to enable AI agents to make and receive phone calls. Each provider requires a specific medium object when creating calls. Note that SIP connectivity is handled separately via the SIP Guide.
    ProviderCall MediumStreaming APIImport Credentials?HD audioOut of Band DTMF?Cold transfer?
    Twilio{"twilio": {}}Media Streams✅❌❌✅
    Telnyx{"telnyx": {}}Media Streaming✅✅✅❌
    Plivo{"plivo": {}}AudioStream✅✅✅❌
    Exotel{"exotel": {}}Voice Streaming❌❌❌❌

  7. Overview of Ultravox Realtime Architecture

    Ultravox Realtime is a voice AI platform that processes speech directly using an open-weight model, bypassing traditional Automatic Speech Recognition (ASR) pipelines. This architecture reduces latency and improves context awareness by eliminating the speech-to-text stage. The platform is API-first, allowing for dynamic agent configuration, custom tool integration, and deployment across phone calls, web apps, and native apps.

  8. Overview of Ultravox Realtime capabilities

    Ultravox Realtime is a fully hosted platform for creating real-time, low-latency voice AI applications based on an open-weight model. Key capabilities include:

    • Multi-lingual Support: Support for 26 spoken languages including English, Chinese, Spanish, French, German, Japanese, and others.
    • Tools (Function Calling): Ability for agents to connect to external systems via function calling.
    • Knowledge (RAG): Integration of custom, contextual knowledge to enhance agent responses.
    • Call Stages: Support for advanced agent branching and conversation flows.
    • Conversation History: Access to audio recordings and full text transcripts.
    • Voice Cloning: Ability to create custom voices for agents.
    • Telephony Integrations: Quick integration with telephony partners.
    • Webhooks: Real-time notifications for key system events.

  9. Overview of Ultravox Built-in Tools

    Ultravox Realtime provides several built-in tools for common voice application functionality. These tools follow the same definition structure as custom tools and can be overridden by creating a custom tool with the same name. To view the complete specifications of all available tools (built-in and custom), use the List Tools API.
    curl -X GET "https://api.ultravox.ai/api/tools" \
  -H "X-API-Key: your-api-key"

  10. Understand the purpose of Ultravox Client Tools

    Client tools in Ultravox enable direct interaction between AI agents and the frontend application. Unlike server-side (HTTP) tools that handle backend operations, client tools are specifically designed for real-time UI updates, frontend state management, processing user actions, and dispatching custom events.

  11. Use the Outbound Call Scheduler for high-volume campaigns

    Available to paid subscription accounts, the Outbound Call Scheduler (Expected July 2025) abstracts concurrency management. Instead of handling 429s and load balancing manually, users provide a time window, a list of destination phone numbers, and the desired agent. Ultravox handles the queuing, load balancing, and retries automatically. This is ideal for marketing campaigns, proactive follow-ups, and appointment reminders.

  12. Use the Ultravox Web Console

    The Ultravox Web Console, available at https://app.ultravox.ai, provides a centralized interface for managing voice agents and account settings. Key features include:

    • Agent Builder: Build and iterate on voice agents using 'Blocky'.
    • Dashboard: Monitor usage charts and live call counts.
    • Call History: Access timestamps, durations, summaries, and full transcripts; copy call IDs for REST API troubleshooting.
    • Voices Explorer: Discover built-in voices or create custom ones.
    • Tools: Create, edit, and test agent tools.
    • RAG: Manage knowledge bases (corpora) for company-specific information.
    • Webhooks: Configure integrations and automations.
    • Settings: Manage API keys, custom TTS keys, and billing/invoices.