Pipecat Examples

This repository provides implementation examples for building voice bots that integrate with the Exotel Voice API using Pipecat. It covers two primary telephony scenarios:

1. Inbound Calling: Handling incoming calls where users dial an Exotel number to interact with the bot.
2. Outbound Calling: Programmatically initiating calls where the bot dials a user.

Both scenarios leverage Exotel's WebSocket streaming to enable real-time, bidirectional audio processing through a Pipecat pipeline.

The code-helper example demonstrates advanced text processing and client-side rendering using LLMTextProcessor.

Key capabilities demonstrated:

• Categorized LLM Output: Uses LLMTextProcessor to tag LLM text so the client can render different types (e.g., code snippets, links) and the TTS can handle them uniquely (e.g., spelling out credit card numbers, skipping 'https' in URLs, or ignoring code snippets).
• Client-Bot Communication: Uses RTVI for seamless communication, including handling user and bot transcripts via user-transcript and bot-output messages.
• Text Input: Demonstrates how a client can send text input to the server, which the bot can respond to with either audio or text.
• Tool Calling: Shows how to handle sensitive information during tool calls for both TTS and RTVI purposes.

The Simple Chatbot example demonstrates a real-time AI chatbot with audio/video interaction. It is designed to showcase the flexibility of Pipecat by providing multiple server-side AI backends and various client-side SDK implementations.

To use this project, you must first run a bot server, which then allows different client applications to connect and interact with the AI.

This project demonstrates how to connect Vonage to a Pipecat-powered conversational bot using FastAPI and WebSockets. It implements a standard Pipecat pipeline (STT → LLM → TTS) with telephony-grade audio framing. It supports two distinct operational modes:

1. Vonage Video API (Audio Connector): Streams audio from an existing Vonage Video session into the bot.
2. Vonage Voice API (Phone Calls): Connects incoming phone calls directly to the bot via a linked Vonage number.

Both modes utilize a WebSocket connection to a /ws endpoint to exchange raw PCM audio frames.

The Gemini Bot Server is a Pipecat implementation of Google's Gemini Live Multimodal model. It is designed for real-time, multimodal interaction.

Key Capabilities:

• Gemini Live Multimodal: Native audio integration for real-time conversation.
• Screen Sharing Analysis: The bot can see, analyze, and reason about shared screens (e.g., guessing locations on Google Maps based on visual content).
• Real-time Conversation: Uses Voice Activity Detection (VAD) for natural interaction.
• RTVI Events: Emits RTVI events to allow client-side UI integration.
• Daily Transport: Uses WebRTC via Daily for communication.

The Storytelling Chatbot is a voice-driven, interactive 'choose your own adventure' experience. It uses a multimodal pipeline to generate a story and corresponding illustrations in real-time.

Key Features:

• Dynamic Storytelling: Google Gemini writes the story in segments.
• Visual Illustration: A Gemini image model generates illustrations for each story page, which are streamed to the client as a video track.
• Voice Interaction: Uses Deepgram for Speech-to-Text (STT), ElevenLabs for Text-to-Speech (TTS), and Gemini for the LLM logic.
• Asynchronous Processing: The StoryImageProcessor handles image generation in a background task so that audio narration is not delayed by image generation latency.

Pipeline Architecture: transport.input() → stt → user_aggregator → llm → story_processor → image_processor → tts → transport.output() → assistant_aggregator

This repository provides implementation examples for building voice bots that integrate Plivo's Voice API with Pipecat. It specifically utilizes Plivo's WebSocket streaming capabilities to enable real-time bidirectional audio processing between a phone call and AI services.

There are two primary interaction patterns supported:

1. Inbound Calling: Handling calls made by users to a Plivo phone number.
2. Outbound Calling: Programmatically initiating calls from the bot to a user.

The architecture follows this data flow: Phone Call ↔ Plivo ↔ WebSocket Stream ↔ Pipecat ↔ AI Services

This repository provides examples for building voice bots that integrate Telnyx's Voice API with Pipecat. It covers two primary interaction patterns:

1. Inbound Calling: Handling incoming calls where users dial a Telnyx number to interact with a bot.
2. Outbound Calling: Programmatically initiating calls where the bot dials a user.

These examples utilize Telnyx's TeXML for call control and WebSocket streaming to enable real-time audio processing through Pipecat.

The SageMaker wrapper for the NVIDIA Nemotron ASR Streaming NIM (nemotron-asr-streaming) translates SageMaker's expected interface to the NIM's actual API.

Because Nemotron ASR Streaming is a realtime model, batch transcription via POST /invocations is not supported and will return a 410 Gone error. Instead, you must use the WebSocket bidirectional stream endpoint.

The aws-strands folder provides two distinct ways to interact with a Strands agent using Pipecat:

• black-box.py: A minimal implementation. The agent handles multi-step queries by calling tools, but it performs its reasoning silently without verbalizing its thought process.
• explain-thinking.py: An enhanced implementation. The agent explains each step of its reasoning in clear, simple language as it works through a multi-step task.

Both examples are simplified for demonstration and use mock API calls. They are optimized for complex queries like: "What's the weather where the Golden Gate Bridge is?"

Examples in this repository are organized by use case. Common categories include:

• Telephony & Voice Calls: Integration with Daily PSTN, SIP, Twilio, Telnyx, Plivo, and Exotel, including IVR navigation.
• Web & Client Applications: React/JavaScript/Swift/Kotlin/React Native clients, Push-to-Talk (using Voice UI Kit), WebSockets, MoQ (Media over QUIC), and P2P WebRTC.
• Realtime APIs: Gemini Live integrations and AWS Strands for multi-step tasks.
• Multimodal & Creative: Illustrated storytelling using SmallWebRTC and Next.js.
• Translation & Localization: Multi-language conference calls.
• Advanced Features: Smart turn-taking, custom audio/video track handling, and local input selection.
• Deployment: Examples for Pipecat Cloud, Fly.io, Modal, and Cerebrium.
• Monitoring: Observability using OpenTelemetry with Langfuse, LangSmith, and Jaeger.

This deployment example demonstrates how to deploy a FastAPI webapp to Modal.com with an RTVI-compatible /connect endpoint. The endpoint launches a Pipecat pipeline in a separate Modal container and returns a room/token for the client to join.

Supported pipeline types via the /connect endpoint parameter:

• openai
• gemini
• vllm (points to a self-hosted OpenAI-compatible LLM deployed to Modal).

Architecture Note: Unlike other Pipecat examples that use Popen to launch processes, this example uses a Modal function to launch pipelines. This allows pipelines to run in isolated containers using specifically defined Modal images.