Pipecat Framework

Pipecat supports a wide range of third-party services across multiple categories to build multimodal AI agents. These services include providers for speech processing, language models, transport layers, and more.

Service Categories:

• Speech-to-Text (STT): Converts audio to text (e.g., AssemblyAI, Deepgram, OpenAI Whisper, Google).
• LLMs: Large Language Models for reasoning (e.g., Anthropic, OpenAI, Gemini, Groq, Ollama).
• Text-to-Speech (TTS): Converts text to audio (e.g., ElevenLabs, Cartesia, Deepgram, OpenAI).
• Speech-to-Speech (S2S): End-to-end audio processing (e.g., OpenAI Realtime, Gemini Multimodal Live, Ultravox).
• Transport: Methods for connecting users (e.g., Daily WebRTC, LiveKit, WebSocket, WhatsApp).
• Serializers: Telephony integrations (e.g., Twilio, Vonage, Telnyx).
• Video: Digital human/avatar services (e.g., HeyGen, Simli, Tavus).
• Memory: Long-term context (e.g., mem0).
• Vision & Image: Visual understanding and generation (e.g., fal, Google Imagen, Moondream).
• Audio Processing: Utilities for noise reduction and VAD (e.g., Silero VAD, Krisp Viva, RNNoise).
• Analytics & Metrics: Observability (e.g., OpenTelemetry, Sentry).

LLM service implementation depends on whether the provider is OpenAI-compatible:

• OpenAI-Compatible Services: Inherit from OpenAILLMService. (e.g., Azure, Grok).
• Non-OpenAI Compatible Services: Requires a full implementation of the service and a custom adapter_class (subclassing BaseLLMAdapter). (e.g., Anthropic, Google).

Key Requirements:

• _process_context(self, context: LLMContext): The main method to override. It processes context and generates a response. Each service overrides process_frame to extract context from LLMContextFrame and call _process_context.
• adapter_class: Must implement methods like get_llm_invocation_params(context), to_provider_tools_format(tools_schema), and get_messages_for_logging(context).
• Standard Frame Sequence: Output must follow: LLMFullResponseStartFrame $\rightarrow$ LLMTextFrame $\rightarrow$ LLMFullResponseEndFrame.
• Reasoning/Thought Frames: For models with chain-of-thought, emit: LLMThoughtStartFrame $\rightarrow$ LLMThoughtTextFrame $\rightarrow$ LLMThoughtEndFrame alongside the response.

Note: Context aggregation is handled by the framework via LLMContext and LLMContextAggregatorPair; you do not need to implement aggregators.

The 'Every Input Acts' pattern is used when every user voice turn should drive a UI update, even if the voice assistant doesn't speak. In this pattern, the voice layer is a standard transport → STT → LLM → TTS pipeline that does not mutate the UI state directly.

Key Workflow:

1. Dispatching Updates: Instead of the LLM deciding to call a tool to change the UI, the voice layer's user aggregator fires an on_user_turn_stopped handler. This handler dispatches the transcript to a UIWorker as a respond job using a bus message: worker.job("ui", name="respond", payload={"query": transcript}).
2. Snapshot-Driven Action: The UIWorker uses an LLM to resolve relative terms (like "the last one" or "the milk") by injecting the current <ui_state> into the prompt via the on_before_process_frame hook. This ensures the worker acts against the live state of the UI.
3. Shared Source of Truth: The UIWorker manages the state, while the voice layer uses a read-only tool (e.g., check_list) to read the ListWorker.list_summary snapshot. This allows the voice assistant to answer questions like "what's left?" based on the actual UI state, including manual user edits.

# Dispatching a transcript to the UIWorker from the voice layer
worker.job("ui", name="respond", payload={"query": transcript})

Services can broadcast metadata via service_metadata_frame() immediately after the StartFrame. This allows other pipeline components to auto-configure themselves.

Key Metadata Fields

• service_name: The name of the service.
• user_turn_strategies: Recommended turn strategies. If a service performs server-side end-of-turn detection, return ExternalUserTurnStrategies(). This tells the user aggregator to defer to the service's turn frames instead of local VAD.

Subtype Metadata

• STTMetadataFrame: Includes ttfs_p99_latency (99th percentile time from end-of-speech to final transcript).
• LLMServiceMetadataFrame: Includes is_realtime_service to flag speech-to-speech LLMs for realtime mode enablement.

from pipecat.frames.frames import STTMetadataFrame
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies

def service_metadata_frame(self) -> STTMetadataFrame:
    # Recommend external turn strategies if service handles turn detection
    frame = super().service_metadata_frame()
    frame.user_turn_strategies = ExternalUserTurnStrategies()
    return frame

Pipecat supports parallel execution patterns using worker.job_group(...). In a multi-worker architecture like the parallel-debate example, the system is structured as follows:

1. Main Worker: Handles the primary transport (STT, TTS) and acts as a moderator. It uses a tool (e.g., a debate function) to trigger a fan-out via worker.job_group(...).
2. Debate Workers: These are LLMContextWorker instances spawned on the runner. Each worker is assigned a specific role (e.g., advocate, critic, analyst) and maintains its own independent LLMContext across multiple rounds of conversation.
3. Aggregation: Once the parallel jobs complete, the results are shipped back to the main worker via the assistant-aggregator's on_assistant_turn_stopped event to synthesize a final response.

Main worker (transport + LLM + `debate` tool)
  └── job_group(advocate, critic, analyst)
        └── DebateWorker (LLMContextWorker, one per role)