Pipecat Gemini WebRTC Reference Implementation

repository·main·Indexed 42 days ago

https://github.com/pipecat-ai/gemini-webrtc-web-simple

A reference implementation of a voice AI application using Gemini's Multimodal Live API. It demonstrates using the Pipecat orchestration framework to bridge WebRTC media streams from a web client to a multimodal AI model via a Python-based pipeline. Includes a Vite-based web client using the Pipecat JavaScript SDK and a FastAPI server implementing the RTVI open standard.

Tokens
3K
Snippets
9
Records
13
Agent score
53%

What's inside Pipecat Gemini WebRTC

  1. Initialize the Gemini Multimodal Live WebRTC Example

    main

    This project demonstrates a voice AI application using the Gemini Multimodal Live API and WebRTC. It consists of a web client (using the Pipecat JavaScript SDK and Daily WebRTC transport) and a Python server (using the Pipecat orchestration framework).

    Note: This repository is no longer maintained. For recent demos, refer to pipecat-examples.

      const rtviClient = new RTVIClient({
        transport,
        params: {
          baseUrl: "http://localhost:7860/", 
        },
        enableMic: true,
        enableCam: false,
        timeout: 30 * 1000,
      });
  2. Understand why WebRTC is preferred over WebSockets for real-time media

    main

    When building production-grade voice AI applications, WebRTC is recommended over WebSockets for client-server real-time media connections due to several architectural advantages:

    • Avoids Head-of-Line Blocking: Unlike WebSockets (built on TCP), WebRTC is designed to handle audio streams without waiting for delayed packet resends that cause latency spikes.
    • Congestion Control: WebRTC's Opus audio codec is tightly coupled with bandwidth estimation and packet pacing, making it resilient to real-world network fluctuations.
    • Error Resilience: WebRTC supports forward error correction (FEC) and allows for dropping late-arriving packets, which is critical for maintaining low latency.
    • Simplified Synchronization: Audio sent and received over WebRTC is automatically time-stamped, making playout and interruption logic easier to implement.
    • Observability: WebRTC provides hooks for detailed performance and media quality statistics (audio/video specific) that are difficult to implement with WebSockets.
    • Robust Reconnection: WebRTC implementations handle connection states more reliably than custom ping/ack frameworks required for WebSockets.
    • Built-in Audio Processing: Modern WebRTC implementations include echo cancellation, noise reduction, and automatic gain control.
    • Optimized Routing: Using a WebRTC platform (like Daily) allows users to connect to edge servers close to them, routing traffic through efficient backbones rather than unpredictable public Internet routes.
  3. How the WebRTC Proxy Architecture works

    main

    The application uses a 'smart proxy' architecture where the Pipecat server acts as a relay between the client and the Gemini Multimodal Live API.

    Workflow:

    1. Client: Sends/receives audio and events via the WebRTC protocol (using the Daily transport).
    2. Pipecat Pipeline (Server): Receives WebRTC media, translates it, and relays audio/events to the Gemini Multimodal Live API.

    Advantages of this approach:

    • Lower Latency: WebRTC often provides faster voice-to-voice response times for real-world users compared to direct WebSocket connections.
    • Extensibility: You can add server-side logic (e.g., transcription via Gemini's standard API) or custom Python logic within the Pipecat pipeline.
    • Standardization: Uses the RTVI open standard for rich client-server events.
    • Cross-Platform: Compatible with Web, React, React Native, iOS, Android, Python, and C++ SDKs.
  4. Set up the Python Server

    main

    The server is a Pipecat pipeline that implements the Gemini Multimodal Live WebSocket API.

    1. Navigate to the server directory and prepare environment variables:
      cd server
      cp env.example .env
    2. Add your Gemini API key (from Google AI Studio) and Daily API Key (from Daily.co) to the .env file.
    3. Create a virtual environment and run the server:
      python3.12 -m venv venv
      source venv/bin/activate
      pip install -r requirements.txt
      python server.py
    python3.12 -m venv venv
    source venv/bin/activate
    pip install -r requirements.txt
    python server.py
  5. Configure the server with environment variables

    main

    The server uses several environment variables for configuration. Ensure these are set in your .env file or your environment:

    • DAILY_API_KEY: Your Daily API key (required).
    • DAILY_API_URL: The Daily API URL (defaults to https://api.daily.co/v1).
    • HOST: The host address to bind the server to (defaults to 0.0.0.0).
    • FAST_API_PORT: The port number for the FastAPI server (defaults to 7860).
  6. Explore additional Pipecat and Gemini resources

    main

    For further development and deep dives, refer to the following resources:

    Gemini Multimodal Live

    Pipecat SDKs

    Pipecat provides architecture-compatible, open-source SDKs for various platforms:

    Specialized Repositories & Demos

    Technical Deep Dives

  7. Connect an RTVI client via the /connect endpoint

    main

    The /connect POST endpoint is designed for RTVI clients to establish a connection. When called, the server creates a new Daily room, generates an access token, and spawns a new bot process. It returns an authentication bundle containing the room_url and token in the format expected by DailyTransport.

    POST /connect
    
    Returns:
    {
      "room_url": "<room_url>",
      "token": "<token>"
    }
  8. Handle RTVIClient events

    main

    The RTVIClient uses an event-driven model to communicate session state, transcription, and media changes. You can subscribe to events using the .on(event, callback) method.

    Commonly used RTVIEvent types include:

    • TransportStateChanged: Emits the current state of the transport.
    • Connected / Disconnected: Emits when the user connection changes.
    • BotConnected / BotDisconnected: Emits when the bot connection changes.
    • BotReady: Emitted when the bot is ready for interaction.
    • TrackStarted: Emitted when a new media track (audio/video) is received. The callback provides the MediaStreamTrack and the Participant who owns it.
    • UserStartedSpeaking / UserStoppedSpeaking: Emits when user voice activity is detected.
    • BotStartedSpeaking / BotStoppedSpeaking: Emits when the bot is generating audio.
    • UserTranscript: Emits transcription data for the user. The callback receives TranscriptData which includes a text string and a final boolean.
    • BotTranscript: Emits text data from the bot. The callback receives BotLLMTextData.
    • Error / MessageError: Emits error information as an RTVIMessage.
    • Metrics: Emits performance metrics, such as ttfb (Time To First Byte).
    import { RTVIClient, RTVIEvent } from "@pipecat-ai/client-js";
    
    export async function setupEventHandlers(rtviClient: RTVIClient) {
      rtviClient.on(RTVIEvent.Connected, () => {
        console.log("-- user connected --");
      });
    
      rtviClient.on(RTVIEvent.UserTranscript, (transcript) => {
        if (transcript.final) {
          console.log('final:', transcript.text);
        } else {
          console.log('interim:', transcript.text);
        }
      });
    
      rtviClient.on(RTVIEvent.BotTranscript, (data) => {
        console.log('bot text:', data.text);
      });
    
      rtviClient.on(RTVIEvent.TrackStarted, (track, participant) => {
        if (!participant.local) {
          // Handle remote track
        }
      });
    }
  9. Initialize and connect an RTVIClient

    main

    To establish a WebRTC connection using the RTVI protocol, you must instantiate an RTVIClient with a Transport (such as DailyTransport) and a configuration object. After initialization, you must call initDevices() to prepare media hardware and connect() to establish the session.

    Key configuration options for RTVIClient include:

    • transport: An instance of a Transport class.
    • params: An object containing baseUrl (the URL of your Pipecat server).
    • enableMic: Boolean to enable/disable microphone access.
    • enableCam: Boolean to enable/disable camera access.
    • timeout: Connection timeout in milliseconds.
    import { RTVIClient } from "@pipecat-ai/client-js";
    import { DailyTransport } from "@pipecat-ai/daily-transport";
    
    const transport = new DailyTransport();
    const rtviClient = new RTVIClient({
        transport,
        params: {
          baseUrl: "http://localhost:7860/", 
        },
        enableMic: true,
        enableCam: false,
        timeout: 30 * 1000,
    });
    
    try {
        await rtviClient.initDevices();
        await rtviClient.connect();
    } catch (e) {
        console.log('Error connecting', e);
    }