Pipecat Client Web SDK

The UI Worker Protocol (v1) allows a server-side Pipecat worker to observe and drive your React application using a structured wire format.

Key features include:

• useUISnapshot(): Streams accessibility snapshots so the server knows what is on the screen.
• UIJobGroupsProvider: Subscribes to ui-job-group envelopes for long-running, fan-out tasks, providing per-worker progress and cancellation.
• Command Handlers: Use useUICommandHandler to register custom app-specific commands (e.g., toast) and useDefaultScrollToHandler, useDefaultHighlightHandler, etc., to opt-in to standard DOM interactions driven by the server.
• Form Controls: Follows standard semantics (e.g., set_input_value for inputs/selects, click for checkboxes/radios).

import { PipecatClient } from "@pipecat-ai/client-js";
import {
  PipecatClientProvider,
  UIJobGroupsProvider,
  useUISnapshot,
  useDefaultScrollToHandler,
  useDefaultHighlightHandler,
  useUICommandHandler,
} from "@pipecat-ai/client-react";

const client = new PipecatClient({ transport: myTransport });

function App() {
  useUISnapshot();
  useDefaultScrollToHandler();
  useDefaultHighlightHandler();
  useUICommandHandler("toast", (payload) => showToast(payload));
  return <YourApp />;
}

render(
  <PipecatClientProvider client={client}>
    <UIJobGroupsProvider>
      <App />
    </UIJobGroupsProvider>
  </PipecatClientProvider>
);

To start a session, configure the PipecatClient with a transport, media preferences, and lifecycle callbacks. Use startBotAndConnect with a server-side endpoint to initiate the connection.

Note: It is highly recommended to use your own server-side endpoints to handle authentication and protect bot process secrets (like API keys).

import { PipecatClient, RTVIEvent, RTVIMessage } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

const pcClient = new PipecatClient({
  transport: new DailyTransport(),
  enableMic: true,
  enableCam: false,
  callbacks: {
    onConnected: () => console.log("[CALLBACK] User connected"),
    onDisconnected: () => console.log("[CALLBACK] User disconnected"),
    onTransportStateChanged: (state: string) => console.log("[CALLBACK] State change:", state),
    onBotConnected: () => console.log("[CALLBACK] Bot connected"),
    onBotDisconnected: () => console.log("[CALLBACK] Bot disconnected"),
    onBotReady: () => console.log("[CALLBACK] Bot ready to chat!"),
  },
});

try {
  await pcClient.startBotAndConnect({ endpoint: "https://your-server-side-url/connect" });
} catch (e) {
  console.error(e.message);
}

// Event Listeners
pcClient.on(RTVIEvent.TransportStateChanged, (state) => {
  console.log("[EVENT] Transport state change:", state);
});
pcClient.on(RTVIEvent.BotReady, () => {
  console.log("[EVENT] Bot is ready");
});

To start a conversation, instantiate a PipecatClient with a transport (such as DailyTransport), configure media settings, and provide lifecycle callbacks. Use startBotAndConnect with your connection endpoint to begin the session.

import { RTVIEvent, RTVIMessage, PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

const pcClient = new PipecatClient({
  transport: new DailyTransport(),
  enableMic: true,
  enableCam: false,
  callbacks: {
    onConnected: () => {
      console.log("[CALLBACK] User connected");
    },
    onDisconnected: () => {
      console.log("[CALLBACK] User disconnected");
    },
    onTransportStateChanged: (state: string) => {
      console.log("[CALLBACK] State change:", state);
    },
    onBotConnected: () => {
      console.log("[CALLBACK] Bot connected");
    },
    onBotDisconnected: () => {
      console.log("[CALLBACK] Bot disconnected");
    },
    onBotReady: () => {
      console.log("[CALLBACK] Bot ready to chat!");
    },
  },
});

try {
  await pcClient.startBotAndConnect({ endpoint: "https://your-connect-end-point-here/connect" });
} catch (e) {
  console.error(e.message);
}

// Events
pcClient.on(RTVIEvent.TransportStateChanged, (state) => {
  console.log("[EVENT] Transport state change:", state);
});
pcClient.on(RTVIEvent.BotReady, () => {
  console.log("[EVENT] Bot is ready");
});
pcClient.on(RTVIEvent.Connected, () => {
  console.log("[EVENT] User connected");
});
pcClient.on(RTVIEvent.Disconnected, () => {
  console.log("[EVENT] User disconnected");
});

The @pipecat-ai/client-react package provides a suite of React hooks and components for building real-time AI interfaces. Key capabilities include:

• Client Management: usePipecatClient to access the core client instance.
• Media Control: Hooks like usePipecatClientMicControl, usePipecatClientCamControl, and usePipecatClientScreenShareControl for managing audio, video, and screen sharing.
• Conversation UI: usePipecatConversation and useConversationContext for managing and displaying chat/transcript history.
• UI Command Handling: useUICommandHandler and useUIEventSender for interacting with the agent via UI events.
• Pre-built Components: PipecatClientVideo, PipecatClientMicToggle, PipecatClientCamToggle, and VoiceVisualizer for common UI elements.
• State & Events: useRTVIClientEvent for listening to client events and usePipecatClientTransportState for monitoring connection status.

The PipecatClient extends RTVIEventEmitter, allowing you to subscribe to a wide range of events using .on(event, callback) or by providing a callbacks object in the constructor.

Common Event Categories:

• Connection: onConnected, onDisconnected, onBotReady, onBotConnected, onBotDisconnected.
• Media/Devices: onMediaStateChanged, onDeviceError, onCamUpdated, onMicUpdated, onTrackStarted, onTrackStopped.
• Audio/Speech: onUserStartedSpeaking, onUserStoppedSpeaking, onBotStartedSpeaking, onBotStoppedSpeaking, onUserTranscript, onBotOutput.
• LLM/Bot Intelligence: onBotLlmText, onBotTtsText, onLLMFunctionCallStarted, onLLMFunctionCallInProgress, onLLMFunctionCallStopped.
• UI/System: onUICommand, onUIJobGroup, onUnsupportedFeature, onMessageError.

Example: Using Event Listeners

client.on(RTVIEvent.BotReady, (data) => {
  console.log('Bot is ready with data:', data);
});

client.on(RTVIEvent.UserTranscript, (data) => {
  console.log('User said:', data.text);
});

The useConversationEventWiring hook is an internal mechanism used by the PipecatConversationProvider to synchronize RTVI (Real-Time Voice Interface) client events with the React application state (managed via Jotai atoms).

It automates the lifecycle of a conversation by listening to events such as BotOutput, UserTranscript, and LLMFunctionCall events, and translating them into message updates in the conversation history.

Key behaviors include:

• Assistant Turn Management: It uses a delayed timer (BOT_STOPPED_FINALIZE_DELAY_MS) to finalize assistant messages. This prevents premature message finalization during brief pauses in speech. The timer is reset if the bot starts speaking again.
• Protocol Switching: It automatically detects the RTVI protocol version (e.g., v2 vs legacy) from the BotReady event to handle BotOutput data correctly (server-side vs client-side speech progress).
• User Turn Management: It finalizes user messages when the user starts a new turn or after a period of silence following UserStoppedSpeaking.
• Function Call Tracking: It wires LLM function call lifecycle events (LLMFunctionCallStarted, LLMFunctionCallInProgress, LLMFunctionCallStopped) to the conversation state to show tool usage in the UI.