LiveKit Agents Playground

The application's global behavior and appearance can be configured using the NEXT_PUBLIC_APP_CONFIG environment variable. This variable expects a YAML-formatted string that is parsed into an AppConfig object. If this variable is not set, the application falls back to a default configuration.

AppConfig Schema

title: "My Custom Playground"
description: "Testing my custom agent"
video_fit: "cover"
settings:
  editable: true
  theme_color: "blue"
  chat: true
  inputs:
    camera: true
    screen: true
    mic: true
  outputs:
    audio: true
    video: true

The useStreamingWaveform hook extracts amplitude data from a LiveKit Track and stores it in a buffer synchronized with a WaveformClock.

Important: The getData() method returns a WaveformSnapshot. The buffer property is a live reference to the internal Uint8Array. Its contents are mutated in-place by the sampling interval. You must consume this data synchronously (e.g., inside a requestAnimationFrame callback). If you need to persist the data for later use, you must copy it using buffer.slice(0, count).

Parameters:

• track: The LiveKit Track to analyze. If undefined, the buffer is cleared.
• clock: An instance of WaveformClock to synchronize the sampling with a timeline.

Returns:

• An object containing getData(), which returns the current WaveformSnapshot.

import { useStreamingWaveform } from './useStreamingWaveform';
import type { Track } from 'livekit-client';

// Assuming 'clock' is provided by useWaveformClock()
const { getData } = useStreamingWaveform(track, clock);

// Inside a render loop or requestAnimationFrame
const frame = () => {
  const { buffer, count } = getData();
  // Use buffer and count to draw the waveform
  // Note: buffer is mutated in-place!
  requestAnimationFrame(frame);
};
requestAnimationFrame(frame);

The DebugPanel component is used to inspect real-time session state, including audio waveforms, event logs, performance metrics, and session usage. It provides a visual interface for debugging agent-user interactions, latency, and state transitions.

Key Props

Customization Types

DebugPanelTrackLabels

type DebugPanelTrackLabels = {
  user?: string;
  agent?: string;
};

DebugPanelTrackColors

type DebugPanelTrackColors = {
  agent?: string;
  user?: string;
};

DebugPanelHighlightConfig

type DebugPanelHighlightConfig = {
  /** Label for interruption highlights. @default "Interruption" */
  interruptionLabel?: string;
  /** Label for backchannel highlights. @default "Backchannel" */
  backchannelLabel?: string;
  /** Color for interruption highlights. @default "#FA4C39" */
  interruptionColor?: string;
  /** Color for backchannel highlights. @default "#23DE6B" */
  backchannelColor?: string;
};

import { DebugPanel } from "./debug-panel";

// Example usage:
<DebugPanel
  userTrack={userTrack}
  agentTrack={agentTrack}
  events={events}
  overlappingSpeechEvents={overlappingSpeechEvents}
  sessionUsage={sessionUsage}
  onClearEvents={handleClearEvents}
  networkLatency={networkLatency}
  uplinkLatency={uplinkLatency}
  trackLabels={{ user: "Customer", agent: "AI Assistant" }}
  trackColors={{ agent: "#BA1FF9", user: "#666666" }}
  highlightConfig={{
    interruptionLabel: "User Interrupted",
    interruptionColor: "#FF0000",
  }}
/>

The useRemoteSession hook provides a React interface for interacting with a remote LiveKit agent session via a Room instance. It handles the low-level byte stream communication, event tracking, and RPC (Remote Procedure Call) requests.

Use this hook to:

• Monitor real-time agent events.
• Track network latency based on speech overlap events.
• Access session usage statistics.
• Send requests to a specific agent identity.
• Clear the event history.

import { Room } from 'livekit-client';
import { useRemoteSession } from './hooks/useRemoteSession';

function MyComponent({ room }: { room: Room }) {
  const {
    events,
    overlappingSpeechEvents,
    sessionUsage,
    networkLatency,
    clearEvents,
    sendRequest
  } = useRemoteSession(room);

  const handleAgentRequest = async () => {
    try {
      const response = await sendRequest('agent-identity-here', { /* request payload */ });
      console.log('Agent response:', response);
    } catch (error) {
      console.error('Request failed:', error);
    }
  };

  return (
    // Render UI using session data
  );
}

The Playground component is the main entry point for the agents-playground UI. It requires a tokenSource for authentication and themeColors for styling. You can optionally provide agentOptions to pre-configure the agent being connected to (e.g., setting its name or metadata) and autoConnect to start the session immediately upon mounting.

Props

• themeColors: An array of strings representing available theme colors.
• tokenSource: A TokenSourceConfigurable object used to fetch LiveKit tokens.
• agentOptions (optional): A PartialMessage<RoomAgentDispatch> used to pass initial configuration to the agent (like agentName or metadata).
• autoConnect (optional): A boolean that, if true, triggers session.start() automatically.
• logo (optional): A React node to display in the header.

import Playground from './components/playground/Playground';

// Example usage
<Playground 
  themeColors={['blue', 'red', 'green']} 
  tokenSource={myTokenSource} 
  agentOptions={{ agentName: 'my-agent' }}
  autoConnect={true}
/>