repository·main·Indexed 23 days ago
https://github.com/google-gemini/live-api-web-consoleA React-based starter application for interacting with the Gemini Multimodal Live API over WebSockets. It features built-in modules for audio streaming, media recording (microphone, webcam, screen), and development logging. The project includes the GenAILiveClient for WebSocket management, the useLiveAPI hook for React state management, and utility classes like AudioRecorder and AudioStreamer for handling PCM16 audio data.
To use this starter app, you must first obtain a Gemini API key from Google AI Studio.
.env file in the project root..env file.npm install && npm start.The app will be available at http://localhost:3000.
$ npm install && npm startThe client instance provided by useLiveAPIContext is an event emitter. To respond to model-generated tool calls (like function calling), listen for the toolcall event using client.on('toolcall', callback).
In the callback, you can inspect toolCall.functionCalls to identify which function the model wants to execute and access its arguments.
useEffect(() => {
const onToolCall = (toolCall: ToolCall) => {
const fc = toolCall.functionCalls.find(
(fc) => fc.name === "your_function_name"
);
if (fc) {
const args = fc.args as any;
// Handle function logic here
}
};
client.on("toolcall", onToolCall);
return () => {
client.off("toolcall", onToolCall);
};
}, [client]);The useLiveAPIContext hook provides access to the client (an event-emitting websocket client) and a setConfig function. Use setConfig to define the model, system instructions, and tools (such as Google Search or function declarations) for the session.
Common configuration keys include:
model: The model identifier (e.g., models/gemini-2.0-flash-exp).systemInstruction: An object containing parts with text instructions.tools: An array of tool objects, such as { googleSearch: {} } or { functionDeclarations: [...] }.import { useLiveAPIContext } from "../../contexts/LiveAPIContext";
// Inside a component:
const { client, setConfig } = useLiveAPIContext();
useEffect(() => {
setConfig({
model: "models/gemini-2.0-flash-exp",
systemInstruction: {
parts: [{ text: 'Your instructions here' }],
},
tools: [{ googleSearch: {} }],
});
}, [setConfig]);The project provides the following scripts for development and production:
npm start: Runs the application in development mode at http://localhost:3000. The page reloads on edits.npm run build: Builds the application for production, creating a minified and optimized bundle in the build folder.npm start
npm run buildThe AudioStreamer class manages the playback of raw PCM16 audio data using the Web Audio API. It handles converting Uint8Array chunks of PCM16 data into Float32Array buffers, queuing them, and scheduling playback to ensure smooth streaming.
AudioContext.resume() (typically in response to a user interaction) to unblock the AudioContext and prepare the stream.addPCM16(chunk) to push new Uint8Array chunks of PCM16 data into the playback queue.stop() to clear the queue and fade out the audio.The class automatically converts 16-bit signed integer PCM data (normalized between -32768 and 32767) to the Float32 format expected by the Web Audio API.
Use the useLiveAPIContext hook within any component that is a child of a LiveAPIProvider to access the Live API instance and its results.
Note: This hook will throw an error if it is called from a component that is not wrapped in a LiveAPIProvider.
createWorketFromSrc utility allows you to convert a raw JavaScript string containing AudioWorklet processor code into a URL that can be passed to audioContext.audioWorklet.addModule(). It wraps the provided workletSrc with a registerProcessor call using the specified workletName.When initiating a Live client, use the LiveClientOptions type. This type extends GoogleGenAIOptions and requires an apiKey string to be provided.
export type LiveClientOptions = GoogleGenAIOptions & { apiKey: string };base64ToArrayBuffer function converts a base64-encoded string into an ArrayBuffer. This is useful for processing audio data received as base64 strings from the Live API.To stream real-time multimodal data (like audio or video frames) to the model, use sendRealtimeInput. This method accepts chunks of media data.
Parameters:
chunks: An array of objects containing mimeType (e.g., audio/pcm or image/jpeg) and data (a base64 encoded string).This is typically used for continuous streaming of audio or video frames to maintain a live interaction.
The GenAILiveClient class manages a WebSocket connection to the Gemini Multimodal Live API. It extends EventEmitter to provide a reactive interface for handling real-time audio, content, and tool calls. You can use this class in non-React environments by listening to its emitted events.
LiveClientOptions (which wraps GoogleGenAI configuration).connect(model, config) with the target model string and a LiveConnectConfig object.send(), sendRealtimeInput(), or sendToolResponse() to communicate with the model.disconnect() to close the session.When the client receives a toolcall event, you must execute the requested function and then send the result back to the model using sendToolResponse.
Parameters:
toolResponse: An object of type LiveClientToolResponse containing functionResponses. Each response must include the ID of the tool call it is responding to.