OpenAI Realtime API Reference Client (beta)

The library provides three main primitives for interacting with the API:

1. RealtimeClient: The high-level abstraction recommended for most users. It simplifies control flow and provides high-level utility events (e.g., conversation.updated).
2. RealtimeAPI (accessed via client.realtime): A thin wrapper over the WebSocket. It is used for low-level connection, authentication, and sending raw items. It does not perform item validation.
3. RealtimeConversation (accessed via client.conversation): A client-side cache of the current conversation. It includes event validation to ensure items are cached correctly.

If you want to handle tool calls manually (e.g., to generate a schema without automatic execution), use client.updateSession(). When doing this, you must specify type: 'function' for each tool.

Note: Tools added via .addTool() are persisted and appended to tools defined in updateSession(). However, every call to updateSession() overrides previous session configurations.

client.updateSession({
  tools: [
    {
      type: 'function',
      name: 'get_weather',
      description: 'Retrieves weather.',
      parameters: { /* ... schema ... */ },
    },
  ],
});

// Handle the function call manually via events
client.on('conversation.updated', ({ item, delta }) => {
  if (item.type === 'function_call') {
    if (delta.arguments) {
      // process arguments
    }
  }
});

To use the reference client in your Node.js or browser project, install it directly from the GitHub repository using npm:

$ npm i openai/openai-realtime-api-beta --save

Before running tests, ensure you have a .env file in your project root with your OPENAI_API_KEY configured:

OPENAI_API_KEY=your_key_here

Use the following commands to execute the test suite:

• Standard test run: npm test
• Debug mode: Use npm test -- --debug to enable debug logs, which will output all events sent to and received from the WebSocket.

$ npm test

# To run tests with debug logs
$ npm test -- --debug

Sending Text Messages

Use client.sendUserMessageContent() to send text or empty audio items to the server:

client.sendUserMessageContent([{ type: 'input_text', text: `How are you?` }]);

Sending Streaming Audio

To stream audio, use client.appendInputAudio(). The default format is pcm16 at 24,000 Hz. If turn_detection is set to 'none', you must call client.createResponse() to trigger a model response after sending the audio chunks.

// Append audio chunks (Int16Array or ArrayBuffer)
client.appendInputAudio(new Int16Array(2400));

// Trigger response if turn detection is disabled
client.createResponse();

client.sendUserMessageContent([{ type: 'input_text', text: `How are you?` }]);

// Streaming audio example
client.appendInputAudio(new Int16Array(2400));
client.createResponse();

The RealtimeClient is the primary abstraction for interfacing with the Realtime API. You can configure session parameters like instructions, voice, turn_detection, and input_audio_transcription using client.updateSession() before or after connecting.

Note: When using in a browser, you must set dangerouslyAllowAPIKeyInBrowser: true to acknowledge the security risk of exposing your API key.

import { RealtimeClient } from '@openai/realtime-api-beta';

const client = new RealtimeClient({ apiKey: process.env.OPENAI_API_KEY });

// Configure session
client.updateSession({ instructions: 'You are a great, upbeat friend.' });
client.updateSession({ voice: 'alloy' });
client.updateSession({
  turn_detection: { type: 'none' },
  input_audio_transcription: { model: 'whisper-1' },
});

// Connect to the API
await client.connect();

To avoid exposing your API key in a browser environment, it is recommended to use a relay server (such as the Realtime Console). You can connect the client to the relay server URL instead of using an API key directly:

const client = new RealtimeClient({ url: RELAY_SERVER_URL });

The RealtimeClient is the primary entry point for interacting with the Realtime API. You can initialize it with connection settings and then call .connect() to establish a WebSocket connection. It is recommended to use .waitForSessionCreated() after connecting to ensure the server has initialized the session before sending commands.

Constructor Settings:

• url: The WebSocket URL.
• apiKey: Your OpenAI API key.
• dangerouslyAllowAPIKeyInBrowser: Boolean to allow API key usage in client-side environments.
• debug: Boolean to enable debug logging.

import { RealtimeClient } from './lib/client.js';

const client = new RealtimeClient({
  url: 'YOUR_WS_URL',
  apiKey: 'YOUR_API_KEY',
  dangerouslyAllowAPIKeyInBrowser: true
});

await client.connect();
await client.waitForSessionCreated();

To gain fine-grained control over your application, you can listen to the realtime.event event. This allows you to filter for events where the source is 'server', enabling you to respond specifically to payloads emitted by the Realtime API. The event object contains a time (ISO timestamp), a source (either 'client' or 'server'), and the event (the raw JSON payload).

// all events, can use for logging, debugging, or manual event handling
client.on('realtime.event', ({ time, source, event }) => {
  // time is an ISO timestamp
  // source is 'client' or 'server'
  // event is the raw event payload (json)
  if (source === 'server') {
    doSomething(event);
  }
});