Azure OpenAI Realtime Audio SDK

The service uses a custom WebSocket protocol for client-server communication. The protocol is divided into several message types:

• Control Messages: Manage connection status and lifecycle actions (e.g., connected, speech_started, text_done).
• User Messages: Used to send text inputs from the client (user_message).
• Transcriptions: Sent by the server to provide transcribed text from audio inputs (transcription).
• Text Deltas: Used to stream partial text responses to the client (text_delta).

Message Schema (Java Types)

Below are the structural definitions for the protocol messages:

class TextDelta {
    String id;
    MessageType type = MessageType.TEXT_DELTA;
    String delta;
}

class UserMessage {
    String id;
    MessageType type = MessageType.USER_MESSAGE;
    String text;
}

class TranscriptionMessage {
    String id;
    MessageType type = MessageType.TRANSCRIPTION;
    String text;
}

class ControlMessage {
    MessageType type = MessageType.CONTROL;
    String action;
    String greeting;
    String id;
}

The /realtime API follows a hierarchical structure for managing interactions:

• Session: Established when a caller connects to /realtime. The session manages shared settings like audio behavior, voice activity detection (VAD), and tool configurations. A session automatically creates a default conversation.
• Conversation: An accumulation of input signals (audio/text) within a session. Currently, one session supports one active conversation.
• Response: Triggered either by a direct command (e.g., response.create) or automatically via voice activity detection. A response generates output from the model.
• Items: A response consists of one or more items. Items can encapsulate messages, function calls, or other information.
• Content Parts: Each message item can contain multiple content_parts, allowing a single item to represent multiple modalities (e.g., both text and audio) simultaneously.

The middle-tier uses a lightweight, custom protocol for WebSocket communication. The protocol consists of several message types:

• Control Messages: Manage connection status and lifecycle actions (e.g., connected, speech_started, text_done).
• User Messages: Used to send text inputs from the user (user_message).
• Transcriptions: Sent by the server to provide transcribed text from audio inputs (transcription).
• Text Deltas: Used to stream partial text responses to the client (text_delta).

Protocol Type Definitions

The following Python TypedDict structures define the message shapes:

class TextDelta(TypedDict):
    id: str
    type: Literal["text_delta"]
    delta: str

class Transcription(TypedDict):
    id: str
    type: Literal["transcription"]
    text: str

class UserMessage(TypedDict):
    id: str
    type: Literal["user_message"]
    text: str

class ControlMessage(TypedDict):
    type: Literal["control"]
    action: str
    greeting: str | None = None
    id: str | None = None

The /realtime API is accessed via the RealtimeConversationClient. This client is obtained from a top-level OpenAIClient (or AzureOpenAIClient for Azure resources).

Lifecycle and Configuration

1. Instantiate Client: Call .GetRealtimeConversationClient() on your existing OpenAI client.
2. Start Session: Call ConversationClient.StartConversationSessionAsync() to create a RealtimeConversationSession (which implements IDisposable).
3. Configure Session: Use session.ConfigureSessionAsync(sessionOptions) where sessionOptions is a ConversationSessionOptions object. This object allows you to define:
   • Input and output audio formats
   • Turn end detection behavior
   • Model instructions
   • Tools

Sending Data

• Text/Tools: Use AddItemAsync() to add non-audio content (like text or tool responses) to the conversation history.
• Audio: Use SendInputAudioAsync(Stream) to automatically chunk and transmit audio from a stream, or SendInputAudioAsync(BinaryData) for individual message transmissions.

Receiving Updates

Use ReceiveUpdatesAsync() to get an IAsyncEnumerable<ConversationUpdate>. Each update's Kind property (of type ConversationUpdateKind) maps to the wire protocol's type field.

// Conceptual usage pattern
var client = azureOpenAIClient.GetRealtimeConversationClient();
using var session = await client.StartConversationSessionAsync();
await session.ConfigureSessionAsync(new ConversationSessionOptions { /* ... */ });

// Send audio
await session.SendInputAudioAsync(audioStream);

// Receive updates
await foreach (var update in session.ReceiveUpdatesAsync()) 
{
    // Handle update
}

The application follows a specific lifecycle to manage the real-time conversation:

1. Initialization: Configures a client from environment variables and connects a new session.
2. Session Configuration: Enables input audio transcription and a custom "I'm finished" tool (allowing the model to decide when to end the conversation).
3. Audio Setup: Starts playback to the default output device.
4. Command Processing Loop:
   • Session Start: Once the session starts, microphone input begins.
   • Speech Detection: When user speech is detected, any active audio output is aborted and cleared.
   • Transcription Feedback: When user audio input transcription is finished, the transcript is printed to the console.
   • Streaming: Incremental transcripts and audio data are immediately printed or rendered to the speaker.
   • Termination: The loop ends if an output item invokes the "I'm finished" tool or if an error is received.