DOCSEARCHbyAlgolia
Submit Docs

swift-realtime-openai

repository·main·Indexed 19 days ago

https://github.com/m1guelpf/swift-realtime-openai

A modern Swift SDK for OpenAI's Realtime API that enables multi-modal text and audio conversations. It provides a high-level Conversation class for managing history, microphone recording, and playback, as well as a RealtimeAPI class for direct interaction via WebRTC or WebSockets.

Tokens
1.2K
Snippets
4
Records
4
Agent score
16%

What's inside swift-realtime-openai

  1. How the Conversation class works

    main

    The Conversation class is a high-level wrapper around RealtimeAPI. It manages:

    • Sending and receiving messages.
    • Conversation history.
    • Automatic microphone recording.
    • Automatic playback of model responses.

    Accessing Messages

    • messages: Returns only the messages between the user and the model (excludes function calls/responses).
    • entries: Returns the full conversation history, including all event types.

    Customizing the Session

    To modify session settings (like system instructions or transcription), use updateSession(withChanges:) after the connection is established. It is recommended to do this within a whenConnected callback.

    Sending Content

    • Text: send(from:text:response:) using roles .user, .assistant, or .system.
    • Audio: send(audioDelta:commit:) to send raw audio chunks. Setting commit: true tells the model the message is finished.
    • Events: send(event:) allows sending raw RealtimeAPI.ClientEvent objects, though this bypasses automatic interrupt handling.
    // Customizing session settings
try await conversation.whenConnected {
    try await conversation.updateSession { session in
        session.instructions = "You are a helpful assistant."
        session.inputAudioTranscription = Session.InputAudioTranscription()
    }
}

// Sending a text message
try await conversation.send(from: .user, text: "Hello!")

// Sending audio chunks
try await conversation.send(audioDelta: audioData, commit: true)

  2. Quickstart: Build a simple AI chat or voice app

    main

    The library provides a high-level Conversation class to manage multi-modal interactions.

    Text-based Chat Example

    Use conversation.send(from:text:) to send messages and conversation.entries or conversation.messages to retrieve history.

    Voice-only Example

    Simply connecting the conversation will enable automatic microphone recording and audio playback.

    Note: You must provide an ephemeral key to the connect(ephemeralKey:) method.

    import SwiftUI
import RealtimeAPI

struct ContentView: View {
	@State private var conversation = try! Conversation()

	var body: some View {
		Text("Say something!")
			.task { try! await conversation.connect(ephemeralKey: YOUR_EPHEMERAL_KEY_HERE) }
	}
}

  3. Install swift-realtime-openai via Swift Package Manager

    main

    You can integrate this library into your Xcode project or Swift package using Swift Package Manager (SPM).

    Via Xcode Project

    1. Go to File > Swift Packages > Add Package Dependency.
    2. Enter the URL: https://github.com/m1guelpf/swift-realtime-openai.git.
    3. Select the main branch.

    Via Package.swift

    Add the Git link to your dependencies array:

    dependencies: [
    .package(url: "https://github.com/m1guelpf/swift-realtime-openai.git", .branch("main"))
]

  4. Use RealtimeAPI for direct API interaction

    main

    If you need more control than Conversation provides, use RealtimeAPI directly. You can initialize it using WebRTC (with an ephemeral key) or WebSockets (with an OpenAI API key).

    Initialization

    // WebRTC
let api = RealtimeAPI.webRTC(ephemeralKey: YOUR_EPHEMERAL_KEY, model: .gptRealtime)

// WebSocket
let api = RealtimeAPI.webSocket(authToken: YOUR_OPENAI_API_KEY, model: .gptRealtime)

    Listening for Events

    Iterate over the events property using for try await to react to API updates:

    for try await event in api.events {
    switch event {
    case let .sessionCreated(event):
        print(event.session.id)
    default: break
    }
}

    Sending Events

    Use the send(event:) method to transmit ClientEvent instances directly to the server.

    for try await event in api.events {
    switch event {
    case let .sessionCreated(event):
        print(event.session.id)
    }
}

try await api.send(event: .createResponse())