DOCSEARCHbyAlgolia
Submit Docs

OpenAI Realtime Solar System Demo

repository·main·Indexed 19 days ago

https://github.com/openai/openai-realtime-solar-system

A demo showcasing the OpenAI Realtime API (via WebRTC) integrated with a 3D Spline scene. It uses function calling with the gpt-realtime-1.5 model to allow a voice-enabled AI to control animations, camera views, and UI elements in a solar system simulation.

Tokens
3.7K
Snippets
13
Records
16
Agent score
68%

What's inside openai-realtime-solar-system

  1. Start and manage a Realtime session

    main

    Once the application is running, you can interact with the model using voice:

    • Start a session: Click the wifi icon in the top right corner. When the icon turns green, the session is active and you can start talking.
    • Mute/Unmute: Use the mic icon next to the wifi icon to toggle your microphone.
    • Stop a session: Click the wifi icon again. This will stop the session and reset the conversation.

    Tip: Ensure there is no background noise or echo to prevent model interruptions.

  2. Install and run the Realtime Solar System Demo

    main

    To run this demo locally, follow these steps:

    1. Clone the repository:
      git clone https://github.com/openai/openai-realtime-solar-system.git
    2. Set your OpenAI API key: Create a .env file in the project root and add:
      OPENAI_API_KEY=<your_api_key>
    3. Install dependencies:
      npm install
    4. Run the application:
      npm run dev

    The app will be available at http://localhost:3000. Note that the 3D scene may take a moment to load on the first run.

    git clone https://github.com/openai/openai-realtime-solar-system.git
# Create .env with OPENAI_API_KEY=<your_api_key>
npm install
npm run dev

  3. Configure custom Spline scenes

    main

    You can replace the default solar system scene with your own Spline scene by modifying components/scene.tsx.

    1. Update the scene URL: Change the scene prop in the <Spline /> component to point to your .splinecode file.
      <Spline
  scene="https://prod.spline.design/<scene_id>/scene.splinecode"
  onLoad="{onLoad}"
/>
    2. Trigger animations/events: To trigger events (like a mouseDown event you've configured in Spline) from your code, use the emitEvent method on the spline instance:
      spline.current.emitEvent("mouseDown", "object_name")

  4. Customize model behavior and tools

    main

    The model's personality, instructions, and available functions are defined in lib/config.ts. You can modify the following to change how the demo behaves:

    • Instructions: Update the system prompt to change how the model answers questions.
    • Tools: Add or modify the function calling definitions to map model intents to different application actions.
    • Voice: Change the model's voice using the OpenAI voice options.

  5. Handle tool calls and function outputs

    main

    The application implements a pattern for handling function_call events received from the model via the data channel:

    1. Detection: When a response.done event is received, the app scans the outputs for a type: "function_call".
    2. Execution: The handleToolCall function is triggered. For example, if the tool name is "get_iss_position", it fetches data from /api/iss.
    3. Reporting: The result is wrapped in a ToolCallOutput object and sent back to the model using a conversation.item.create event with type: "function_call_output".
    4. Resumption: For specific tools like "get_iss_position" or "display_data", a response.create event is sent to ensure the model continues the conversation after receiving the tool result.
    async function handleToolCall(output: any) {
  // 1. Identify tool
  // 2. Execute logic (e.g., fetch ISS position)
  // 3. Send output back via sendClientEvent({
  //      type: "conversation.item.create",
  //      item: { type: "function_call_output", ... }
  //    })
  // 4. Optionally trigger response.create
}

  6. Configure the Realtime session instructions and voice

    main

    The session behavior is governed by the INSTRUCTIONS string and the VOICE constant.

    • INSTRUCTIONS: Defines the persona (a friendly classroom assistant), the logic for when to trigger tools (e.g., using focus_planet when a planet is mentioned, or display_data for numeric comparisons), and the communication style (concise, fast-speaking).
    • VOICE: Specifies the voice model used for the assistant. Currently set to "coral".
    export const INSTRUCTIONS = `...`;
export const VOICE = "coral";

  7. Stop a session with stopSession()

    main

    Call stopSession() to terminate the current Realtime interaction and clean up all resources. This function:

    • Closes the RTCDataChannel.
    • Closes the RTCPeerConnection.
    • Stops all active audio tracks in the audioStream.
    • Resets internal state (isSessionStarted, isSessionActive, isListening, etc.).
    • Clears the audioElement source.
    function stopSession() {
  // ... implementation details for cleaning up WebRTC and audio
}

  8. Define and use the Component interface for UI mapping

    main

    The Component interface defines the structure for data-driven UI elements used to map tool function outputs to visual components (like charts). When providing data to getComponent, ensure the object adheres to this schema to allow for correct chart rendering.

    Interface Schema:

    • title: (string) The heading for the component.
    • text: (string, optional) Supplemental descriptive text.
    • chart: (string) The type of chart to render. Supported values are "pie" and "bar".
    • data: (DataItem[]) An array of objects containing the data points.

    DataItem Schema:

    • label: (string) The identifier for the data point.
    • value: (string) The numeric value represented as a string (will be parsed as a float for charts).
    export interface Component {
  title: string;
  text?: string;
  chart: string;
  data: DataItem[];
}

interface DataItem {
  label: string;
  value: string;
}

  9. Use the focus_planet tool to control the camera

    main

    The focus_planet tool allows the assistant to zoom in on a specific celestial body. The planet parameter must be one of the following values:

    Sun, Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto.

    {
  "name": "focus_planet",
  "parameters": {
    "type": "object",
    "properties": {
      "planet": {
        "type": "string",
        "enum": ["Sun", "Mercury", "Venus", "Earth", "Mars", "Jupiter", "Saturn", "Uranus", "Neptune", "Pluto"]
      }
    },
    "required": ["planet"]
  }
}

  10. Manage microphone state with startRecording() and stopRecording()

    main

    The application allows toggling the microphone without disconnecting the WebRTC session. This is achieved by replacing the active audio track with a silent placeholder track.

    • startRecording(): Requests a new microphone stream and uses sender.replaceTrack(micTrack) to swap the current track with the live microphone track.
    • stopRecording(): Stops the current microphone tracks and uses sender.replaceTrack(placeholderTrack) to swap the live track with a silent track generated by createEmptyAudioTrack() (using an AudioContext destination).
    async function startRecording() { /* ... */ }
function stopRecording() { /* ... */ }

  11. Start a Realtime session with startSession()

    main

    To establish a connection with the Realtime API, call startSession(). This function performs the following orchestration steps:

    1. Fetches a Session Token: Calls /api/session to retrieve a RealtimeClientSecret containing a value (the token) and an optional session.id.
    2. Initializes WebRTC: Creates an RTCPeerConnection and sets up an ontrack listener to pipe remote audio from the model into an HTML <audio> element.
    3. Captures Local Audio: Requests microphone access via navigator.mediaDevices.getUserMedia and adds the resulting audio track to the peer connection.
    4. Creates a Data Channel: Opens an RTCDataChannel named "oai-events" for sending and receiving JSON events.
    5. Negotiates SDP: Generates an SDP offer, sends it via a POST request to REALTIME_CALLS_URL (including the session token in the Authorization header), and sets the returned SDP answer as the remote description.

    If any step fails, the function cleans up tracks and closes the connection.

    async function startSession() {
  // ... implementation details involving fetch('/api/session') 
  // and POST to REALTIME_CALLS_URL
}

  12. Send events to the model with sendClientEvent()

    main

    Use sendClientEvent(message) to send JSON-formatted events to the model over the established RTCDataChannel.

    Requirements:

    • The dataChannel must be in the "open" state.
    • The function automatically assigns a crypto.randomUUID() to message.event_id if one is not provided.

    Common Event Types:

    • conversation.item.create: Used to send tool outputs (function call results) or manual conversation items.
    • response.create: Used to force the model to generate a new response (often required after a tool call).
    • session.update: Used to configure the session's tools and instructions immediately after the connection opens.
    const sendClientEvent = useCallback(
  (message: any) => {
    if (dataChannel?.readyState === "open") {
      message.event_id = message.event_id || crypto.randomUUID();
      dataChannel.send(JSON.stringify(message));
    }
  },
  [dataChannel]
);