OpenAI Realtime Agents

To build your own multi-agent voice application using this project's templates:

1. Create a new agent set configuration.
2. Register the new configuration in src/app/agentConfigs/index.ts.
3. The new agent will then be available for selection in the UI via the 'Scenario' dropdown menu.

Each agentConfig can define:

• instructions: The system prompt for the agent.
• tools: The list of available tools.
• toolLogic: Custom logic that runs when a tool is called. By default, tool calls return True. If you define toolLogic, it will execute your specific code and return an object to the conversation (useful for RAG context or external API data).

This project is a Next.js TypeScript application. To set up the development environment:

1. Install dependencies:

   npm i

2. Configure your OpenAI API Key. You can either add OPENAI_API_KEY to your shell environment (e.g., .bash_profile) or create a .env file by copying .env.sample:

   cp .env.sample .env

   Then, add your key to the .env file.
3. Start the development server:

   npm run dev

4. Access the application at http://localhost:3000.

The demo interface provides several controls and views:

• Scenario Dropdown: Select different agent scenarios. Selecting a scenario automatically switches the active agent in the Agent dropdown.
• Conversation Transcript (Left): Displays the dialogue, including tool calls, tool call responses, and agent handoffs. Click non-message elements to expand them.
• Event Log (Right): Shows real-time client and server events. Click an event to view the full JSON payload.
• Bottom Controls:
  • Disconnect: End the session.
  • Voice Activity Detection: Toggle between automated VAD or Push-to-Talk (PTT).
  • Audio Playback: Toggle audio output on/off.
  • Logs: Toggle visibility of logs.

If you need assistance creating prompts that follow the conventions used in the customerServiceRetail example (including state machine definitions), you can use the following resources:

• Metaprompt File: src/app/agentConfigs/voiceAgentMetaprompt.txt
• Voice Agent Metaprompter GPT: https://chatgpt.com/g/678865c9fb5c81918fa28699735dd08e-voice-agent-metaprompt-gpt

For complex flows (like CustomerServiceRetail), you can define an agent graph by assigning downstreamAgents to each agent and then using the injectTransferTools utility to wrap them with the necessary transfer capabilities.

import authentication from "./authentication";
import returns from "./returns";
import sales from "./sales";
import simulatedHuman from "./simulatedHuman";
import { injectTransferTools } from "../utils";

authentication.downstreamAgents = [returns, sales, simulatedHuman];
returns.downstreamAgents = [authentication, sales, simulatedHuman];
sales.downstreamAgents = [authentication, returns, simulatedHuman];
simulatedHuman.downstreamAgents = [authentication, returns, sales];

const agents = injectTransferTools([
  authentication,
  returns,
  sales,
  simulatedHuman,
]);

export default agents;

Assistant messages are checked for safety and compliance. The guardrail logic is implemented in src/app/App.tsx.

Workflow:

1. When a response.text.delta stream starts, the message is marked as IN_PROGRESS.
2. If the server emits guardrail_tripped, the message is marked as FAIL.
3. If the server emits response.done, the message is marked as PASS.

To modify how moderation is triggered or displayed, search for the guardrail_tripped token within src/app/App.tsx and update the logic there.

The allAgentSets object provides a registry of pre-configured agent scenarios. Each key in the object corresponds to a specific agentic pattern, and the value is an array of RealtimeAgent objects.

Available scenario keys:

• simpleHandoff: A basic handoff scenario.
• customerServiceRetail: A retail-focused customer service flow.
• chatSupervisor: A chat supervisor pattern.

You can use these keys to load specific sets of agents for your application.

Tools are defined using the Tool interface, which follows the standard function calling schema. A Tool must have a type of "function", a name, a description, and a parameters object of type ToolParameters.

export interface Tool {
  type: "function";
  name: string;
  description: string;
  parameters: ToolParameters;
}

export interface ToolParameters {
  type: string;
  properties: Record<string, ToolParameterProperty>;
  required?: string[];
  additionalProperties?: boolean;
}