DOCSEARCHbyAlgolia
Submit Docs

$ au$-Bench

repository·main·Indexed 23 days ago

https://github.com/sierra-research/tau2-bench

A simulation framework for benchmarking tool-using agents in real-world customer service scenarios. It supports text-based turn-taking and full-duplex voice interactions across domains including airline, retail, telecom, and banking knowledge. The framework includes a CLI for running evaluations, an Agent-to-Agent (A2A) protocol for agentified benchmarking via Green and White agents, and tools for hyperparameter experiments and result analysis.

Tokens
64.2K
Snippets
119
Records
265
Agent score
81%

What's inside tau2-bench

  1. What is Tau2 Gym

    main
    Tau2 Gym is a Gymnasium-compatible environment designed for evaluating conversational agents within the $\tau$-bench framework. It provides a standardized interface that allows developers to run agents step-by-step in controlled simulation environments, following the standard Gymnasium API for reinforcement learning.

  2. Overview of $\tau$-Bench domains and modes

    main

    $\tau$-Bench is a simulation framework for evaluating customer service agents. Each domain consists of a policy (rules the agent must follow), tools (available to the agent), and tasks (evaluation scenarios).

    Available Domains

    • mock: Basic testing domain.
    • airline: Airline customer service.
    • retail: Retail customer service.
    • telecom: Telecom customer service.
    • banking_knowledge: Knowledge-retrieval-based domain using RAG pipelines and document search.

    Evaluation Modes

    ModeDescription
    Text (half-duplex)Turn-based chat with tool use.
    Voice (full-duplex)End-to-end audio via realtime providers (OpenAI, Gemini, xAI).

  3. How Audio Native Providers architecture works

    main

    Audio native providers use a DiscreteTimeAdapter interface to connect different voice AI services to the simulation framework.

    All providers (except livekit, which uses a cascaded pipeline) extend the DiscreteTimeAdapter base class. The base class manages core simulation logic including:

    • TickResult creation
    • Audio buffering and capping
    • Proportional transcript distribution
    • Barge-in buffer clearing
    • Tool result queuing
    • Tick timing enforcement
    • Cumulative state tracking

    Each provider implementation typically consists of three components:

    1. provider.py: The WebSocket or API client responsible for connection, sending audio, and receiving events.
    2. events.py: Pydantic models defining provider-specific event types.
    3. discrete_time_adapter.py: Implements _execute_tick() to convert audio, send/receive data, and process events.

  4. How voice ID resolution works

    main

    The $\tau$-Bench framework resolves voice IDs for personas using a specific priority order:

    1. Environment Variable: It first looks for TAU2_VOICE_ID_<PERSONA_NAME_UPPER> in the environment.
    2. Built-in Default: If the environment variable is not set, it falls back to the default defined in src/tau2/data_model/voice_personas.py.

    Note on Pre-sampled Configs: If you are using tasks_voice.json files in domain data directories, these files store persona names, not voice IDs. The voice ID is resolved dynamically at synthesis time based on the persona name, meaning you can update your .env with new voice IDs without needing to regenerate your task configuration files.

  5. How to end a user simulation session

    main

    To manage the lifecycle of a simulated conversation, you must use specific termination tokens based on the state of the interaction:

    • Task Success: If the instruction goal is satisfied, generate the ###STOP### token.
    • Agent Transfer: If the agent clearly indicates you are being transferred to another representative, generate the ###TRANSFER### token.
    • Insufficient Information: If the scenario does not provide enough information to continue the conversation, generate the ###OUT-OF-SCOPE### token.
    ###STOP###
###TRANSFER###
###OUT-OF-SCOPE###

  6. Retail Agent Operational Policy

    main

    As a retail agent, you must adhere to the following operational constraints and procedures:

    Authentication

    • Mandatory Authentication: You must authenticate the user identity at the start of every conversation by locating their user id via email or via name + zip code. This is required even if the user provides their user id upfront.
    • Single User Constraint: You can only assist one user per conversation. Deny any requests related to other users.

    Interaction Rules

    • Confirmation Required: Before any database-updating action (cancel, modify, return, exchange), you must list the action details and obtain explicit user confirmation (e.g., 'yes').
    • Tool Call Protocol: Make at most one tool call at a time. Do not respond to the user and make a tool call simultaneously. If you respond to the user, do not make a tool call in the same turn.
    • Information Integrity: Do not invent information, procedures, or subjective recommendations. Use only provided tools and user input.
    • Human Handoff: If a request is outside your scope, transfer to a human agent by:
      1. Calling the transfer_to_human_agents tool.
      2. Sending the exact message: YOU ARE BEING TRANSFERRED TO A HUMAN AGENT. PLEASE HOLD ON.

  7. How communication protocols work in tau2 agents

    main

    Agents in $\tau$-Bench operate under two distinct communication protocols, which determine how they interact with users and environments:

    1. Half-Duplex (HalfDuplexAgent): Used for turn-based text conversations. Only one party (the agent or the user) speaks at a time. This is the standard for text-based evaluations.
    2. Full-Duplex (FullDuplexAgent): Used for streaming or tick-based interactions, such as voice. Both parties can speak simultaneously, and the agent processes input in discrete time intervals (ticks).

    Choosing the correct base class is the first step in developing a new agent.

  8. Understand the Tau2 Domain structure

    main

    Each Tau2 domain is organized into a specific folder structure containing data models, tools, and environment definitions.

    Core Files

    • data_model.py: Implements a subclass of DB (e.g., FlightDB(DB)) to define the domain's database.
    • user_data_model.py (Optional): Implements a subclass of DB for user-specific data.
    • tools.py: Implements a subclass of ToolKitBase (e.g., AirlineTools(ToolKitBase)) to define domain tools.
    • user_tools.py (Optional): Implements a subclass of ToolKitBase for user-facing tools.
    • environment.py: The entry point for the domain. It must implement:
      • get_environment(): Returns an Environment instance.
      • get_tasks(): Returns a list of tasks.
      • get_tasks_split(): Returns a dictionary mapping split names to lists of task IDs.
    • utils.py: Contains domain-specific utility functions (e.g., data paths).

  9. Standard vs Custom Submissions

    main

    The leaderboard distinguishes between two submission types based on whether you have modified the standard $\tau$-bench evaluation scaffold.

    Standard Submissions (Default)

    Use this if you are evaluating an off-the-shelf, general-purpose LLM using the default $\tau$-bench tools, prompts, and evaluation protocol.

    • Configuration: You do not need to specify submission_type in your submission.json; it defaults to "standard".

    Custom Submissions

    Use this if your approach differs from the standard evaluation. This includes:

    • Modified Scaffolds: Using multi-model routers, ensembles, additional tools, or modified orchestration/prompts.
    • Domain-Specific Training: Models fine-tuned on $\tau$-bench domains or trained using $\tau$-bench tasks/reward signals.

    Requirements for Custom Submissions:

    1. Set submission_type to "custom" in submission.json.
    2. Provide detailed explanations in methodology.notes.
    3. Link to your implementation (GitHub, paper, etc.) in the references array.
    4. Set methodology.verification.modified_prompts to true if you modified any prompts.

  10. Understand observations and the info dictionary

    main

    Both reset() and step() return an observation (string) and an info (dictionary).

    Observation Format

    The observation is a string of the conversation history formatted as "role: content" separated by newlines:

    • user: ...: Messages from the user.
    • assistant: ...: Messages from the agent (plain text or tool calls).
    • tool: ...: JSON results from tool executions.

    Note: In solo_mode=True, the initial observation is None.

    Info Dictionary

    The info dictionary contains metadata:

    • tools: A list of available tools/actions in the current domain.
    • policy: A string defining the agent's behavior and constraints.
    • simulation_run: A JSON representation of the current simulation state.
    # Example of accessing info contents
observation, info = env.reset()

# View available tools
for tool in info['tools']:
    print(f"Tool: {tool.name}")
    print(f"Description: {tool.description}")
    print(f"Parameters: {tool.parameters}")

# View agent policy
print(f"Agent must follow this policy: {info['policy']}")

  11. Use AgentGymEnv to control an agent step-by-step

    main

    To interact with the environment as an external controller (e.g., for Reinforcement Learning or manual testing), use AgentGymEnv.

    Key Workflow

    1. Check Turn: Before providing an action, check is_agent_turn to ensure the environment is waiting for input.
    2. Provide Action: Use set_action(action) to pass the agent's next move. This method will raise a RuntimeError if called when it is not the agent's turn.
    3. Observe: Access the current conversation history via the observation property, which returns a list of messages.

    Thread Safety Rules

    • Only call set_action() when is_agent_turn is True.
    • The orchestrator thread is a daemon; it will automatically terminate when the main thread exits.

  12. How Green and White Agents work together

    main

    Agentification uses the A2A (Agent-to-Agent) protocol to turn a benchmark into a service. The system relies on two distinct agent roles:

    • Green Agent 🟢 (Assessment Manager): Acts as the benchmark orchestrator. It receives evaluation requests, sets up the Tau-Bench environment, manages the conversation between the benchmark and the target agent, and reports the final scores.
    • White Agent ⚪ (Target Agent): The agent being evaluated. It must implement the A2A protocol. It receives instructions and responds via tool calls or messages, remaining unaware that it is part of a benchmark.

    Workflow:

    1. A request is sent to the Green Agent containing the White Agent's URL and the benchmark configuration.
    2. The Green Agent instantiates the Tau-Bench environment.
    3. The Green Agent forwards user messages to the White Agent and collects responses.
    4. The Green Agent evaluates the performance and returns the results.