WorkArena Benchmark for BrowserGym

Dashboard retrieval tasks follow a specific lifecycle to ensure data is accurately extracted from ServiceNow's Highcharts visualizations:

1. Initialization: The task is initialized with a seed, a SNowInstance, and an optional fixed_config.
2. Setup & Goal Generation: The setup_goal method determines the task's objective (e.g., "What is the maximum value in the 'Incident' chart?") based on the sampled configuration.
3. Page Readiness: The task uses _wait_for_ready to ensure the gsft_main iframe, the Highcharts API, and the WORKARENA_HIGHCHARTS_ALL_LOADED flag are all present.
4. Data Extraction: The task uses _get_charts to find available charts and _read_chart to extract series data, including labels, counts, and percentages.
5. Validation: The validate method compares the agent's response against the ground truth extracted from the chart data, supporting various question types like value, max, min, mean, median, and mode.

Infeasible tasks are designed to evaluate how agents handle scenarios where a requested action (like ordering a specific catalog item) cannot be completed based on information retrieved from a ServiceNow dashboard. These tasks typically follow a pattern:

1. Retrieve Information: The agent analyzes a dashboard (using classes like SingleChartMinMaxRetrievalTask or SingleChartMeanMedianModeRetrievalTask) to answer a question (e.g., 'max', 'mean', 'median', 'mode') regarding incidents.
2. Attempt Action: The agent then attempts to order a specific item from the Service Catalog (e.g., 'Developer Laptop (Mac)', 'iPad mini', 'Sales Laptop').
3. Handle Infeasibility: Because the task is marked as 'infeasible', the agent must navigate the constraints of the environment.

Common task variations include:

• ...InfeasibleTask: The base task for a specific item.
• ...InfeasibleWithReasonTask: A variation that requires the agent to provide a reason for its actions or failure.
• ...MaxOrder...Task: Specifically targets the 'max' value from dashboard charts.
• ...MeanOrder...Task: Specifically targets the 'mean' value.
• ...MedianOrder...Task: Specifically targets the 'median' value.
• ...ModeOrder...Task: Specifically targets the 'mode' value.

WorkArena-L1 consists of atomic tasks covering main ServiceNow UI components. The following example uses the env.task.cheat() method to automatically solve tasks using Playwright.

Note: To evaluate a real agent instead of using the oracle/cheat function, you must use env.step() instead of env.task.cheat().

import random

from browsergym.core.env import BrowserEnv
from browsergym.workarena import ATOMIC_TASKS
from time import sleep


random.shuffle(ATOMIC_TASKS)
for task in ATOMIC_TASKS:
    print("Task:", task)

    # Instantiate a new environment
    env = BrowserEnv(task_entrypoint=task,
                    headless=False)
    env.reset()

    # Cheat functions use Playwright to automatically solve the task
    env.chat.add_message(role="assistant", msg="On it. Please wait...")
    cheat_messages = []
    env.task.cheat(env.page, cheat_messages)

    # Send cheat messages to chat
    for cheat_msg in cheat_messages:
        env.chat.add_message(role=cheat_msg["role"], msg=cheat_msg["message"])

    # Post solution to chat
    env.chat.add_message(role="assistant", msg="I'm done!")

    # Validate the solution
    reward, stop, message, info = env.task.validate(env.page, cheat_messages)
    if reward == 1:
        env.chat.add_message(role="user", msg="Yes, that works. Thanks!")
    else:
        env.chat.add_message(role="user", msg=f"No, that doesn't work. {info.get('message', '')}")

    sleep(3)
    env.close()

WorkArena++ (L2/L3) contains compositional tasks that evaluate planning, reasoning, and memorization. Use get_all_tasks_agents to sample tasks.

Note: This example uses the oracle/cheat function. For agent evaluation, use env.step().

import random

from browsergym.core.env import BrowserEnv
from browsergym.workarena import get_all_tasks_agents
 
AGENT_L2_SAMPLED_SET = get_all_tasks_agents(filter="l2")
 
AGENT_L2_SAMPLED_TASKS, AGENT_L2_SEEDS = [sampled_set[0] for sampled_set in AGENT_L2_SAMPLED_SET], [
    sampled_set[1] for sampled_set in AGENT_L2_SAMPLED_SET
]
from time import sleep

for (task, seed) in zip(AGENT_L2_SAMPLED_TASKS, AGENT_L2_SEEDS):
    print("Task:", task)

    # Instantiate a new environment
    env = BrowserEnv(task_entrypoint=task,
                    headless=False)
    env.reset()

    # Cheat functions use Playwright to automatically solve the task
    env.chat.add_message(role="assistant", msg="On it. Please wait...")
    
    for i in range(len(env.task)):
        sleep(1)
        env.task.cheat(page=env.page, chat_messages=env.chat.messages, subtask_idx=i)
        sleep(1)
        reward, done, message, info = env.task.validate(page=env.page, chat_messages=env.chat.messages)
   
    if reward == 1:
        env.chat.add_message(role="user", msg="Yes, that works. Thanks!")
    else:
        env.chat.add_message(role="user", msg=f"No, that doesn't work. {info.get('message', '')}")

    sleep(3)
    env.close()