MedAgentBench

The MedAgentBench task uses a specific prompt structure to guide medical LLM agents. Agents are expected to interact with a FHIR server using three specific command formats. The agent must only output one of these formats per turn and should not include any other text.

1. GET Requests: Used to retrieve data. Format: GET url?param_name1=param_value1&param_name2=param_value2... Note: The system automatically appends &_format=json to these requests.

2. POST Requests: Used to send data. Format:

   POST url
   [JSON payload]

3. FINISH Command: Used to conclude the task and provide answers. Format: FINISH([answer1, answer2, ...]) Note: The content inside the parentheses must be a JSON-loadable list.

If the agent provides an output that does not match one of these three patterns, the task will return a SampleStatus.AGENT_INVALID_ACTION status.

The AssignmentConfig class is the primary configuration object used to define how agents and tasks are distributed and executed. It uses Pydantic for validation and supports several key components:

Key Fields

• assignments: A list of Assignment objects. You can provide these as a list of dictionaries where each dictionary contains agent and task keys. If a single string is provided for agent or task, it is automatically converted into a list.
• concurrency: A ConcurrencyConfig object specifying how many instances of each agent and task should run simultaneously.
• definition: A DefinitionConfig object that maps agent and task names to their respective InstanceFactory implementations.
• output: A string defining the output directory. If not provided, it defaults to output/{TIMESTAMP} where {TIMESTAMP} is replaced by the current date and time (e.g., output/2023-10-27-14-30-05).

Validation Behavior

• Automatic Expansion: If an assignment dictionary contains a single string for agent or task, the system expands it to cover all combinations of the provided agents and tasks.
• Unused Cleanup: The configuration automatically removes agents or tasks from definition and concurrency if they are not present in the assignments list.
• Deduplication: If duplicate (agent, task) pairs are provided in assignments, only the first occurrence is kept, and a warning is issued.

# Example conceptual structure of an AssignmentConfig
config = AssignmentConfig(
    assignments=[
        {"agent": "GPT-4", "task": "diagnosis"},
        {"agent": ["Llama-3", "Claude-3"], "task": "treatment"}
    ],
    concurrency={
        "agent": {"GPT-4": 2, "Llama-3": 4, "Claude-3": 4},
        "task": {"diagnosis": 1, "treatment": 1}
    },
    definition={
        "agent": {"GPT-4": factory_gpt4, "Llama-3": factory_llama, "Claude-3": factory_claude},
        "task": {"diagnosis": factory_diag, "treatment": factory_treat}
    },
    output="my_results/{TIMESTAMP}"
)

To use MedAgentBench as a task within the AgentBench framework, you must instantiate the MedAgentBench class. This class requires a configuration dictionary containing specific keys to load the dataset, available functions, and the FHIR API endpoint.

Required Configuration Keys:

• data_file: Path to the JSON file containing the task data (context and instructions).
• func_file: Path to the JSON file containing the list of available FHIR functions.
• fhir_api_base: The base URL for the FHIR server. The class will attempt to verify this connection upon initialization.

Optional Configuration Keys:

• max_round: The maximum number of tool-calling rounds allowed per task (defaults to 5).

Important Setup Note: This task requires a reference solution file located at src/server/tasks/medagentbench/refsol.py to perform evaluations. If this file is missing, the task will fail to initialize.

# Example initialization (conceptual)
configs = {
    "data_file": "path/to/data.json",
    "func_file": "path/to/functions.json",
    "fhir_api_base": "http://your-fhir-server:8080/fhir",
    "max_round": 10
}
med_agent_bench_task = MedAgentBench(**configs)

The send_get_request utility performs an HTTP GET request and returns a standardized dictionary containing the response.

Arguments:

• url (str): The target URL.
• params (dict, optional): Query parameters.
• headers (dict, optional): HTTP headers.

Returns:

• A dictionary with keys status_code and data (if successful).
• A dictionary with key error (if an exception occurs).

Note: If the response Content-Type is application/json, the data field will contain the parsed JSON object; otherwise, it contains the raw response text.

from src.server.tasks.medagentbench.utils import send_get_request

# Example: Successful JSON request
result = send_get_request('https://api.example.com/resource', params={'id': '123'})
if 'error' not in result:
    print(f"Status: {result['status_code']}", result['data'])
else:
    print(f"Error occurred: {result['error']}")