Bananalyzer AI Agent Evaluation Framework

To run an agent with bananalyzer, you must create a Python file (conventionally named bananalyzer-agent.py) that contains a class inheriting from AgentRunner.

When the CLI loads your file, it uses ast to inspect the class definitions and ensures the class is a valid AgentRunner instance. Because pytest creates a new global context for each test run, your AgentRunner implementation must be concurrency safe.

Requirements

1. The file must contain at least one class that inherits from AgentRunner.
2. The class must be instantiable without arguments (or via the mechanism used by the runner).
3. The class must be thread/process safe for parallel execution via xdist if using the -n flag.

When saving a captured page, create a metadata dictionary (the values object) to track the source, goal, and evaluation criteria. This object is typically appended to an examples.json file.

Key fields for the values object include:

• id: A unique UUID string.
• url: The captured page URL.
• source: Set to "mhtml".
• type: The type of fetch (e.g., "links_fetch").
• goal: A string describing the extraction task.
• evals: A list of evaluation objects (e.g., {"type": "json_match", "expected": []}).

import uuid

values = {
    "id": str(uuid.uuid4()),
    "url": page.url,
    "source": "mhtml",
    "category": "government",
    "subcategory": "download",
    "type": "links_fetch",
    "goal": 'Fetch all of the links to the documents on the current page...', 
    "evals": [{"type": "json_match", "expected": []}],
}

The BananalyzerPytestPlugin automatically collects metadata from pytest markers that start with the prefix bananalyzer_. This allows you to group and report test results by specific dimensions (e.g., fields, classes, or custom properties) in the final terminal summary.

Marker Format: Use @pytest.mark.bananalyzer_<property_name> where <property_name> is the value you want to track.

Example Usage:

import pytest

@pytest.mark.bananalyzer_field("user_id")
@pytest.mark.bananalyzer_type("fetch")
def test_example():
    assert True

When the test suite finishes, the plugin will generate a summary table showing the pass/fail rates for field and type separately.

To capture page content as MHTML using Chrome DevTools Protocol (CDP), you must initialize a Playwright instance with a CDP session. This allows you to access low-level browser functions like Page.captureSnapshot.

from playwright.async_api import async_playwright

# Setup a playwright instance
p = await async_playwright().__aenter__()
browser = await p.chromium.launch(headless=False)
context = await browser.new_context(viewport={"width": 1280, "height": 1280})
page = await context.new_page()
client = await context.new_cdp_session(page)
page.set_default_navigation_timeout(50000)

To use the Bananalyzer reporting features, you must register the BananalyzerPytestPlugin. If you are using pytest-xdist for parallel execution, you must load this plugin through a conftest.py file to ensure it is properly registered on each worker node.

While the plugin is dynamically imported by the bananalyzer.runner.runner module, manual integration in your test suite involves ensuring the plugin is active in your pytest configuration.

# In your conftest.py
# The plugin is typically registered automatically if using the bananalyzer runner,
# but for manual pytest usage with xdist, ensure the plugin is available to workers.

You can define a custom scrape function that uses Playwright's query_selector_all to find specific elements and extract their attributes (like title and href) into a structured list of dictionaries.

from playwright.async_api import Page
from typing import List, Dict, Any

async def scrape(page: Page) -> List[Dict[str, Any]]:
    link_elements = await page.query_selector_all("ul.nav.navbar-nav li a.subSubButton")
    document_links: List[Dict[str, Any]] = []
    for link in link_elements:
        name = await link.get_attribute("title")
        url = await link.get_attribute("href")
        if name and url:
            document_links.append({"name": name, "url": url})
    return document_links

The enrich_report function modifies an existing JUnit XML report file by injecting additional metadata into the <properties> section of every <testsuite> element.

By default, it adds:

• bananalyzer_version: The current version of the bananalyzer package.
• git_commit_sha: The current Git commit SHA (retrieved from the GITHUB_SHA environment variable or via git rev-parse HEAD).

This is useful for CI/CD pipelines to ensure test results are traceable to specific code versions and tool versions.

from bananalyzer.junit import enrich_report

# Path to your existing JUnit XML report
report_path = "path/to/junit_report.xml"

# This will modify the file in-place to add version and git metadata
enrich_report(report_path)

The Example class and ExampleType enum define the schema for data used in evaluations. You can retrieve all available examples using get_all_examples().

from bananalyzer import get_all_examples, Example, ExampleType

examples = get_all_examples()
# Each example follows the Example schema

The XDistArgs class defines settings for distributed testing using pytest-xdist.

Fields:

• dist: The distribution mode. Supported values are: "load", "loadscope", "loadfile", "loadgroup", "worksteal", or "no".
• n: The number of workers. Can be an int, or the literals "logical" or "auto".

from bananalyzer.schema import XDistArgs

xdist_config = XDistArgs(dist="load", n="auto")

The AgentRunnerClass schema is used to specify a class that can be dynamically instantiated, typically for running different types of agents.

Fields:

• class_name: The name of the class.
• class_path: The full module path to the class.

from bananalyzer.schema import AgentRunnerClass

runner_config = AgentRunnerClass(class_name="MyAgent", class_path="my_module.agents.MyAgent")