ClawBench Documentation

ClawBench is an open-source benchmark designed to evaluate AI browser agents on everyday online tasks (e.g., booking travel, ordering food, applying for jobs, managing email) across live websites.

Key features include:

• Task Coverage: V1 contains 153 tasks across 144 websites; V2 contains 130 tasks.
• Evaluation Methodology: Measures end-to-end task success using a 5-layer recording pipeline and an agentic evaluator that compares agent runs against human references.
• Environment: Runs on any Chrome browser and utilizes Docker-isolated harnesses for execution.
• Performance Context: Current top scores are approximately 33.3%.

ClawBench is an open-source benchmark designed to evaluate the performance of AI browser agents on everyday online tasks, such as booking hotels, ordering food, submitting resumes, and managing emails across real-world websites.

Benchmark Versions

• V1: Located in test-cases/v1/, containing 153 tasks covering 144 websites.
• V2: Located in test-cases/v2/, containing 130 tasks.

Evaluation Methodology

ClawBench measures end-to-end task completion rates using:

1. A 5-layer recording pipeline.
2. An agentic evaluator that compares agent actions against human reference trajectories.

Currently, the highest recorded score is 33.3%.

The ClawBench Runtime Server is a Python-based instrumentation server (FastAPI) that connects to Chromium via the Chrome DevTools Protocol (CDP). It is used to instrument browser sessions for benchmarking by:

• Capturing DOM actions: Injected listeners store actions in actions.jsonl format.
• Capturing screenshots: Saves screenshots via CDP.
• Logging HTTP requests: Monitors and can block specific requests using the CDP Fetch domain.
• Recording video: Records the local Xvfb display to recording.mp4 (local mode) or manages remote provider URLs (provider mode).
• Bridging remote browsers: Provides a credential-free WebSocket bridge and discovery endpoints when using remote browsers (e.g., Browserbase), keeping sensitive credentials out of harness logs and metadata.

The structeval-promotion-pr-tracker.md document maintains a registry of pull requests from various external repositories that have added StructEval (a TMLR 2025 benchmark for structured-output generation and cross-format conversion) to their respective catalogs.

This tracker is useful for monitoring the adoption of StructEval across different domains such as:

• LLM Evaluation & Benchmarking: General LLM, MLLM, and VLM evaluation lists.
• Domain-Specific Catalogs: JSON Schema, XML, YAML, CSV, and Code-specific resource lists.
• Specialized AI Fields: Generative UI, AI Agents, Prompt Engineering, and Multimodal Reasoning.

The tracker categorizes submissions by repository, pull request link, fit/rule check details, and current status (Open, Merged, or Closed).

The ClawBench Chrome Extension is designed for browser fingerprint hardening during benchmark runs.

Key Responsibilities:

• Stealth Injection: Injects stealth.js at document_start in the page's MAIN world to patch window and navigator objects before any page JavaScript executes.
• Anti-Bot Detection: Applies patches for webdriver, plugins, WebGL, permissions, and other browser fingerprints to reduce detection by systems like reCAPTCHA and Cloudflare Turnstile.

Important Note on Data Flow:

• The extension does not send data to the server.
• Action capture (clicks, keystrokes), screenshot capture, request logging, request interception, and tab activation are handled by the runtime server via the Chrome DevTools Protocol (CDP).

Because EdgeBench submits an agent-controlled archive, the Judge does not automatically trust the intercepted flag in interception.json. Instead, it recomputes Stage-1 verification by matching submitted requests against the task's eval_schema (URL pattern, method, and body/params).

To prevent an agent from fabricating matching requests (full tamper-resistance), use a shared secret:

1. Set CLAWBENCH_EVIDENCE_SECRET in both the trusted runtime/entrypoint and the Judge via SFORGE_JUDGE_EXTRA_ENV.
2. The runtime will sign each intercepted request using HMAC-SHA256(secret, canonical-json(request)).
3. The Judge will reject any evidence that is unsigned or forged.

Warning: Never provide CLAWBENCH_EVIDENCE_SECRET to the agent.

ClawBench uses a multi-stage evaluation process to grade AI agents on their ability to complete online tasks. The primary metrics used in the V2 (Hermes) leaderboard are:

• Intercepted: A deterministic Stage 1 metric. It checks if the final HTTP request made by the agent matched the task's target URL and method.
• Reward (lenient): A Stage 2 metric where a judge LLM (deepseek/deepseek-v4-pro) evaluates if the instruction was fulfilled using a "no contradiction $\rightarrow$ match" rubric.
• Reward (strict): A Stage 2 metric using the same judge but with a stricter rubric where any ambiguity results in a mismatch.

In V1 evaluations, the Pass Rate was determined by a Claude Code agentic-eval subagent comparing runs against human reference trajectories.

ClawBench is specifically designed to evaluate agents on live consumer websites performing everyday tasks with end-to-end recordings.

Key differentiators from other benchmarks include:

• WebArena: Uses synthetic web apps/self-hosted replicas; ClawBench uses live consumer sites.
• GAIA: Focuss on closed-book text + tools; ClawBench is browser-centric and end-to-end.
• SWE-bench: Focuses on software engineering/GitHub; ClawBench focuses on non-code consumer workflows.
• Mind2Web: Uses static traces; ClawBench uses dynamic live websites.
• WebVoyager: Uses LLM-judge-only; ClawBench uses interception-graded metrics.
• TheAgentCompany: Focuses on enterprise/office sandboxes; ClawBench focuses on consumer everyday tasks.

The eval_schema field in task.json allows you to define a request interceptor. When a request matches the specified criteria, the interceptor blocks the request, records it in data/interception.json, and stops the container. This is useful for detecting when an agent attempts an irreversible action (like a POST request to a submission endpoint).

Fields in eval_schema:

• url_pattern (Required): A regex string matched against request URLs using re.search().
• method (Required): The HTTP method to match (e.g., POST, GET).
• body (Optional): Exact key-value filters for parsed request bodies.
• params (Optional): Exact key-value filters for URL query parameters.

Note: If no interception is needed, use "__PLACEHOLDER_WILL_NOT_MATCH__" as the url_pattern to allow the session to run until other stop conditions are met.