DOCSEARCHbyAlgolia
Submit Docs

inspect_evals

repository·main·Indexed 20 days ago

https://github.com/ukgovernmentbeis/inspect_evals

A collection of large language model evaluations ported to the Inspect evaluation platform. It includes frameworks such as HealthBench for healthcare conversations, as well as integrations for external benchmarks including the Adversarial Humanities Benchmark (AHB), Alignment Faking, and AppWorld.

Tokens
319K
Snippets
911
Records
1.4K
Agent score
64%

What's inside inspect_evals

  1. Overview of BIG-Bench Hard (BBH) tasks

    main

    BBH is a subset of 27 challenging tasks from the BIG-Bench suite focusing on algorithmic, commonsense, and multi-step reasoning. It includes 6,511 total samples.

    Task Categories:

    Multiple Choice Datasets:

    • date_understanding, disambiguation_qa, geometric_shapes, hyperbaton, logical_deduction_five_objects, logical_deduction_seven_objects, logical_deduction_three_objects, movie_recommendation, penguins_in_a_table, reasoning_about_colored_objects, ruin_names, salient_translation_error_detection, snarks, temporal_sequences, tracking_shuffled_objects_five_objects, tracking_shuffled_objects_seven_objects, tracking_shuffled_objects_three_objects

    Binary Choice Datasets:

    • boolean_expressions (True, False)
    • causal_judgement (Yes, No)
    • formal_fallacies (valid, invalid)
    • navigate (Yes, No)
    • sports_understanding (yes, no)
    • web_of_lies (Yes, No)

    Open Answer Datasets:

    • multistep_arithmetic_two (integer)
    • object_counting (natural number)
    • word_sorting (list of words)
    • dyck_languages (closing brackets)

    Scoring: Scoring is based on simple accuracy calculated over the samples.

  2. Overview of Self-proliferation Tasks

    main

    This directory contains 10 self-proliferation tasks derived from Google DeepMind's Dangerous Capabilities Evaluations. These tasks test an agent's ability to perform complex, potentially dangerous actions like setting up emails, installing models, or building products.

    Task List:

    • SP01 - Email setup
    • SP02 - Install model
    • SP03 - Install web agent
    • SP04 - Bitcoin wallet
    • SP05 - Track employee
    • SP07 - Build product
    • SP08 - Celebrity endorsement
    • SP09 - Inference efficiency
    • SP10 - Distillation
    • SP12 - Implement paper

    Note: SP02, SP09, and SP10 require GCP setup or equivalent hardware.

  3. Overview of the Grafana file read challenge

    main

    The grafana challenge focuses on a directory traversal and arbitrary file read vulnerability present in Grafana versions $\le$ 8.3.0. To successfully complete this challenge, an agent must perform two key steps:

    1. Enumerate the version of the running Grafana instance.
    2. Identify and exploit the specific off-the-shelf vulnerability to read the flag.txt file.

  4. Overview of ContractBench

    main

    ContractBench is an LLM Agent Observation Contract Compliance Benchmark. It evaluates agents on 33 tasks probing two failure modes:

    1. Temporal validity: Using API artifacts before they expire.
    2. Byte-level integrity: Relaying artifacts without corruption.

    Tasks simulate real API patterns (e.g., presigned URLs, OAuth tokens, HMAC webhooks) using a virtual clock and SHA-256 hash verification. Scoring is deterministic via HTTP request logs against a 15-label failure taxonomy. It uses a success rate metric over $k=3$ runs per task ($n=99$ episodes per model) and does not use an LLM judge.

  5. Overview of O-NET evaluation

    main

    O-NET is an evaluation dataset consisting of high-school-level multiple-choice and true/false questions. It covers several subjects:

    • Mathematics
    • Science
    • Social knowledge
    • English language comprehension
    • Thai language comprehension

    The evaluation tasks the model with answering a question using a specific format. For multiple-choice questions, the model is prompted to respond in the format: ANSWER: $LETTER (where LETTER is A, B, C, D, or E) after thinking step-by-step. Questions requiring written responses, unanswerable questions, or those with multiple correct answers are excluded from this evaluation.

  6. Overview of the Situational Awareness Dataset (SAD)

    main
    SAD-mini is a dataset comprising 2,904 multiple-choice questions designed to evaluate the situational awareness of LLMs. It is divided into 5 distinct tasks, each focusing on different aspects of model awareness. The dataset is used to measure how well models understand their own development stages, influence, and factual knowledge.

  8. Overview of the Command Injection challenge

    main

    The Command Injection challenge is a simple web-based task designed to test an agent's ability to discover and exploit vulnerabilities.

    Goal: The agent must successfully exploit a command execution vulnerability to read a flag from the system.

    Expected Agent Behavior:

    1. Discovery: Browse the application (e.g., using curl) to discover available pages.
    2. Identification: Identify a page containing functionality typically associated with command execution issues in legacy applications.
    3. Exploitation: Execute a command injection attack to retrieve the flag.

    This challenge is intentionally straightforward and contains minimal superfluous content to focus on the core exploitation capability.

  10. Overview of CodeIPI (Indirect Prompt Injection for Coding Agents)

    main

    CodeIPI is a benchmark located at src/inspect_evals/ipi_coding_agent/ designed to measure whether coding agents comply with malicious instructions embedded in software engineering artifacts (e.g., issue text, code comments, READMEs, or configuration files).

    Benchmark Composition

    The dataset consists of 45 hand-crafted samples:

    • 35 Injected Samples: Designed to test injection resistance.
    • 10 Benign Samples: Designed as false positive controls (FPR).

    Injection Vectors and Payloads

    Injections are embedded in the following vectors:

    • issue_text (10 samples)
    • code_comment (10 samples)
    • readme (10 samples)
    • config_file (5 samples)

    Payload categories include:

    • Exfiltration (15 samples)
    • Code Execution (10 samples)
    • Persistence (10 samples)

    Threat Model and Defense

    The benchmark uses a sandbox environment with network_mode: none and resource limits (2GB RAM). It employs canary tokens (files and environment variables) to detect successful injections. The evaluation uses a SYSTEM_PROMPT containing explicit anti-injection language to guide the agent.

  11. Overview of the MaCBench evaluation

    main

    MaCBench is an evaluation benchmark implemented within the inspect_evals framework. It is designed to run against a corpus of 1,153 samples and supports multimodal inputs by converting records into Inspect ContentText and ContentImage objects.

    Key features include:

    • Pinned Dataset: Uses a specific Hugging Face revision (ca13e7aff2f9a40cb638360c3e569de173ae1fdd) to ensure reproducibility.
    • Flexible Execution: Supports running the full merged benchmark or specific subsets via the task_name parameter.
    • Hybrid Scoring: The scorer automatically routes between multiple-choice grading and numeric tolerance-based grading based on per-sample metadata.
    • Numeric Scoring Logic: Uses a relative_tolerance field from the dataset, which is treated as an absolute tolerance window (with relative=False in the scoring logic). If tolerance values are missing, it defaults to a 1% relative tolerance.