Inspect AI

Inspect includes agentic tools designed to assist models with long-horizon tasks, specialized knowledge, and human interaction. These tools include:

Agentic Tools

• Skill: Provides specialized knowledge and expertise for specific tasks via agent skill specifications.
• Update Plan: Enables the model to track steps and progress during long-horizon tasks.
• Memory: Allows for storing and retrieving information using a memory file directory.
• Think: Provides a dedicated thinking step for the model to process information before generating a final answer.
• Intervention: Enables the model to ask questions or send notifications to the user.

The inspect ctl command is used to interact with running Inspect processes. It is organized into logical groups based on the resource being managed: task, sample, config, and process.

Command Groups

• task: Manages logical tasks within a running process (e.g., list, cancel, pause, resume).
• sample: Manages individual samples or task samples (e.g., list, show, errors, messages, cancel).
• config: Views or modifies running configuration knobs (e.g., --max-samples, --max-connections).
• process: Manages the Inspect process itself (e.g., list, keep, release, pause).
• eval-set: (Planned) Manages groups of evaluations.

inspect ctl
├── task
├── sample
├── config
├── process
└── eval-set

The Control Channel is being rolled out in phases. The current status of features is as follows:

The disk-usage skill is designed to analyze disk space usage, filesystem mounts, and storage allocation on Linux systems. It is specifically useful for addressing queries regarding disk full errors, free space, storage usage, du/df output, finding large files, or identifying high-consumption directories.

--- 
name: disk-usage
description: Analyze disk space usage, filesystem mounts, and storage allocation on Linux systems. Identifies large files and directories, checks partition usage, and reports inode consumption. Use when the user asks about disk full errors, free space, storage usage, du/df output, finding large files, or checking which directories consume the most space.
---

Inspect supports a wide range of model providers categorized by their access method:

• Lab APIs: OpenAI, Anthropic, Google, Grok, Mistral, DeepSeek, Moonshot AI, Perplexity.
• Cloud APIs: AWS Bedrock, AWS SageMaker, Azure AI.
• Open (Hosted): Groq, Together AI, Fireworks AI, Cloudflare, HF Inference Providers, SambaNova.
• Open (Local): Hugging Face, vLLM, Ollama, Llama-cpp-python, SGLang, TransformerLens, nnterp.

The Headless Browser Tool provides a way to test web browsing agents by running a headless Chromium browser inside a Docker container. The tool operates via a client-server model where a WebClient sends commands to a WebServer that manages a stateful browser session using Playwright.

Workflow

1. Start the Docker container: This automatically launches a web server with the headless browser.
2. Send commands: Use the web_client.py script inside the container to issue browser instructions.
3. Observe results: The client prints observations (URL, accessibility tree, etc.) to stdout.

# Inside the Docker container
$ python web_client.py [COMMAND] [args]

The computer_tool in Inspect requires a specific environment to interact with a computer via X11 and xdotool. This Docker image provides the minimum infrastructure for headless graphical interaction, observability, and remote access.

Key components include:

• Xvfb (X Virtual Framebuffer): A display server that runs in memory, allowing graphical applications to run in headless environments without a physical monitor.
• xfce4: A lightweight desktop environment used to manage the graphical session.
• x11vnc: A VNC server that enables remote access to the X11 display.
• noVNC: A web-based VNC client that allows you to view and interact with the virtual desktop directly through a web browser.

The image also includes pre-installed applications: VS Code, Firefox, XPaint, and galculator.

The inspect_ai architecture for handling arbitrarily large samples is designed to support massive datasets that exceed typical memory limits. The design targets scalability for samples containing millions of messages (e.g., 10M+ messages per sample).

As a reference workload, the system is architected to handle a 'monster' sample containing:

• ~337,351 events
• ~129,676 messages
• ~85,062 attachments
• ~10B tokens
• ~963MB uncompressed data

This architecture addresses the format, viewer, and API requirements necessary to manage such scale without performance degradation.

When defining a Subagent, you can choose between two dispatch mechanics via the fork parameter:

1. Isolated Context (fork=False): The subagent is invoked with an isolated context. The parent cannot see the subagent's conversation history or tool calls. Only the result returned via submit() is passed back to the parent.
2. Forked Context (fork=True): The agent() tool preserves the parent's message history (stripping only the trailing assistant message) and appends the subagent's instructions and task as a user message. This allows the subagent to see the context of the conversation that led to its dispatch.

When implementing tools, consider the following based on the tool's workload to ensure compatibility with Inspect's concurrency model:

• Simple/Inline Tools: For tools performing basic logic or math, standard inline Python code is sufficient.
• Network Requests: If your tool performs I/O or network calls, use async HTTP libraries like httpx to avoid blocking the event loop.
• Heavy Computation: For CPU-intensive tasks, tools should use subprocesses to prevent blocking Inspect's execution flow.