ProgramBench

Inference should be performed using Docker images with the task_cleanroom_v6 tag from the ProgramBench DockerHub repository.

Important Requirements:

• Architecture: Images are built for linux/amd64 (x86_64) only. They will not run natively on macOS or Windows; use a Linux x86_64 machine.
• Network: The agent MUST NOT have access to the internet during inference.

Task Image Naming Convention: To solve a task like ffmpeg__ffmpeg.360a402, the corresponding image name replaces the double underscore __ with _1776_. Example: ffmpeg_1776_ffmpeg.360a402:task_cleanroom_v6.

To evaluate an agent's run, the agent's produced codebase must be extracted into a .tar.gz archive and organized into a specific directory structure. Each task directory should be named using the pattern author__task_id (e.g., abishekvashok__cmatrix.5c082c6), containing a submission.tar.gz file.

Required Directory Structure:

my-amazing-agent-run
├── abishekvashok__cmatrix.5c082c6
│   └── submission.tar.gz
├── agourlay__zip-password-finder.704700d
│   └── submission.tar.gz
└── ...

You can use ProgramBench without a permanent installation using uvx, or install it directly into your project or environment using pip or uv pip.

For development, clone the repository and use uv sync to install editable and development dependencies.

# Run without installing
uvx programbench --help

# Or install into a project
uv pip install programbench

# Or with pip
pip install programbench

# For development
git clone https://github.com/facebookresearch/programbench.git
cd programbench
uv sync

ProgramBench provides a command-line interface for evaluating LM-based SWE-agents. The main entrypoint is the programbench command, which includes subcommands for evaluation (eval), information retrieval (info), and specialized tasks via blob and submit subcommands.

To see all available commands and help, run:

programbench --help

The ProgramBench leaderboard supports optional statistics (e.g., cost or model calls). These cannot be entered manually; they must be recoverable from the run.

To include extra stats:

1. Write a script that reads each traj.json file from your trajectories.
2. The script must emit a flat JSON map in the format {instance_id: value}.
3. Save this output to _stats/<name>.json.
4. Include the script in your submission (e.g., under _scripts/) to ensure reproducibility.

To reproduce a ProgramBench submission, follow these steps in order. Note that per the evaluation protocol, inference must be run without internet access.

1. Install the agent and necessary dependencies.
2. Run inference for each task.
3. Run the evaluation command: programbench eval <run-dir>.
4. Submit the package: programbench submit package <run-dir> --upload-to <org>.

# 1. install the agent / dependencies
# 2. run inference per task (no internet, per the eval protocol)
# 3. programbench eval <run-dir>
# 4. programbench submit package <run-dir> --upload-to <org>

To run the mini-swe-agent baseline for ProgramBench, you can use uvx to run the tool directly or install mini-swe-agent via pip and then use the mini-extra command. Refer to the mini-swe-agent baseline docs for specific implementation details.

# Using uvx
uvx --from mini-swe-agent mini-extra programbench --help

# Using pip
pip install mini-swe-agent && mini-extra programbench --help

When a command executed via execute() hits a host-side timeout, Python sends a SIGKILL to the docker exec process. However, the Docker daemon does not propagate this signal to the process running inside the container, which can lead to orphaned processes competing with subsequent commands.

ContainerEnvironment.execute() handles this automatically by attempting to sweep the container: it executes kill -KILL -1 inside the container to kill all processes except PID-1 (the long-lived sleep process). This ensures a clean environment for the next execution.