terminal-bench

The EvoEval Adapter converts EvoEval Python programming problems into the Terminal-Bench format. It specifically targets the EvoEval_difficult subset, which consists of 100 challenging problems designed to test an agent's ability to handle complex requirements and robust algorithms.

Key features include:

• Automatic Repository Management: Supports auto-cloning the EvoEval GitHub repo or using a local installation.
• Task Generation: Extracts function signatures, converts extensive test suites, and includes reference solutions.
• Docker Integration: Runs each task in an isolated Docker container using pytest for validation.
• Tailored Evaluation: Uses dynamic imports and type normalization to ensure rigorous testing of the agent's solution.py.

The Sheets Clone API is a RESTful interface for managing spreadsheets, sheets, and cell data. It supports features like pagination, batch operations, and rich cell formatting. Currently, the API does not require authentication.

Base URL: http://localhost:8000

The AlgoTune adapter integrates 154 Python-based algorithm optimization tasks into the Terminal-Bench framework. The goal for an agent is to modify the Solver class in /app/solver.py to produce code that is both correct (identical outputs to the reference) and faster than the baseline implementation.

Key Metrics: When using this adapter, refer to the benchmark as AlgoTune-Pass and the metric as resolve rate. This is distinct from the original AlgoTune 'score' which uses continuous speedup metrics. In Terminal-Bench, a task is a 'pass' if the solution is valid and not slower than the best solver in the original AlgoTune repository.

Each task generated by the Aider Polyglot adapter follows a specific directory structure to support multi-language builds and oracle solutions:

• task.yaml: Terminal-bench task configuration.
• run-tests.sh: Language-specific script to execute tests.
• solution.sh: Script used to decrypt and apply the oracle solution.
• Dockerfile: Defines the containerized runtime environment.
• docker-compose.yaml: Orchestrates the container.
• workspace/: The student's working directory, containing:
  • solution_files: Files intended for modification.
  • build_configs: Language-specific build files (e.g., Cargo.toml, build.gradle).
• tests/: The test environment, containing:
  • .meta/: Exercism metadata and examples.
  • test_files: Unit tests.
  • helper_files: Additional files required for testing.

The Model Data Format (MDF) JSON uses the following conventions for spatial and temporal data:

• Spatial Vectors: Positions, normals, translations, and scales are expressed as [x, y, z] arrays.
• Rotations: Expressed as quaternions in the format [x, y, z, w].
• Texture Coordinates: Expressed as [u, v] arrays.
• Time: All durations and timestamps are in seconds.
• Matrices: Stored as flat arrays of 16 floats representing a row-major 4x4 matrix.
• Hierarchy: All transformations are relative to the parent joint. Root joints are identified by a parent index of -1.

Cell IDs must follow the A1 notation format.

Rules:

1. Format: Letter(s) followed by a number (e.g., A1, B2, AA34).
2. Column Limitations: Must be 1-2 uppercase letters (A-Z). The maximum column is ZZ.
3. Row Limitations: Must be a positive number. The maximum row is 9999.

Examples:

• Valid: A1, B2, Z9, AA1, ZZ999
• Invalid: 1A, A0, AAA1, A-1, 11

Docker Images

The adapter uses task-specific Docker images containing the environment and repository setup. Images follow the pattern: betty1202/sweb.eval.x86_64.<instance_id> (e.g., betty1202/sweb.eval.x86_64.scikit-learn_s_scikit-learn-11674).

Task Instructions

Instructions are derived from the original SWE-Perf dataset on Hugging Face. For OpenHands users, a Jinja2 template sweperf-openhands-instruction.j2 is provided to match the original authors' inference logic.

Evaluation Metrics

Currently, the adapter supports two of the three original SWE-Perf metrics:

1. Apply (integrated into correctness)
2. Correctness

Note: The Performance improvement metric is planned for a future update.

The adapter uses a copy + append approach. It preserves the original CyBench structure while adding Terminal-Bench compatibility files.

Key Generated Files:

• task.yaml: Terminal-Bench task configuration (includes instruction, category, difficulty, and max_agent_timeout_sec: 3000.0).
• Dockerfile.agent: An agent container based on cybench/cybench:latest (a 17GB Kali Linux image).
• docker-compose.yaml: Orchestrates the agent and necessary services.
• solution.sh: A rewritten oracle solution wrapper compatible with the Terminal-Bench container framework.

Agent Environment: The agent runs in a container with tools like pwntools, ROPgadget, z3, and angr. Python 3.9 is provided with a virtual environment at /venv.

The bot includes a FastAPI server that listens for GitHub webhooks on the /webhook endpoint.

When a Pull Request is merged, the following workflow occurs:

1. Signature Verification: The bot verifies the x-hub-signature-256 header using the GITHUB_PAT to ensure the request came from GitHub.
2. File Detection: It scans the files changed in the PR for any file named task.yaml.
3. Content Fetching: For every task.yaml found, the bot fetches the full content from the GitHub API.
4. Task Parsing: The YAML content is parsed to extract instruction, author_email, difficulty, and category.
5. Discord Notification: An embed message is sent to the configured DISCORD_CHANNEL_ID containing the task details and PR metadata.