repo2rlenv

The cve_patches pipeline is an experimental tool for the Python ecosystem that maps OSV (Open Source Vulnerabilities) records to fixing commits in a target repository. It creates reusable tasks for the Repo2RLEnv format by replaying the pre-fix state in a sandbox and using the upstream security patch as an oracle.

Key Characteristics:

• Status: Experimental (Python ecosystem).
• Requirements: Requires a sandbox for generation and an LLM (for bootstrapping and PoC synthesis).
• Reward Kinds: Emits test_execution and diff_similarity.
• Yield: Typically low (~5–25%) because it requires a verifiable 'fail-to-pass' oracle (either a shipped regression test or a synthesized PoC).

The code_instruct pipeline is a specialized data generation pipeline that creates coding tasks grounded in a specific target repository. Unlike standard OSS-Instruct methods that produce text-only problem/solution pairs, code_instruct ensures every task is:

1. Repo-anchored: The task is solvable within the target repository's specific environment.
2. Verified by execution: Every task includes an executable pytest verifier.
3. Validated by an oracle: The LLM-generated solution (the oracle) must pass the generated test within the repository's Docker sandbox.

This pipeline is currently shipped (v0.8.6 hardened) and supports Python only. It requires both an LLM and a sandbox environment during the generation phase.

Bootstrapping is the process of creating a Docker image where a specific GitHub repository builds cleanly and its test suite runs. Because different repositories have diverse setup requirements (e.g., Python with uv/poetry, JS with pnpm/npm, Rust with specific toolchains), repo2rlenv uses an LLM agent to automate this.

The agent reads the repo, executes shell commands in a sandbox, and iterates until build and tests succeed. The resulting Docker image is cached locally and reused for all subsequent tasks generated from that repo, ensuring you only pay the LLM cost once per (repo, commit).

The repo2rlenv CLI is a thin wrapper around the Python API. Most CLI commands have a direct Python equivalent:

The reward for a test_synthesis task is designed to encourage tests that are both accurate (pass on fixed, fail on broken) and useful (increase code coverage).

Reward Formula:

reward = (1 - coverage_weight) * b_pass * b_fail + coverage_weight * cov

Components:

• b_pass: The fraction of the agent's emitted tests that pass on the fixed variant.
• b_fail: The fraction of the agent's emitted tests that fail on the broken variant.
• cov: The coverage delta, calculated as min(1.0, added_lines_covered / target_new_lines). This measures how many new lines in the touched function were covered by the agent's tests compared to the baseline coverage of the existing test suite.

Note that the product b_pass * b_fail ensures that tests which fail to discriminate (e.g., tests that pass on both or fail on both) result in a zero score for that component.

Pipelines that generate per-PR tasks default to re-installing dependencies after a git checkout.

• Additive dependencies: Automatically captured by the pipeline; no re-bootstrap required.
• Build system replacements: If a PR changes the core build system, you must trigger a fresh bootstrap using the --force-bootstrap flag.

Because CVE tasks have high contamination risk (public advisories, PyPI release notes, etc.), the pipeline implements multiple layers of defense:

• Network Isolation: An egress guard (_env_guard.py) blackholes PyPI to prevent the environment from downloading the already-patched version of a package.
• Git-history Scrubbing: Mandatory scrubbing of git history during emission.
• Leak-stripped Instructions: The pipeline strips CVE/GHSA IDs, PR/commit URLs, and phrases like "fixed in vX.Y" or "Closes #N" from the LLM prompts. The prompt only contains the symptom.
• Agent Constraints: The PoC agent is both prompted and regex-checked to ensure it does not include the fix commit SHA or CVE ID in the synthesized test.