SWE-smith Documentation

SWE-smith provides a pipeline for creating high-quality training data for software engineering agents. The core capabilities include:

• Build Environments: Create reproducible Docker images for any repository, capturing dependencies and validating with automated testing.
• Create Instances: Generate task instances using LM prompts, procedural modifications, PR mirroring, or combined techniques.
• Validate & Evaluate: Use built-in harnesses to filter candidates that break tests and verify proposed solutions.
• Generate Issue Text: Use LM generation or other methods to add natural language problem statements to task instances.
• Rate Difficulty: Classify tasks as easy, medium, or hard using a fine-tuned Qwen 2.5 Coder model.
• Train SWE-agents: Execute a complete RSFT (Rejection Sampling Fine-Tuning) pipeline: generate trajectories, filter successful solutions, fine-tune models, and evaluate on SWE-bench.

Criteria are used to filter functions so that the LLM only attempts to generate bugs in code where that specific bug is actually possible (e.g., only attempting 'off-by-one' bugs on functions containing loops or list indexing).

To create a new criterion, add a function to swesmith/bug_gen/llm/criteria.py following this pattern:

1. The function name should follow the pattern filter_<criteria_name>.
2. It must accept a CodeEntity object.
3. The CodeEntity object provides:
   • src_code: The raw string representation of the function.
   • src_node: An AST node representation of the function.
4. The function must return True if the function satisfies the criteria, or False otherwise.

def filter_<criteria>(code_entity: CodeEntity) -> bool:
    """
    `code_entity` is an object representing a function. It includes several
    pieces of information, most notably:
        * `src_code`: The raw string repr. of a function
        * `src_node`: An AST node representation of a function.
    """
    node = code_entity.src_node
    # Logic for checking whether a function has a property
    if satisfies_criteria:
        return True
    return False

You can use Language Models (LMs) to introduce bugs by either modifying an existing programmatic entity (class or function) or rewriting it from scratch.

Artifacts Produced: Under logs/bug_gen/<repo>, artifacts are organized by file path and then by individual entity.

• For LM Modify: bug__lm_modify__<hash>.diff and metadata__lm_modify__<hash>.json.
• For LM Rewrite: bug__lm_rewrite__<hash>.diff and metadata__lm_rewrite__<hash>.json.

The <hash> is computed over the contents of the .diff file.

# To prompt an LM to modify a function to introduce a bug:
python -m swesmith.bug_gen.llm.modify $repo \
  --n_bugs 1 \
  --model openai/gpt-4o \
  --config_file configs/bug_gen/lm_modify.yml

# To prompt an LM to rewrite a function from scratch:
python -m swesmith.bug_gen.llm.rewrite $repo \
  --model anthropic/claude-3-7-sonnet-20250219 \
  --config_file configs/bug_gen/lm_rewrite.yml \
  --n_workers 1

Once the model is hosted on Modal, you can run inference using the sweagent run-batch command from your local SWE-agent repository.

Key configuration requirements:

• --agent.model.api_base: Set this to the Modal endpoint URL (e.g., <MODAL_LINK>/v1).
• --agent.model.api_key: Set this to swesmith (as configured in serve_sglang.py).
• --agent.model.name: The name of the served model.
• --instances.type: Set to swe_bench.
• --instances.dataset_name: The name of the dataset (e.g., jyang20/swebv-mini).
• --instances.split: The dataset split (e.g., test).
• --config: Path to the SWE-agent configuration file.

sweagent run-batch \
    --agent.model.api_base <REPLACE WITH MODAL LINK>/v1 \
    --agent.model.api_key swesmith \
    --agent.model.name gpt-4o \
    --instances.type swe_bench \
    --instances.dataset_name jyang20/swebv-mini \
    --instances.split test \
    --config config/anthropic_no_fcalls.yaml