Trae Agent

Trae Agent automatically records detailed execution trajectories (LLM interactions, tool usage, and metadata) for debugging and analysis.

• Automatic Storage: Trajectories are saved to trajectories/trajectory_YYYYMMDD_HHMMSS.json.
• Custom Storage: Use the --trajectory-file <filename> flag to specify a custom path.

Trajectory files are useful for studying agent behavior and conducting ablation studies.

trae-cli run "Optimize database queries" --trajectory-file optimization_debug.json

When configuring Trae Agent, settings are applied according to the following priority (from highest to lowest):

1. Command-line arguments
2. Configuration file values (trae_config.json)
3. Environment variables
4. Default values

When running selector.py, the result_path will contain the following directory structure:

• log/: Contains JSON files storing the LLM interaction history (e.g., group_0/astropy__astropy-14369_voting_0_trail_1.json).
• output/: Stores raw stdout and stderr from the agent's execution.
• patch/: Contains the actual selected .patch files.
• statistics/: Contains JSON files indicating whether the selected patch was correct or not.

├── log
│   └── group_0
│       └── astropy__astropy-14369_voting_0_trail_1.json
├── output
│   └── group_0
│       └── astropy__astropy-14369.log
├── patch
│   └── group_0
│       └── astropy__astropy-14369_1.patch
└── statistics
    └── group_0
        └── astropy__astropy-14369.json

Trajectory recording captures detailed logs of LLM interactions and agent execution steps for debugging, auditing, and analysis. It records raw LLM messages (inputs/responses), token usage, tool calls, agent state transitions, tool results, reflections, and errors.

Recording is handled by the TrajectoryRecorder class and is automatically integrated into supported LLM clients (Anthropic, OpenAI, Google Gemini, Azure, Doubao, Ollama, and OpenRouter) and the base Agent class. When a recorder is attached to an agent or client, data is captured and saved to JSON files.

# Conceptual flow of recording
if self.trajectory_recorder:
    self.trajectory_recorder.record_llm_interaction(...)
    self.trajectory_recorder.record_agent_step(...)

Evaluation results are stored in a directory structured as results/{benchmark}_{dataset}_{run_id}/.

Key Output Files:

• predictions.json: All generated patches intended for evaluation.
• results.json: The final evaluation results (pass/fail status).

Instance-specific data (per {instance_id}/ folder):

• problem_statement.txt: The original GitHub issue description.
• {instance_id}.patch: The patch generated by Trae Agent.
• {instance_id}.json: The trajectory file containing agent steps.

Workspace Artifacts (trae-workspace/):

• trae_config.yaml: The configuration used.
• trae-agent.tar, uv.tar, uv_shared.tar: Build artifacts used for efficient container reuse.

To execute agent tools within Docker containers, you must manually provide a specific Python 3.12 environment.

1. Download the Python 3.12 package from this Google Drive link.
2. Unzip the downloaded package into the following directory: evaluation/patch_selection/trae_selector/tools/py312.

⚠️ DEPRECATED

This JSON configuration format is deprecated. For new installations, use the YAML configuration format instead.

To set up the legacy JSON configuration:

1. Copy the example configuration file:

   cp trae_config.json.example trae_config.json

2. Edit trae_config.json and replace placeholder values with your actual credentials, such as:

   • your_openai_api_key
   • your_anthropic_api_key
   • your_google_api_key
   • your_azure_base_url

Note: trae_config.json is ignored by git to prevent accidental credential leaks.

To evaluate Trae Agent, you must first install the necessary dependencies and set up the benchmark harness.

Prerequisites:

• Docker (required for containerized environments)
• Python 3.12+
• Git
• Sufficient disk space (several GBs per instance)
• API Keys (OpenAI or Anthropic)

Installation Steps:

1. Install evaluation dependencies using uv:

   uv sync --extra evaluation
   cd evaluation

2. Run the setup.sh script to clone and configure the benchmark harness. You must specify which benchmark you are using:
   • swe_bench
   • swe_bench_live
   • multi_swe_bench

chmod +x setup.sh
./setup.sh [swe_bench|swe_bench_live|multi_swe_bench]

uv sync --extra evaluation
cd evaluation
chmod +x setup.sh
./setup.sh swe_bench

Trae Agent requires uv and an API key from a supported provider (OpenAI, Anthropic, Google Gemini, etc.).

To install from the repository:

1. Clone the repository.
2. Use uv sync to install dependencies with all extras.
3. Activate the virtual environment.

Requirements:

• uv (package manager)
• API key for your chosen LLM provider

git clone https://github.com/bytedance/trae-agent.git
cd trae-agent
uv sync --all-extras
source .venv/bin/activate

After running the selection process, results are organized into log, output, patch, and statistics directories. You can use the analysis.py script to visualize these results. This script can be run even while the selection process is still active to view intermediate progress.

python3 analysis.py --output_path "path/to/save/results"

To migrate from the deprecated JSON format to the recommended YAML format:

1. Create a new YAML configuration file:

   cp trae_config.yaml.example trae_config.yaml

2. Transfer settings: Move your existing settings from trae_config.json to trae_config.yaml following the new YAML structure.

3. Remove the old JSON file (optional):

   rm trae_config.json

Refer to the main README.md for detailed YAML configuration instructions.

Use run_evaluation.py to execute benchmarks. The script supports different modes and datasets.

Evaluation Modes:

• e2e (default): End-to-end; both generates and evaluates patches.
• expr: Expression only; generates patches without evaluation.
• eval: Evaluation only; evaluates existing patches using a provided --benchmark-harness-path.

Common Usage Examples:

Run on all instances of a specific dataset:

python run_evaluation.py --dataset SWE-bench_Verified --working-dir ./trae-workspace

Run on specific instances:

python run_evaluation.py --instance_ids django__django-12345 scikit-learn__scikit-learn-67890

Run with custom configuration and run ID:

python run_evaluation.py --config-file trae_config.yaml --run-id experiment-1