DOCSEARCHbyAlgolia
Submit Docs

AgentEvolver Documentation

repository·main·Indexed 23 days ago

https://github.com/modelscope/agentevolver

An end-to-end, self-evolving training framework for AI agents featuring self-questioning, self-navigating, and self-attributing mechanisms. It supports Hybrid Experience Training (HET) to fuse historical trajectories with online sampling. The framework includes a Game Arena for evaluating agents in games like Avalon and Diplomacy, a web interface for observer and participate modes, and tools for generating training tasks in Parquet format and running PPO-based training.

Tokens
33.5K
Snippets
66
Records
159
Agent score
81%

What's inside AgentEvolver

  1. Env Service API Overview

    main

    The Env service provides interfaces for environment configuration queries, instance management, and step execution. It is designed for applications that need to interact with dynamic environments.

    Base Information:

    • Default Service Address: http://localhost:8000
    • Method: All interfaces use POST.
    • Data Format: JSON.
    • Timeout: Typically between 150-350 seconds.

    Error Handling:

    • All interfaces implement a retry mechanism (default: 3 retries).
    • On retry failure, a fallback default value is returned.
    • Error logs are written to /mnt/data/eric.czq/rl_log/error.out by default. You can customize this by setting the CLIENT_LOG_PATH environment variable.

  2. Understand the AgentFlow directory structure

    main

    The AgentFlow source code is organized into functional modules. Use this overview to locate specific logic for agents, orchestration, data, environments, and execution stages:

    • agents/: Contains agent logic (llm_agent.py) and trajectory evaluation metrics (trajectory_evaluator.py).
    • core/: Orchestration logic. pipeline.py coordinates the execution stages, and api_client.py handles model provider interactions.
    • data/: Data models (models.py) and I/O helpers (storage.py).
    • environment/: Environment abstraction and the env_factory.py used to instantiate environments.
    • envs/: Managers for EnvService-backed environments (e.g., appworld_manager.py, webshop_manager.py) that use HTTP-based integration.
    • stages/: Implementations of the three core stages: stage1_triplet_generation.py, stage2_task_abstraction.py, and stage3_trajectory_generation.py.
    • prompts/: Prompt templates and builders for various agent tasks.
    • utils/: Shared utilities like logger.py.

  3. Overview of AgentEvolver Game Arena

    main

    The AgentEvolver Game Arena is an extension designed for multi-agent social game environments (e.g., Avalon, Diplomacy). It provides a unified arena for interaction, evaluation, and training in long-horizon social reasoning tasks.

    Key features:

    • Web-based interaction: Real-time observation of agent reasoning/communication or human participation.
    • Scalable evaluation: Support for large-scale self-play or mixed-model tournaments with leaderboards.
    • End-to-end training: Direct LLM agent training within social games using RL-based methods like GRPO.

  4. What is the Advantage Processor and ADCA-GRPO?

    main

    The Advantage Processor is a core component of AgentEvolver that implements a self-attributing mechanism to solve the credit assignment problem in long-horizon tasks. It uses LLM-based causal reasoning to decompose learning signals into process quality (logical correctness of actions) and outcome effectiveness (final success).

    The primary implementation is ADCA-GRPO. It uses an LLM to assign an attribution signal (GOOD/BAD) to each step in a trajectory. This signal is fused with the traditional outcome signal to create a fine-grained advantage, which improves sample efficiency and reduces training steps compared to traditional GRPO.

  5. Understand Experience Pool Usage Modes

    main

    The experience pool system supports several modes depending on whether you want to use existing data, initialize new data, or update the pool during training.

    Configuration Matrix for Usage Modes

    Modeupdated_freqinit_exp_onlyinit_exp_before_trainingenable_summarizerenable_context_generatorworkspace_id
    No Pool0FalseFalseFalseFalse-
    Init Only0TrueTrueTrueFalseNew ID
    Init + Train0FalseTrueFalseTrueNew ID
    Init + Train + Updatek != 0FalseTrueTrueTrueNew ID
    Exist + Train0FalseFalseFalseTrueExist ID
    Exist + Train + Updatek != 0FalseFalseTrueTrueExist ID

    Key Notes:

    • updated_freq: Set to a non-zero value (e.g., 100) to enable periodic updates. 0 disables updates.
    • workspace_id: Use a new ID to create a fresh pool. Use an existing ID to reuse a pool. When using vector_store.default.backend=local, pools are saved at ReMe/local_vector_store/{workspace_id}.jsonl.
    • Recommendation: Start with Init + Train mode for most use cases.

  6. Manage persistent background services and logs

    main

    Services launched via flags (e.g., --with-appworld) are managed as detached background processes. This provides:

    • Single-instance guarantee: Services won't re-launch if already running.
    • Detached execution: Services continue running even if the terminal session ends.
    • Structured logging: Logs are stored in logs/companion/ using the pattern <tag>.<hash>.<hostname>.log.

    To view service logs, locate the log file path printed to the console during launch and use tail:

    tail -f logs/companion/appworld_env_service.*.log

    Note: To restart a service that is already running, you must either kill the process group or remove the corresponding .pgid file in the logs directory.

  8. Understand the AgentFlow data I/O layout

    main

    AgentFlow persists data in a specific directory structure under the configured data_dir. Understanding this layout is essential for inspecting outputs or resuming work:

    • Stage 1 (Triplets): data/triplets/*.jsonl
    • Stage 2 (Tasks): data/tasks/*.jsonl
    • Stage 3 (Trajectories): data/trajectories/trajectory_*.json and data/trajectories/failed_tasks/*.json
    • Query Rewrite: data/rewrites/trajectories/trajectory_*_rw*.json

  9. Select Training Method Configuration

    main

    Depending on your goal, you can configure the training method using different combinations of rollout and sampling modes. Use the following mapping to set up your experiment:

    Methodval_rollout_expmodetrain_rollout_expmodetrain_sample_expmoderollout_expratiotrain_sample_keepratio
    baseline"woexp""woexp"-0.0-
    EC"mixed""mixed""keep"(0, 1)1.0
    EI"mixed""mixed""discard"(0, 1)0.0
    HET"mixed""mixed""hybrid"(0, 1)(0, 1)

  10. How the ADCA-GRPO implementation workflow works

    main

    The ADCA-GRPO advantage calculation follows a three-stage pipeline:

    1. Stage 1: Semantic Evaluation: An LLM acts as a "step-by-step evaluator" using evaluate_step_flags_parallel_sync (from semantic_attribution.py) to generate GOOD/BAD labels for each step in a trajectory.
    2. Stage 2: Signal Fusion: The pipeline balances process vs. outcome using compute_prm_grpo_advantages (from adca_grpo.py). In the recommended "decouple" scheme, it performs independent Z-Score normalization on both step-level PRM rewards and trajectory-level outcome scores before fusing them using a weighting coefficient alpha.
    3. Stage 3: Advantage Computation: Step-level rewards are converted into token-level advantages. suffix_sum_on_steps calculates the cumulative sum of future rewards for each step, and broadcast_step_adv_to_tokens maps these values to individual tokens using step_ids.

  11. How AgentEvolver's self-evolving mechanisms work

    main

    AgentEvolver uses three core mechanisms to evolve agent capabilities from the environment to the policy:

    1. Automatic Task Generation (Self-Questioning): The system explores the environment to autonomously create diverse tasks, reducing the need for manual dataset construction.
    2. Experience-guided Exploration (Self-Navigating): It summarizes and reuses experiences across different tasks to guide higher-quality rollouts and improve exploration efficiency.
    3. Attribution-based Credit Assignment (Self-Attributing): It processes long trajectories to identify the causal contribution of intermediate steps, allowing for fine-grained policy optimization.

  12. Understand the AgentEvolver configuration hierarchy

    main

    AgentEvolver uses a hierarchical configuration system powered by Hydra. Settings are managed through a three-layered inheritance model where each layer can override the previous one:

    1. Base Layer (external/config_fallback/ppo_trainer.yaml): Provides system-wide defaults for core training parameters (derived from veRL).
    2. Framework Layer (config/agentevolver.yaml): Extends the base layer with AgentEvolver-specific settings, such as algorithm overrides, self-questioning/navigating/attributing configurations, and experiment definitions.
    3. Application Layer: The final layer of customization provided by specific example files (like examples/basic.yaml or examples/overall.yaml) or execution scripts.