imitation

The imitation library provides modular PyTorch implementations of several imitation and reward learning algorithms designed to work with Stable Baselines 3 (SB3) policies.

Supported algorithms include:

• Behavioral Cloning (BC)
• DAgger (with synthetic examples)
• Density-based reward modeling
• Maximum Causal Entropy Inverse Reinforcement Learning (MCE IRL)
• Adversarial Inverse Reinforcement Learning (AIRL)
• Generative Adversarial Imitation Learning (GAIL)
• Deep RL from Human Preferences

Key features:

• SB3 Compatibility: Built on and compatible with Stable Baselines 3.
• Modular Implementations: GAIL and AIRL allow for customizable reward and discriminator networks.
• Demonstration Management: Includes scripts and data structures for loading, storing, and saving expert demonstrations (rollouts).

The imitation library provides clean implementations of several imitation and reward learning algorithms. Algorithms are categorized by whether they support discrete or continuous action/state spaces.

The imitation library is organized into several functional subpackages located in src/imitation:

• algorithms: Core implementation of imitation and reward learning algorithms.
• data: Modules for collecting, storing, and manipulating transitions and trajectories.
• envs: Test environments.
• policies: Modules defining policies and manipulation methods (e.g., serialization).
• regularization: Neural network weight regularization techniques.
• rewards: Modules for building, serializing, and preprocessing neural network-based reward functions.
• scripts: Command-line scripts for running experiments via Sacred.
• util: Utility functions for logging, configurations, and neural network building.

Density-based reward modeling is an Inverse Reinforcement Learning (IRL) technique that assigns higher rewards to states or state-action pairs that occur more frequently in expert demonstrations. It uses kernel density estimation to model the distribution of expert behavior and assigns rewards based on the estimated log-likelihood of the agent's actions under that distribution.

Key Characteristics:

• Goal: Incentivize the agent to take actions that resemble the expert's actions in similar states.
• Limitations:
  • Assumes expert demonstrations are perfectly representative.
  • Does not provide an interpretable reward function.
  • Kernel density estimation scales poorly to high-dimensional state-action spaces.

Reward networks in imitation support two distinct modes of operation:

1. forward(state, action, next_state, done): Produces a reward that is differentiable. This is used during the training of the reward network (e.g., in GAIL or AIRL).
2. predict_processed(...): Produces a reward used for training policies. This method applies post-processing (like normalization or shaping) that is not needed during reward network training. It is generally not used for gradient-based optimization of the reward network itself.

Training scripts use sacred to manage experiments. Outputs are organized into two main areas:

1. Algorithm/Environment folders: Grouped by algorithm and environment name. These contain log files, model checkpoints, and a symlink to the sacred run folder.
2. sacred folder: Contains all runs grouped by training script, with each run having a unique ID folder.

Each run folder contains:

• config.json: Hyperparameters used (includes environment.gym_id).
• run.json: Run metadata, including result.imit_stats.monitor_return_mean (the score) and result.expert_stats.monitor_return_mean (the expert score).
• cout.txt: Standard output of the run.

The imitation.scripts package provides a CLI built on the Sacred library to run imitation learning algorithms and utility tasks.

Key concepts:

• Experiments: The core unit of execution.
• Ingredients: Reusable components (like environments, expert policies, or reward functions) that have their own configuration namespaces.
• Named Configurations: Pre-defined sets of configuration values (e.g., cartpole or seals_mountain_car) that can be passed to a script to set up an environment quickly.
• Configuration Namespaces: Each ingredient (e.g., expert, demonstrations, environment) has its own namespace for setting parameters.

To explore all available configuration values for a specific script, use the print_config command.

python -m imitation.scripts.<script> print_config

In imitation, trajectories (also called rollouts or episodes) are sequences of observations and actions generated by an agent interacting with an environment. They are represented by the Trajectory dataclass.

If your data includes rewards, use the TrajectoryWithRew subclass. Some algorithms may prefer individual Transitions instead of full trajectories, which can be obtained by flattening trajectories using imitation.data.rollout.flatten_trajectories.

@dataclasses.dataclass(frozen=True)
class Trajectory:
    obs: np.ndarray
    """Observations, shape (trajectory_len + 1, ) + observation_shape."""

    acts: np.ndarray
    """Actions, shape (trajectory_len, ) + action_shape."""

    infos: Optional[np.ndarray]
        """An array of info dicts, shape (trajectory_len, )."""

    terminal: bool
    """Does this trajectory (fragment) end in a terminal state?"""