PPO for Beginners

The project is structured into several key modules:

• main.py: The primary executable. It parses arguments via arguments.py, initializes the environment and PPO model, and executes either training or testing. Training is triggered by calling the learn function.
• arguments.py: Handles command-line argument parsing.
• ppo.py: The core implementation of the PPO model. It follows the OpenAI Spinning Up pseudocode (referenced by ALG STEP # in the code).
• network.py: Provides a sample Feed Forward Neural Network used to define actor and critic networks.
• eval_policy.py: A standalone module for evaluating the policy.
• graph_code/: Contains scripts for data collection and graph generation.

This implementation is designed for continuous observation and action spaces. When selecting environments (e.g., from OpenAI Gym), ensure they use the Box space for both observations and actions.

To change hyperparameters or the environment, modify the configurations directly in main.py.

To set up the project, create a Python virtual environment, activate it, and install the required dependencies using pip.

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Hyperparameters for the PPO algorithm are defined as a dictionary within the main() function. To change the training behavior, modify the following keys in the hyperparameters dictionary:

• timesteps_per_batch: Number of timesteps collected before an update.
• max_timesteps_per_episode: Maximum length of an episode.
• gamma: Discount factor for future rewards.
• n_updates_per_iteration: Number of gradient updates per iteration.
• lr: Learning rate.
• clip: PPO clipping parameter.
• render: Boolean to enable rendering.
• render_every_i: Frequency of rendering during training.

hyperparameters = {
    'timesteps_per_batch': 2048, 
    'max_timesteps_per_episode': 200, 
    'gamma': 0.99, 
    'n_updates_per_iteration': 10,
    'lr': 3e-4, 
    'clip': 0.2,
    'render': True,
    'render_every_i': 10
}

To evaluate a previously trained model, use the --mode test flag and provide the path to the actor model using --actor_model.

python main.py --mode test --actor_model ppo_actor.pth

To continue training using existing weights, provide both the actor and critic model files using the --actor_model and --critic_model flags.

python main.py --actor_model ppo_actor.pth --critic_model ppo_critic.pth

To start the training process using default settings, run main.py without additional arguments. Note that hyperparameters and environment settings are configured directly within main.py rather than via command line flags.

python main.py

The test() function is used to evaluate a previously trained actor model in a Gymnasium environment. It demonstrates that the trained policy can exist independently of the training algorithm.

Workflow:

1. Extracts observation and action dimensions from the environment.
2. Reconstructs the policy using FeedForwardNN with the correct dimensions.
3. Loads the saved weights from the actor_model file.
4. Calls eval_policy to run the environment with the loaded policy (with render=True).

Note: An actor_model path must be provided, otherwise the function exits.

test(env, actor_model)

The train() function orchestrates the training process for a Proximal Policy Optimization (PPO) model. It initializes a PPO instance using a FeedForwardNN policy class and a provided Gymnasium environment.

Key behaviors:

• Training from scratch: If no actor or critic models are provided, training starts from scratch.
• Resuming training: If both actor_model and critic_model paths are provided, the function loads the existing state dicts into the model to continue training.
• Error handling: If only one model path is provided, the function exits to prevent accidental overwriting of partial weights.
• Timesteps: The training runs for a fixed number of total_timesteps (defaulting to 200,000,000 in the implementation).

train(env, hyperparameters, actor_model, critic_model)

The main.py file serves as the CLI entrypoint. It uses get_args() to parse command-line arguments which determine the execution mode (train or test) and the model file paths.

Modes:

• train: Executes the training loop. Requires --actor_model and --critic_model if you wish to resume training from existing weights.
• test: Executes the evaluation loop. Requires --actor_model to load the policy weights.

Environment Requirements: Custom environments must inherit from gym.Env and feature both continuous observation and action spaces.