DOCSEARCHbyAlgolia
Submit Docs

Unity Machine Learning Agents (ML-Agents) Toolkit

repository·develop·Indexed 12 days ago

https://github.com/unity-technologies/ml-agents

An open-source framework for training intelligent agents in Unity environments using reinforcement learning (PPO, SAC) and imitation learning (BC, GAIL). It includes a Unity package (v4.1.0) and Python packages (mlagents, mlagents_envs v1.1.0) to enable PyTorch-based training for games, robotics, and film simulations across 2D, 3D, and VR/AR environments.

Tokens
79.3K
Snippets
185
Records
374
Agent score
94%

What's inside ML-Agents

  1. Overview of Python APIs in ML-Agents

    develop

    ML-Agents provides several Python APIs to control and interact with Unity environments. The choice of API depends on your reinforcement learning (RL) workflow requirements:

    • Python Gym API: Provides an OpenAI Gym-compatible interface, ideal for standard RL workflows and using existing Gym-based algorithms.
    • Python PettingZoo API: Provides a multi-agent environment interface compatible with PettingZoo, suitable for multi-agent reinforcement learning (MARL).
    • Python Low-Level API: Offers direct low-level access for custom training loops and advanced research use cases where higher-level abstractions are too restrictive.
    • On/Off Policy Trainer: Specialized documentation for implementing specific on-policy or off-policy training methods.
    • Python Optimizer: Documentation regarding the optimizers used within the training algorithms.

  2. Overview of Unity ML-Agents Toolkit

    develop

    The Unity Machine Learning Agents (ML-Agents) toolkit is an open-source project that provides an environment for training intelligent agents for use in games and various simulations. It allows developers to train agents in 2D, 3D, and VR/AR environments using modern PyTorch-based algorithms via a simple Python API.

    Key capabilities include:

    • Reinforcement Learning: Using algorithms like Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC).
    • Imitation Learning: Built-in support for Behavioral Cloning and Generative Adversarial Imitation Learning.
    • Advanced Training Techniques: Self-play for adversarial scenarios, curriculum learning for complex tasks, and environment randomization for robustness.
    • Scalability: Support for training multiple Unity environment instances simultaneously.
    • Deployment: Uses the Unity Inference Engine for native cross-platform support.
    • Integration: Flexible Unity SDK that can be integrated into games or custom Unity scenes.

  4. Overview of the Unity ML-Agents Toolkit

    develop

    The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that allows games and simulations to serve as environments for training intelligent agents. It provides PyTorch-based implementations of state-of-the-art reinforcement learning and imitation learning algorithms.

    Key capabilities include:

    • Training Scenarios: Support for single-agent, multi-agent cooperative, and multi-agent competitive scenarios.
    • Algorithms: Includes PPO, SAC, MA-POCA, self-play, and Imitation Learning (BC and GAIL).
    • Environment Integration: A flexible Unity SDK for custom scenes, support for curriculum learning, environment randomization, and concurrent environment instances.
    • Python Interoperability: Ability to control Unity environments from Python, wrap them as gym environments, or wrap them as PettingZoo environments.
    • Deployment: Uses an Inference Engine for native cross-platform support.

  6. Use the Unity ML-Agents Custom Trainers Plugin

    develop

    The Custom Trainers Plugin allows you to extend the mlagents-learn CLI with new reinforcement learning algorithms by implementing the High-level trainer API. This system supports both on-policy and off-policy algorithms and allows you to define custom hyper-parameters in configuration files.

    To implement a custom trainer, you must create a new Python package that includes a setup.py file to register the new RL algorithm into the ML-Agents ecosystem. This registration enables the mlagents-learn command to recognize your custom trainer and its specific configuration schema.

  7. Legacy Cloud & Deployment Options

    develop

    The following methods for deploying ML-Agents training in cloud environments are deprecated and no longer actively maintained or recommended. Use them only if necessary for legacy workflows:

    • Docker: Containerized deployment using Docker.
    • Amazon Web Services (AWS): Training on AWS cloud infrastructure.
    • Microsoft Azure: Training on Azure cloud services.

  9. Understand the Unity ML-Agents Python Low Level API

    develop

    The Python Low Level API (mlagents_envs) allows you to interact directly with a Unity Environment. It provides the mechanism to control the simulation loop, making it suitable for developing and evaluating custom reinforcement learning or imitation learning algorithms outside of the standard mlagents-learn entry point.

    To use this API, the Agent in your Unity simulation must be configured with Behavior Parameters where the Behavior Type is set to Default and a specific Behavior Name is assigned.

  11. What is Deep Learning in the context of ML-Agents

    develop

    Deep learning is a family of algorithms used to learn complex functions from large amounts of training data. In ML-Agents, deep learning is particularly effective for reinforcement learning tasks because Unity can act as a simulator to generate the massive datasets (hundreds of thousands of simulations) required to learn policies for complex environments. A complex environment is defined as one where the number of observations perceived by the agent and the number of possible actions are large.

    Many of the algorithms provided in the ML-Agents toolkit are built on top of the open-source PyTorch library.

  12. Explore different training scenarios

    develop

    ML-Agents can be configured for various multi-agent and single-agent training scenarios:

    • Single-Agent: A single agent with its own reward signal.
    • Simultaneous Single-Agent: Multiple independent agents with independent reward signals but sharing the same Behavior Parameters. This parallelizes training for similar characters.
    • Adversarial Self-Play: Two agents with inverse reward signals (e.g., one wins, the other loses), allowing an agent to train against itself.
    • Cooperative Multi-Agent: Multiple agents interacting with a shared reward signal, often working together to solve a task.
    • Competitive Multi-Agent: Multiple agents with inverse reward signals competing for resources or victory.
    • Ecosystem: Multiple agents with independent reward signals interacting in a shared environment (e.g., a biological simulation).