DOCSEARCHbyAlgolia
Submit Docs

TMRL Documentation

repository·master·Indexed 20 days ago

https://github.com/trackmania-rl/tmrl

A distributed Reinforcement Learning (RL) framework for real-time applications like robotics and video games. It features a specialized pipeline for autonomous driving in TrackMania 2020, including Gymnasium integration via rtgym, support for SAC and REDQ-SAC algorithms, and a client-server architecture for remote training on HPC clusters.

Tokens
18K
Snippets
46
Records
73
Agent score
70%

What's inside TMRL

  1. Overview of TMRL

    master

    TMRL is a distributed Reinforcement Learning (RL) framework designed for real-time applications such as robotics, video games, and high-frequency control. It uses a single-server / multiple-clients architecture, allowing users to collect samples locally from many workers and train remotely on High Performance Computing (HPC) clusters.

    Key features include:

    • Ready-to-use TrackMania 2020 pipeline: Includes state-of-the-art algorithms like SAC and REDQ, analog control via virtual gamepads, and support for both LIDAR and raw screenshot (CNN-based) observations.
    • Gymnasium Integration: Provides a Gymnasium environment for TrackMania 2020 based on rtgym.
    • Extensibility: Designed as a Python library for implementing custom, ad-hoc RL pipelines for real-world industrial applications.

  2. Overview of the TMRL framework

    master

    TMRL is a complete Python framework designed for implementing deep reinforcement learning (DRL) pipelines in real-world applications, such as robotics or video games. It provides a structured way to build training pipelines, including a ready-to-use example for the TrackMania 2020 videogame.

    For developers looking to build custom training pipelines, the framework is centered around three core classes located in the tmrl.networking module:

    • Server: Manages the central coordination of the training process.
    • RolloutWorker: Handles the execution of environments and data collection.
    • Trainer: Manages the learning process and model updates.

  3. Overview of the TMRL package modules

    master

    The tmrl package is organized into several core modules that handle different aspects of the reinforcement learning pipeline for Trackmania:

    • tmrl.networking: Handles communication and network protocols.
    • tmrl.actor: Manages the agent's actor components.
    • tmrl.training: Contains logic for online reinforcement learning training.
    • tmrl.training_offline: Contains logic for offline reinforcement learning training.
    • tmrl.memory: Manages experience replay buffers and memory storage.
    • tmrl.envs: Provides the environment interfaces for interacting with Trackmania.

  4. Understand the available action spaces in tmrl

    master

    The tmrl framework supports two types of control mechanisms for the agent:

    1. Analog Controls: The policy emulates an XBox360 controller using the vgamepad library. This allows for continuous, analog inputs (e.g., precise steering and gas/brake pressure).
    2. Binary Controls: The policy outputs simple arrow key presses (discrete actions).

  5. Understand the remote training architecture

    master

    TMRL uses a client-server architecture built with tlspyo to support distributed training across multiple machines (rollout workers, central servers, and trainers).

    • Rollout Workers: Collect training samples from the environment (e.g., multiple computers or robots). They store samples in a local buffer, periodically send this buffer to the central server, and receive updated policy weights from the server.
    • Central Server: Acts as a hub. It collects samples from all connected rollout workers, stores them in a local buffer, and sends this buffer to the trainer. It also receives updated policy weights from the trainer and broadcasts them to all workers.
    • Trainer Interface: Typically located on a high-compute machine (like a GPU cluster). It receives samples from the central server, appends them to a replay memory, and periodically sends new policy weights back to the central server.

  6. How the Trainer, Server, and RolloutWorkers interact

    master

    The TMRL training pipeline follows a distributed architecture:

    1. RolloutWorkers: Gather samples from the environment and send them (often compressed) to the Server.
    2. Server: Acts as a central hub, receiving samples from workers and broadcasting updated model weights back to them.
    3. Trainer: Connects to the Server to receive samples. It stores these in a Memory object. A TrainingAgent then uses these samples to optimize the policy. Once optimized, the Trainer sends the new weights back to the Server for distribution.

  7. Understand competition evaluation and penalties

    master

    Entries are evaluated over 10 runs using the evaluation script. Your score is the mean time achieved on the tmrl-test.Map.Gbx track.

    Penalties and Elimination:

    • If the car crashes (episode auto-resets), the episode is not counted, and a 10-second penalty is added to the next episode.
    • After 3 crashes, the submission is eliminated.

    Note: For evaluation, the SLEEP_TIME_AT_RESET entry in config.json is set to 0.0 to speed up the process, though 1.5 is recommended for training.

  8. Understand the available observation spaces in TrackMania

    master

    The TrackMania pipeline provides several types of observations that can be fed into the neural network:

    • Screenshots: A history of raw screenshots (typically a sequence of 4).
    • LIDAR: A history of LIDAR measurements computed from raw screenshots (useful in tracks with black borders).
    • Velocity: The norm of the car's velocity is provided as part of the observation space.

    In TrackMania 2020, raw speed is retrieved directly via the OpenPlanet API, whereas in TrackMania Nations Forever, it is computed from screenshots using a 1-NN algorithm.

  9. Use Gymnasium environments with TMRL

    master

    TMRL is designed for asynchronous remote training of real-time applications. It uses Real-Time Gym (rtgym) to wrap robots or video games into Gymnasium environments.

    While TMRL is optimized for rtgym interfaces, you can use any environment as long as it is registered as a Gymnasium environment. To build custom real-time environments, you should follow the rtgym tutorial or refer to the tm_gym_interfaces.py in the TMRL repository for TrackMania-specific implementations.

    # Example of an rtgym interface implementation

class DummyRCDroneInterface(RealTimeGymInterface):
    def __init__(self):
        # ... initialization ...

    def get_observation_space(self):
        # Define spaces using gymnasium.spaces
        return spaces.Tuple((...))

    def get_action_space(self):
        return spaces.Box(...)

    def get_default_action(self):
        return np.array([0.0, 0.0], dtype='float32')

    def send_control(self, control):
        # Send actions to the actual robot/game
        self.rc_drone.send_control(vel_x, vel_y)

    def reset(self, seed=None, options=None):
        # Reset logic
        return obs, info

    def get_obs_rew_terminated_info(self):
        # Core method for real-time step
        return obs, rew, terminated, info

    def wait(self):
        pass

    def render(self):
        # Optional rendering logic
        pass

  10. Secure TMRL communication on public networks

    master

    By default, tmrl transfers objects via non-encrypted TCP. This is safe for private networks but constitutes a security breach on public networks (like the Internet).

    To enable secure communication using Transport Layer Security (TLS), follow these steps on all machines:

    1. Open config.json.
    2. Set the "TLS" entry to true.
    3. Replace the "PASSWORD" entry with a strong, unique password (must be identical on all machines).
    4. On the Server machine, generate a TLS key and certificate (using tlspyo instructions).
    5. Copy the generated certificate to all other machines (either to the default tlspyo credentials directory or a custom directory).
    6. If using a custom directory, update the "TLS_CREDENTIALS_DIRECTORY" entry in config.json with its path.

    Warning: If you choose not to use TLS, you must at least use a custom password in config.json. However, be aware that without TLS, this password is sent in plain text and can be intercepted.

  11. Use the tmrl.memory module for experience replay

    master

    The tmrl.memory module provides classes for managing experience replay buffers, which are used to store and sample transitions for reinforcement learning. The module contains two primary implementations:

    • Memory: A base class or general implementation for storing experiences.
    • TorchMemory: A specialized implementation designed for use with PyTorch, optimized for storing and retrieving transitions as tensors.

  12. Configure ActorModule observation and action spaces

    master

    When developing for the Full environment in the current competition iteration (Beta), adhere to these constraints:

    • Action Space: Default TrackMania 2020 continuous action space (3 floats between -1.0 and 1.0).
    • Observation Space: A history of 4 raw snapshots plus speed, gear, rpm, and 2 previous actions.
    • Control Frequency: 20 Hz.
    • Image Settings: You can customize IMG_GRAYSCALE (set to false for color), IMG_WIDTH, and IMG_HEIGHT.
    • Window Dimensions: WINDOW_WIDTH and WINDOW_HEIGHT must be between (256, 128) and (958, 488). Dimensions larger than (958, 488) are prohibited.
    • Hardware Constraint: The ActorModule must be lightweight enough to run at 20Hz on a Windows 11 machine with an i7-12700H CPU and RTX3080-Ti (laptop) GPU while running TrackMania 2020 in parallel.