Acme Reinforcement Learning Framework

repository·master·Indexed 26 days ago

A research framework for reinforcement learning (RL) providing modular building blocks, reference implementations, and strong baselines. Acme supports scaling from single-stream to distributed environments and includes agents for continuous control (D4PG, MPO), discrete actions (DQN), offline learning (BC, BCQ), and from-demonstration tasks (DQfD). It integrates with dm_env, OpenAI Gym, and the Behaviour Suite (bsuite), with backend support for JAX and TensorFlow.

Tokens: 7.5K
Snippets: 20
Records: 41
Agent score: 88%

What's inside Acme

Overview of Acme Reinforcement Learning Framework
master
Acme is a reinforcement learning (RL) research framework composed of modular building blocks. It is designed to provide:
- Reference Implementations: High-quality implementations of RL algorithms.
- Strong Baselines: Reliable performance baselines for algorithm comparison.
- Research Flexibility: Simple and flexible components that can be used as starting points for novel research.
- Scalability: Agents designed to run at multiple scales, ranging from single-stream to distributed environments.
https://github.com/google-deepmind/acme/blob/master/docs/index.rst
Use imitation learning baselines
master
The imitation folder contains run scripts for imitation learning algorithms. This setup is designed to imitate expert traces provided in a dataset without the use of a reward function.
https://github.com/google-deepmind/acme/blob/master/examples/baselines/imitation/README.md
Run agents on the Behaviour Suite (bsuite)
master
You can evaluate Acme agents using the Behaviour Suite for Reinforcement Learning (bsuite). Examples include:
- DQN: Off-policy DQN implementation.
- Impala: On-policy Impala agent.
- MCTS: A model-based agent that can use either an environment simulator or a learned model.
https://github.com/google-deepmind/acme/blob/master/examples/README.md
Get started with Acme via Quickstart and Tutorials
master
To begin using Acme, you can follow two primary paths:
1. Quickstart: Use the Quickstart Notebook to learn how to instantiate a simple agent and run it on an environment.
2. Tutorial: Follow the Tutorial Notebook for an in-depth look at constructing a D4PG agent. This tutorial highlights the general structure of most Acme agents, which is applicable across the framework.
https://github.com/google-deepmind/acme/blob/master/examples/README.md
Integrate dm_env environments with Acme
master
Acme is designed to work with environments implementing the dm_env interface. This provides a standard API for actions and observations. To obtain a full environment spec compatible with Acme agents, use acme.make_environment_spec(environment).
Acme provides several wrappers in acme.wrappers to modify environment behavior:
- SinglePrecisionWrapper: Converts float and int components to single-precision.
- AtariWrapper: Implements standard ALE Atari modifications (from the Nature Atari paper).
- acme.wrappers.gym_wrapper.GymWrapper: Allows interaction with OpenAI Gym environments.
- acme.wrappers.gym_wrapper.AtariGymWrapper: A Gym wrapper that exposes a lives count observation.
https://github.com/google-deepmind/acme/blob/master/docs/user/components.md
Run Continuous Control agents (D4PG and MPO)
master
Acme provides agents for continuous control tasks. Note that many examples using the DeepMind Control Suite require a MuJoCo license.
Available agents include:
- D4PG: A deterministic policy gradient agent with a deterministic policy and a distributional critic. It can run on DeepMind Control Suite or OpenAI Gym (defaults to the "half cheetah" environment).
- MPO: A maximum-a-posterior policy optimization agent combining a distributional critic and a stochastic policy.
https://github.com/google-deepmind/acme/blob/master/examples/README.md
Install Acme from source
master
If you want to run the bleeding-edge version of Acme, clone the GitHub repository and install it from the main directory (where setup.py is located) using the following command:
```
pip install .[jax,tf,testing,envs]
```
https://github.com/google-deepmind/acme/blob/master/README.md
Install ROMs for Atari environments
master
If you are using Atari environments, you must follow the specific instructions for installing ROMs to ensure they function correctly within Acme.
https://github.com/google-deepmind/acme/blob/master/docs/faq.md
Run Discrete Agents for Atari (DQN)
master
For discrete-action benchmarks, Acme includes implementations for the Arcade Learning Environment (Atari):
- DQN: A classic benchmark agent for Atari tasks.
https://github.com/google-deepmind/acme/blob/master/examples/README.md
Format data for Acme learners
master
Acme learners consume dataset iterators (typically tf.Dataset instances). These iterators should yield either transition tuples or sequences of (state, action, reward, etc.) tuples. If your data format differs, you will need to write an adaptor to match the expected input of your specific agent.
https://github.com/google-deepmind/acme/blob/master/docs/faq.md
Make environments compatible with Acme
master
Acme agents are designed to work with environments that implement the dm_env environment interface. If your environment uses a different interface, you must write a wrapper to make it conform to dm_env.
For OpenAI Gym environments, you can use the acme.wrappers.gym_wrapper module to facilitate interaction.
https://github.com/google-deepmind/acme/blob/master/docs/faq.md
Implement an environment loop with Acme Actors
master
To run an Acme agent, you connect an actor instance to an environment that conforms to the DeepMind Environment API. The interaction follows a standard reinforcement learning loop where the actor uses select_action to choose actions, observe to record environment transitions, and update to perform learning steps.
Note: While some older implementations use observe_last, the standard pattern is to use observe_first for the initial state and observe for subsequent steps.
```
while True:
  # Make an initial observation.
  step = environment.reset()
  actor.observe_first(step.observation)

  while not step.last():
    # Evaluate the policy and take a step in the environment.
    action = actor.select_action(step.observation)
    step = environment.step(action)

    # Make an observation and update the actor.
    actor.observe(action, next_step=step)
    actor.update()
```
https://github.com/google-deepmind/acme/blob/master/docs/user/overview.md