d3rlpy Documentation

The ReplayBuffer is a highly modularized interface used to represent experience replay buffers. Instead of a monolithic class, it is composed of several sub-components that you can swap out to customize your experiments:

1. Buffer: A list-like component (e.g., FIFOBuffer, InfiniteBuffer) that stores and drops transitions.
2. TransitionPicker: Defines how to pick transition data, typically used for Q-learning-based algorithms.
3. TrajectorySlicer: Defines how to slice trajectory data, typically used for Decision Transformer-based algorithms.
4. WriterPreprocess: Defines how experiences are processed before being written to the buffer.

You can initialize a ReplayBuffer using a Gym environment, a pre-collected dataset, or by manually specifying Signature objects for observations, actions, and rewards.

import d3rlpy
import numpy as np
import gym

# Component setup
buffer = d3rlpy.dataset.FIFOBuffer(limit=100000)
transition_picker = d3rlpy.dataset.BasicTransitionPicker()
trajectory_slicer = d3rlpy.dataset.BasicTrajectorySlicer()
writer_preprocessor = d3rlpy.dataset.BasicWriterPreprocess()

# Option 1: Initialize with Gym environment
env = gym.make("Pendulum-v1")
replay_buffer = d3rlpy.dataset.ReplayBuffer(
    buffer=buffer,
    transition_picker=transition_picker,
    trajectory_slicer=trajectory_slicer,
    writer_preprocessor=writer_preprocessor,
    env=env,
)

# Option 2: Initialize with pre-collected dataset
dataset, _ = d3rlpy.datasets.get_pendulum()
replay_buffer = d3rlpy.dataset.ReplayBuffer(
    buffer=buffer,
    transition_picker=transition_picker,
    trajectory_slicer=trajectory_slicer,
    writer_preprocessor=writer_preprocessor,
    episodes=dataset.episodes,
)

# Option 3: Initialize with manually specified signatures
observation_signature = d3rlpy.dataset.Signature(shape=[(3,)], dtype=[np.float32])
action_signature = d3rlpy.dataset.Signature(shape=[(1,)], dtype=[np.float32])
reward_signature = d3rlpy.dataset.Signature(shape=[(1,)], dtype=[np.float32])
replay_buffer = d3rlpy.dataset.ReplayBuffer(
    buffer=buffer,
    transition_picker=transition_picker,
    trajectory_slicer=trajectory_slicer,
    writer_preprocessor=writer_preprocessor,
    observation_signature=observation_signature,
    action_signature=action_signature,
    reward_signature=reward_signature,
)

# Shortcut for FIFO buffer
replay_buffer = d3rlpy.dataset.create_fifo_replay_buffer(limit=100000, env=env)

Decision Transformer-based algorithms (like DecisionTransformer and DiscreteDecisionTransformer) require specific handling for evaluation and interaction because they are stateful.

To integrate these algorithms into an interaction loop, use the .as_stateful_wrapper(target_return=...) method. This returns an actor that manages the necessary history for the transformer. You must call .reset() on the actor when starting a new episode.

import d3rlpy

dataset, env = d3rlpy.datasets.get_pendulum()

dt = d3rlpy.algos.DecisionTransformerConfig().create(device="cuda:0")

# offline training
dt.fit(
   dataset,
   n_steps=100000,
   n_steps_per_epoch=1000,
   eval_env=env,
   eval_target_return=0,  # specify target environment return
)

# wrap as stateful actor for interaction
actor = dt.as_stateful_wrapper(target_return=0)

# interaction
observation, reward = env.reset(), 0.0
while True:
    action = actor.predict(observation, reward)
    observation, reward, done, truncated, _ = env.step(action)
    if done or truncated:
        break

# reset history
actor.reset()

d3rlpy supports offline policy selection using Fitted Q Evaluation (FQE), an offline on-policy RL algorithm.

The Concept: FQE trains a Q-function using a trained policy in an on-policy manner. This ensures the learned Q-function reflects the expected return of that specific policy. By using FQE's Q-value estimations, you can rank candidate policies using only an offline dataset, without needing to run them in a live environment.

Key Considerations:

• Action Spaces: FQE is confirmed to work well with discrete action-space policies. Ranking continuous action-space policies typically requires hyperparameter tuning.
• Convergence: Note that offline RL training (including FQE) often does not show traditional convergence due to the non-fixed bootstrapped target.
• Scorers: To rank policies, you can use specific evaluators like InitialStateValueEstimationEvaluator (which computes mean action-value at initial states) or SoftOPCEvaluator (which computes the difference between action-value estimations for success episodes vs. all episodes).

The following pattern demonstrates the standard lifecycle in d3rlpy: preparing a dataset, configuring an algorithm, training (offline or online), and predicting actions.

import d3rlpy

# 1. Load dataset and environment
dataset, env = d3rlpy.datasets.get_dataset("hopper-medium-v0")

# 2. Prepare algorithm using a Config object
sac = d3rlpy.algos.SACConfig(compile_graph=True).create(device="cuda:0")

# 3. Train offline using the dataset
sac.fit(dataset, n_steps=1000000)

# 4. Alternatively, train online using the environment
sac.fit_online(env, n_steps=1000000)

# 5. Use the trained model to predict actions
actions = sac.predict(x)

import d3rlpy

dataset, env = d3rlpy.datasets.get_dataset("hopper-medium-v0")

# prepare algorithm
sac = d3rlpy.algos.SACConfig(compile_graph=True).create(device="cuda:0")

# train offline
sac.fit(dataset, n_steps=1000000)

# train online
sac.fit_online(env, n_steps=1000000)

# ready to control
actions = sac.predict(x)

The .pt format saves only the PyTorch model weights. To use a .pt file, you must first manually instantiate the algorithm configuration and then build the model architecture using one of three methods:

1. Using an MDPDataset object: cql.build_with_dataset(dataset)
2. Using a Gym-styled environment: cql.build_with_env(env)
3. Manually setting shapes: cql.create_impl(observation_shape, action_size)

After building the architecture, use cql.load_model("model.pt") to load the weights.

# save pt file
cql_old.save_model("model.pt")

# setup algorithm manually
cql = d3rlpy.algos.CQLConfig().create()

# choose one of three to build PyTorch models
# if you have MDPDataset object
cql.build_with_dataset(dataset)
# or if you have Gym-styled environment object
cql.build_with_env(env)
# or manually set observation shape and action size
cql.create_impl((3,), 1)

# load pretrained model
cql.load_model("model.pt")