Reverb Documentation

repository·master·Indexed 21 days ago

https://github.com/google-deepmind/reverb

Reverb (dm_reverb) is an efficient data storage and transport system designed for machine learning research, specifically optimized for experience replay in distributed reinforcement learning. It provides a server-client architecture for managing data tables with configurable sampling strategies (Uniform, Prioritized, FIFO, LIFO, MinHeap, MaxHeap) and rate limiters. The system supports trajectory writing for complex data structures and checkpointing for state restoration.

Tokens
12.6K
Snippets
36
Records
45
Agent score
73%

What's inside Reverb

  1. Understand TensorFlow dependency constraints

    master

    Reverb depends on TensorFlow to build its C++ extensions. There are critical compatibility rules:

    1. Minor Version Matching: Wheels are only compatible with the specific minor TensorFlow release they were built against. For example, a wheel built against tensorflow 2.20.* is not compatible with tensorflow 2.21.*.
    2. Dependency Alignment: Reverb's internal dependencies (like abseil-cpp, grpc, and protobuf) must match those used by the chosen TensorFlow version.

    To build against a different TensorFlow version, you must:

    1. Update the version in requirements.in and regenerate lock files.
    2. Update the WORKSPACE file to point to the correct TensorFlow commit/release.
    3. Update reverb/pip_package/reverb_version.bzl so the wheel metadata reflects the correct TensorFlow version.
  2. How Reverb Tables and Items work

    master

    A Reverb Server is composed of one or more Tables.

    • Items: These are the primary units stored in a table. Crucially, items do not contain copies of data; they contain references to data elements. This allows multiple items (even in different tables) to reference the same data element, saving memory.
    • Data Elements: The actual underlying data. A data element is only deleted from memory when there are no more items in any table referencing it.
    • Lifecycle: Items are automatically removed when the table reaches its max_size (using the remover strategy) or when an item exceeds its permitted sampling count (controlled by the rate_limiter and max_times_sampled).
  3. Quick Start: Server, Client, and Data Insertion

    master

    This guide demonstrates the basic lifecycle of a Reverb session: starting a server with a table, connecting a client, inserting simple data, and using a trajectory_writer for complex data structures.

    1. Start a Server

    Initialize a reverb.Server with one or more reverb.Table objects. Tables require a name, a sampler (how to pick items), a remover (how to handle overflow), a max_size, and a rate_limiter.

    2. Connect a Client

    Use reverb.Client with the server's address (e.g., localhost:{server.port}).

    3. Insert Data

    • Simple Insertion: Use client.insert(data, priorities={'table_name': priority}).
    • Trajectory Writing: For items that reference multiple data elements (like sequences), use client.trajectory_writer(). This allows you to append data elements and then create an item that references slices of that history.

    4. Sample Data

    Use client.sample('table_name', num_samples=N) to get a generator of sampled items.

    import reverb
    
    # 1. Start a Server
    server = reverb.Server(tables=[
        reverb.Table(
            name='my_table',
            sampler=reverb.selectors.Uniform(),
            remover=reverb.selectors.Fifo(),
            max_size=100,
            rate_limiter=reverb.rate_limiters.MinSize(1)),
        ],
    )
    
    # 2. Connect a Client
    client = reverb.Client(f'localhost:{server.port}')
    print(client.server_info())
    
    # 3. Write simple data
    client.insert([0, 1], priorities={'my_table': 1.0})
    
    # 3b. Write complex trajectories
    with client.trajectory_writer(num_keep_alive_refs=3) as writer:
      writer.append({'a': 2, 'b': 12})
      writer.append({'a': 3, 'b': 13})
      writer.append({'a': 4, 'b': 14})
    
      writer.create_item(
          table='my_table',
          priority=1.0,
          trajectory={
              'a': writer.history['a'][:],
              'b': writer.history['b'][:],
          })
      writer.flush()
    
    # 4. Sample data
    print(list(client.sample('my_table', num_samples=2)))
  4. Checkpoint and restore a Reverb Server

    master

    Reverb supports checkpointing the state and content of a Server to permanent storage.

    To Checkpoint: Use a Client to trigger a checkpoint. This blocks all incoming requests (insert, sample, update, delete) while the server serializes data.

    To Restore: Use a DefaultCheckpointer pointing to the root directory of your checkpoints. When initializing the reverb.Server, ensure the tables= argument matches the configuration used when the checkpoint was created.

    # Checkpointing
    checkpoint_path = client.checkpoint()
    
    # Restoring
    checkpointer = reverb.platform.checkpointers_lib.DefaultCheckpointer(
      path=checkpoint_path.rsplit('/', 1)[0])
    
    server = reverb.Server(tables=[...], checkpointer=checkpointer)
  5. Install Reverb nightly builds

    master

    If you require the latest features, you can install the nightly builds. As with the stable release, using the [tensorflow] extra is recommended for compatibility.

    To install the nightly version with TensorFlow:

    $ pip install dm-reverb-nightly[tensorflow]

    To install the nightly version without TensorFlow:

    $ pip install dm-reverb-nightly
  6. Install dependencies for building Reverb

    master

    To build Reverb Python wheels from source, you must install the following system dependencies:

    • Bazel: Used as the build system for Python extensions. It is recommended to install bazelisk to automatically manage the correct Bazel version (the version used in CI is specified in .bazelversion).
    • uv: Used for managing Python virtual environments and wheel creation tools. Follow the official installation instructions to set up uv.
  7. Build Reverb wheels with Bazel

    master

    You can bypass the shell script and build wheels directly using Bazel. Use the --repo_env=HERMETIC_PYTHON_VERSION flag to specify the Python version the wheel will be compatible with.

    # Build release wheel
    bazel build \
      --repo_env=HERMETIC_PYTHON_VERSION=3.12 \
      --repo_env=WHEEL_NAME=dm_reverb \
      --repo_env=ML_WHEEL_TYPE=release \
      //reverb/pip_package:wheel
    
    # Build nightly wheel
    bazel build \
      --repo_env=HERMETIC_PYTHON_VERSION=3.12 \
      --repo_env=WHEEL_NAME=dm_reverb_nightly \
      --repo_env=ML_WHEEL_TYPE=nightly \
      --repo_env=ML_WHEEL_BUILD_DATE=`date '+%Y%m%d'` \
      //reverb/pip_package:wheel
  8. Start a Reverb server using the CLI

    master

    After installing dm-reverb via pip, you can start a server using the reverb_server script. The server is configured via a textproto file passed to the --config flag.

    $ reverb_server --config="
    port: 8000
    tables: {
      table_name: \"my_table\"
      sampler: {
        fifo: true
      }
      remover: {
        fifo: true
      }
      max_size: 200 max_times_sampled: 5
      rate_limiter: {
        min_size_to_sample: 1
        samples_per_insert: 1
        min_diff: $(python3 -c "import sys; print(-sys.float_info.max)")
        max_diff: $(python3 -c "import sys; print(sys.float_info.max)")
      }
    }"
  9. Update Reverb PyPI requirements

    master

    Reverb uses locked requirements for Bazel builds. To update them, modify reverb/pip_package/requirements.in and then run the update target for your specific Python version.

    # Update locked requirements for Python 3.12
    bazel run --repo_env=HERMETIC_PYTHON_VERSION=3.12 //reverb/pip_package:requirements.update
  10. Build Reverb wheels using oss_build.sh

    master

    The oss_build.sh script automates the building and testing of Reverb wheels. You can build either the release package (dm_reverb) or the nightly package (dm_reverb_nightly).

    Run the script from the root of the repository.

    Flags:

    • --python: The Python version to build for (e.g., '3.11'). Supports multiple versions: --python '3.11 3.12'.
    • --release: Determines if you build the nightly or release package. This affects the wheel name and whether the [tensorflow] optional dependency uses tensorflow or tf_nightly.
    • --python_tests: Set to true to run Python tests in a virtual environment, or false to skip.
    • --output_dir: The directory where wheels are stored (defaults to dist).

    Note: As of 2025-12-05, release builds target TensorFlow 2.21.*. If this version is unavailable on PyPI, the build will fail.

    # Build for Python 3.11
    bash oss_build.sh --python '3.11'
    
    # Install the resulting wheel with TensorFlow support
    python3 -m pip install '<path to .whl file>[tensorflow]'
  11. Install Reverb via pip

    master

    Reverb is designed for Linux-based OSes. The recommended installation method is using pip. To ensure compatibility with underlying libraries, it is highly recommended to install Reverb with the [tensorflow] extra, which automatically installs the correct version of TensorFlow associated with that Reverb release.

    To install the stable version with TensorFlow:

    $ pip install dm-reverb[tensorflow]

    To install without TensorFlow (requires you to manage dependencies manually):

    $ pip install dm-reverb
  12. What are Data Elements and Items in Reverb?

    master

    In Reverb, it is crucial to distinguish between Data Elements and Items to manage memory and storage correctly:

    • Data Element: The actual raw data written via Writer.append. Data elements are immutable once written and are not stored directly in a Table. They can be referenced by multiple items across different tables.
    • Item: The entity actually stored within a Table. Items are created via Writer.create_item and consist of one or more references to data elements, forming a "sequence."

    Memory Management Note: Because multiple items in different tables can reference the same data element, removing an item from a table does not necessarily free up memory. Data remains on the server until the last item referencing it is removed from all tables. To optimize memory, consider using consistent removal strategies across tables that share data.