DOCSEARCHbyAlgolia
Submit Docs

RePlay Documentation

repository·main·Indexed 19 days ago

https://github.com/sb-ai-lab/replay

An advanced framework for the development and evaluation of recommendation systems. RePlay covers the entire lifecycle from data preprocessing and splitting to model training, hyperparameter optimization, and evaluation. It includes support for PySpark and PyTorch, specialized data components for neural networks (such as ParquetDataset and ParquetModule), and a comprehensive suite of ranking, accuracy, diversity, and novelty metrics.

Tokens
37.6K
Snippets
98
Records
138
Agent score
63%

What's inside RePlay

  1. Use Neural Network data components (PyTorch required)

    main

    The replay.data.nn submodule provides specialized data structures for deep learning workflows. Note: This submodule requires PyTorch to be installed.

    Key components include:

    • Schema & Info: TensorFeatureInfo, TensorFeatureSource, and TensorSchema for handling tensor-based data.
    • Tokenization: SequenceTokenizer for processing sequences.
    • Datasets & Batches:
      • PandasSequentialDataset: For sequential data stored in Pandas.
      • TorchSequentialDataset & TorchSequentialValidationDataset: PyTorch-compatible sequential datasets.
      • TorchSequentialBatch & TorchSequentialValidationBatch: Batch representations for training and validation.

  2. Understand model execution chains for fit and predict

    main

    RePlay uses a wrapping pattern for model execution. When you call fit or predict, the call passes through a _wrap method before reaching the core implementation. The specific chain depends on the model type:

    • Standard Models: fit -> _fit_wrap -> _fit | predict -> _predict_wrap -> _predict.
    • LightFMWrap: predict -> _predict_wrap -> _predict -> _predict_selected_pairs.
    • Word2VecRec / NeighbourRec: predict -> _predict_wrap -> _predict -> _predict_pairs_inner.
    • BaseTorchRec (Torch-based):
      • fit -> _fit_wrap -> _fit -> train
      • predict -> _predict_wrap -> _predict -> _predict_by_user -> _predict_pairs_inner.

  3. Access the global Spark session via State

    main

    The library uses session_handler.State to ensure all modules share the same Spark session. By default, a session is created automatically and can be accessed via the .session attribute of the State object.

    from replay.utils.session_handler import State
# Access the automatically created default session
session = State().session

  4. Understand the RePlay Recommender model categories

    main

    RePlay categorizes its models based on implementation and execution environment:

    1. Distributed Models: Implemented in PySpark for both training and inference. Examples include PopRec, RandomRec, UCB, ItemKNN, ALSWrap, and SLIM.
    2. Neural Models with Distributed Inference: Implemented in PyTorch but uses PySpark for distributed inference. Examples include NeuroMF and MultVAE.
    3. Redesigned Neural Networks: Follow a block-based architecture where models receive pre-built component instances (losses, embedders, heads) instead of raw config parameters. They use a unified Lightning wrapper.
    4. Wrappers: Python implementations of popular libraries that use PySpark for distributed inference, such as LightFMWrap and ImplicitWrap.
    5. Hierarchical Models: Experimental models like HierarchicalRecommender.

  5. How to implement Global Temporal Split (GTS) schemes

    main

    Following the strategies in the paper 'Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders (RecSys'25)', you can implement advanced splitting schemes by composing different splitter classes:

    1. GTS with last interaction as target: Compose TimeSplitter with LastNSplitter(N=1).
    2. GTS with a random interaction as target: Compose TimeSplitter with RandomNextNSplitter(N=1).

    These pipelines can be further enhanced using auxiliary utilities:

    • Cold-start filtering: Use replay.preprocessing.filters.filter_cold.
    • Dataset merging: Use replay.preprocessing.utils.merge_subsets.

    All splitters return their results via the .split() method.

  6. Define feature metadata with FeatureInfo and FeatureSchema

    main

    The replay.data module provides several classes to define the structure and properties of data features:

    • FeatureType: An enumeration defining the type of a feature.
    • FeatureSource: An enumeration defining where a feature originates.
    • FeatureHint: An enumeration for providing hints about feature characteristics.
    • FeatureInfo: A class containing metadata about a specific feature.
    • FeatureSchema: A class defining the schema for a set of features.
    • get_schema: A function to retrieve the schema of a dataset.

  7. Quickstart: End-to-end recommendation pipeline

    main

    This example demonstrates a complete pipeline: loading data (MovieLens), splitting it using RatioSplitter, defining a FeatureSchema, creating Dataset objects, encoding labels, training an ItemKNN model, performing inference, and evaluating results with Experiment using NDCG and HitRate metrics.

    from polars import from_pandas
from rs_datasets import MovieLens

from replay.data import Dataset, FeatureHint, FeatureInfo, FeatureSchema, FeatureType
from replay.data.dataset_utils import DatasetLabelEncoder
from replay.metrics import HitRate, NDCG, Experiment
from replay.models import ItemKNN
from replay.utils.spark_utils import convert2spark
from replay.utils.session_handler import State
from replay.splitters import RatioSplitter

spark = State().session

ml_1m = MovieLens("1m")
K = 10

# convert data to polars
interactions = from_pandas(ml_1m.ratings)

# data splitting
splitter = RatioSplitter(
    test_size=0.3,
    divide_column="user_id",
    query_column="user_id",
    item_column="item_id",
    timestamp_column="timestamp",
    drop_cold_items=True,
    drop_cold_users=True,
)
train, test = splitter.split(interactions)

# datasets creation
feature_schema = FeatureSchema(
    [
        FeatureInfo(
            column="user_id",
            feature_type=FeatureType.CATEGORICAL,
            feature_hint=FeatureHint.QUERY_ID,
        ),
        FeatureInfo(
            column="item_id",
            feature_type=FeatureType.CATEGORICAL,
n            feature_hint=FeatureHint.ITEM_ID,
        ),
        FeatureInfo(
            column="rating",
            feature_type=FeatureType.NUMERICAL,
            feature_hint=FeatureHint.RATING,
        ),
        FeatureInfo(
            column="timestamp",
            feature_type=FeatureType.NUMERICAL,
            feature_hint=FeatureHint.TIMESTAMP,
        ),
    ]
)

train_dataset = Dataset(feature_schema=feature_schema, interactions=train)
test_dataset = Dataset(feature_schema=feature_schema, interactions=test)

# data encoding
encoder = DatasetLabelEncoder()
train_dataset = encoder.fit_transform(train_dataset)
test_dataset = encoder.transform(test_dataset)

# convert datasets to spark
train_dataset.to_spark()
test_dataset.to_spark()

# model training
model = ItemKNN()
model.fit(train_dataset)

# model inference
encoded_recs = model.predict(
    dataset=train_dataset,
    k=K,
    queries=test_dataset.query_ids,
    filter_seen_items=True,
)

recs = encoder.query_and_item_id_encoder.inverse_transform(encoded_recs)

# model evaluation
metrics = Experiment(
    [NDCG(K), HitRate(K)],
    test,
    query_column="user_id",
    item_column="item_id",
    rating_column="rating",
)
metrics.add_result("ItemKNN", recs)
print(metrics.results)

  8. Install RePlay via pip

    main

    Install the core RePlay package using pip. Note that the core package does not include PySpark or PyTorch dependencies by default.

    To install the experimental submodule, you must specify a version with the rc0 suffix (e.g., replay-rec==0.21.8rc0).

    pip install replay-rec

  9. Install RePlay with Spark or Torch extras

    main

    You can install additional dependencies for distributed computing or deep learning using extras:

    • [spark]: Includes PySpark functionality.
    • [torch]: Includes PyTorch and Lightning functionality.

    To install the experimental submodule along with an extra, append the rc0 version suffix.

    # Install core package with PySpark dependency
pip install replay-rec[spark]

# Install package with experimental submodule and PySpark dependency
pip install replay-rec[spark]==XX.YY.ZZrc0

# Install package with the CPU version of torch
pip install replay-rec[torch] --extra-index-url https://download.pytorch.org/whl/cpu

  10. Configure a custom Spark session for the library

    main

    You can provide your own Spark session to the library by passing it to the State constructor. This ensures that all library modules use your specific session configuration. You can also use the get_spark_session helper function to generate a session with specific parameters (e.g., number of cores).

    from replay.utils.session_handler import State, get_spark_session

# Option 1: Create a session using the helper and inject it
session = get_spark_session(2)
State(session)

# Option 2: Pass an existing session object
# State(your_existing_session)

  11. Configure transforms for ParquetModule

    main

    Transforms are torch.nn.Module subclasses that take a batch (a Python dictionary) and return a transformed copy. They are used with ParquetModule to prepare data for the model.

    To use them, pass a dictionary to the transforms parameter of ParquetModule, where keys are data splits (e.g., train, validate) and values are lists of transformations.

    {
   "train": [NextTokenTransform(label_field="item_id", shift=1), ...],
   "validate": [...]
}