Transformers4Rec Documentation

The merlin_standard_lib package is a core component of Transformers4Rec that provides standardized utilities, schemas, and registry mechanisms. It is organized into several functional subpackages:

• merlin_standard_lib.proto: Contains Protocol Buffer definitions used for data serialization and communication.
• merlin_standard_lib.schema: Provides tools for defining and managing data schemas, ensuring consistency between data loading and model training.
• merlin_standard_lib.utils: Contains general-purpose utility functions for the library.
• merlin_standard_lib.registry: A module used for managing and looking up components (such as models, layers, or tasks) within the Merlin ecosystem.

The transformers4rec.torch.model package provides the core building blocks for constructing transformer-based recommendation models using PyTorch. It is organized into several specialized submodules:

• transformers4rec.torch.model.model: Contains the primary model architecture components, such as the main transformer blocks, input processing modules, and the end-to-end model classes.
• transformers4rec.torch.model.head: Provides prediction heads that sit on top of the transformer backbone to perform specific tasks (e.g., predicting the next item).
• transformers4rec.torch.model.prediction_task: Defines the logic and objectives for different prediction tasks used during training and evaluation.

The end-to-end session-based recommendation workflow in Transformers4Rec involves several distinct stages using the Yoochoose e-commerce dataset as a reference. The pipeline includes:

1. Preprocessing: Using NVTabular to preprocess the dataset.
2. Feature Generation: Generating session features on the GPU.
3. Data Loading: Using the NVTabular dataloader with PyTorch.
4. Model Training: Training a session-based recommendation model using a Transformer architecture (specifically XLNET).
5. Deployment: Exporting both the preprocessing workflow and the trained model to Triton Inference Server (TIS).
6. Inference: Sending requests to TIS to generate next-item predictions for sessions.

The transformers4rec.torch.features package provides specialized feature modules for processing different data types in recommendation models using PyTorch. You can use these submodules to transform raw input data into model-ready tensors.

Available feature submodules include:

• base: Core base classes and utilities for feature definitions.
• continuous: For handling continuous/numerical features.
• embedding: For handling categorical features that require embedding layers.
• sequence: For handling sequential data (e.g., user interaction histories).
• tabular: For processing tabular-style input features.
• text: For processing text-based features.

Transformers4Rec is a PyTorch-based library that bridges Natural Language Processing (NLP) and Recommender Systems (RecSys). It allows users to apply state-of-the-art Transformer architectures (via Hugging Face integration) to sequential and session-based recommendation tasks.

Unlike standard NLP transformers that only accept token IDs, Transformers4Rec supports rich sequential tabular data by using a schema to automatically create embedding tables, projection layers, and output layers. It is designed to be modular and scalable, integrating with the NVIDIA Merlin ecosystem (NVTabular and Triton Inference Server) for GPU-accelerated end-to-end pipelines.

Transformers4Rec is designed to work alongside the HuggingFace (HF) Transformers library. It leverages the standardized Transformer architecture and configuration classes from HuggingFace, while providing the specialized components required for recommendation systems.

While HF Transformers focuses on NLP building blocks (Tokenizer, Transformer architecture, and NLP heads), Transformers4Rec extends this ecosystem with:

• Input feature processing: Normalization and aggregation of tabular/sequential features.
• Recommendation heads: Specialized heads for recommendation, sequence classification, and prediction tasks.
• Extended Trainer: An extension of the transformers4rec.torch.trainer.Trainer class that supports evaluation using RecSys-specific metrics.

Transformers4Rec uses NVTabular for high-scale feature engineering and preprocessing.

In recommendation tasks, interaction-level features must be converted into sequences (grouped by user or session) where sequence lengths match. NVTabular facilitates this using the Groupby operation, which allows you to:

• Group by categorical columns (e.g., session_id).
• Sort by columns (e.g., timestamp).
• Aggregate columns into sequences (using 'list') or extract specific elements (using 'first' or 'last').

groupby_features = [
    'user_id', 'session_id', 'product_id', 'category_id', 'timestamp'
] >> ops.Groupby(
    groupby_cols=['session_id'],
    sort_cols=['timestamp'],
    aggs={
        'product_id': 'list',
        'category_id': 'list',
        'timestamp': ['first', 'last'],
    },
)

When using NVTabular for preprocessing, two main outputs are generated:

1. Parquet Files: The preprocessed data is saved in Parquet format. You can partition these files using to_parquet(path, partition_on=[cols]).
2. schema.pbtxt: A protobuf text file containing statistics obtained during preprocessing, such as:
   • Cardinality of categorical features.
   • Maximum sequence length for sequential features.
   • Feature tags (e.g., identifying item IDs, user features, or categorical vs. continuous features).

Note: If you are not using NVTabular, you can manually instantiate a Schema object in your code.

Transformers4Rec uses a modular building-block design that allows you to combine standard PyTorch modules into custom architectures. A typical recommendation model is divided into four conceptual layers:

1. Feature aggregation (Input Block): Converts raw sequences (IDs, metadata) into a single vector per element called an interaction embedding.
2. Sequence masking: Defines which positions in a sequence are masked during training (e.g., for causal or masked language modeling).
3. Sequence processing (Transformer/RNN Block): Processes interaction embeddings using architectures like XLNet, GPT-2, or RNNs (LSTM/GRU).
4. Prediction head (Output Block): Produces the final predictions (e.g., next item, binary classification, or regression).

This modularity enables complex setups like multi-task learning with multiple towers and multiple heads.

Transformers4Rec is designed to work within the NVIDIA Merlin ecosystem. Key components for a complete recommendation pipeline include:

• NVTabular: Use this for feature engineering and preprocessing of tabular data at scale before feeding it into Transformers4Rec.
• Triton Inference Server: Once a Transformers4Rec model is trained, export it to be served via Triton for optimized CPU/GPU inference in cloud or edge environments.
• HugeCTR: A GPU-accelerated framework for distributed training and CTR estimation that complements the Merlin ecosystem.

Transformers4Rec leverages the Transformer architecture and its configuration classes from Hugging Face Transformers.

While HF Transformers provides the core architecture, Transformers4Rec adds specialized building blocks required for recommendation systems, including:

• Input features: Normalization and aggregation logic.
• Heads: Specialized heads for recommendation, sequence classification, and prediction.
• Extended Trainer: The Trainer class is extended to support evaluation using RecSys-specific metrics.

Note that Transformers4Rec does not use the HF Tokenizer building block; instead, it focuses on the architecture and specialized recommendation components.

This guide provides a workflow for building session-based recommendation models using Transformers4Rec. The process involves four main stages:

1. Data Generation: Creating synthetic user interaction data.
2. Preprocessing: Using NVTabular on GPU to preprocess sequential data.
3. Data Loading: Utilizing the NVTabular dataloader integrated with PyTorch.
4. Model Training: Training a session-based recommendation model using a Transformer architecture (specifically XLNET).

For detailed implementation, refer to the specific notebooks:

• 01-ETL-with-NVTabular.ipynb for ETL processes.
• 02-session-based-XLNet-with-PyT.ipynb for the PyTorch XLNET implementation.