Recommenders

SARplus is an efficient implementation of the Simple Algorithm for Recommendation (SAR) designed for Spark. SAR is a neighborhood-based algorithm that provides personalized recommendations by finding items similar to those a user has already interacted with (affinity).

Key features include:

• Scalable PySpark-based implementation.
• Fast C++ based predictions.
• Reduced memory consumption: the similarity matrix is cached in-memory once per worker and shared across Python executors.

Recommenders is designed to assist in the full lifecycle of building recommendation systems through Jupyter notebook examples and library utilities. The project focuses on five key tasks:

1. Prepare Data: Preparing and loading data for specific recommendation algorithms.
2. Model: Building models using classical and deep learning algorithms (e.g., ALS, xDeepFM).
3. Evaluate: Evaluating algorithms using offline metrics.
4. Model Select and Optimize: Tuning and optimizing hyperparameters.
5. Operationalize: Moving models into production environments.

The examples/01_prepare_data/ directory contains notebooks demonstrating utility functions for common recommendation system data operations, including import/export, transformation, and splitting.

Available notebooks include:

• data_split.ipynb: Demonstrates various data splitting strategies.
• data_transform.ipynb: Provides guidance on transforming implicit or explicit data for collaborative filtering recommenders.
• wikidata_knowledge_graph.ipynb: Details the creation of a knowledge graph using Wikidata.

SARplus (Simple Algorithm for Recommendation) is a neighborhood-based algorithm designed for personalized recommendations using user transaction history. It is an efficient implementation of the SAR algorithm optimized for Apache Spark.

Core Concepts:

• Affinity: A user has an affinity for an item if they have interacted with it in the past.
• Similarity: Two items are considered similar if users who interacted with one item are also likely to have interacted with the other.

The algorithm works by recommending items that are most similar to the items a user already has an affinity for.

News recommendation systems aim to predict which news articles a user is likely to read to mitigate information overload. When developing these systems, two primary challenges must be addressed:

1. Cold-start Problem: New articles emerge continuously and expire quickly. Systems must effectively represent and recommend new articles that lack historical interaction data.
2. Content Representation: Unlike other domains where item IDs might suffice, news recommendation relies heavily on the rich text within articles. Natural Language Processing (NLP) methods are critical for learning effective news content representations from text.

For theoretical foundations, refer to the ACL paper on MIND.

Recommenders is a project under the Linux Foundation of AI and Data designed to help researchers, developers, and enthusiasts prototype, experiment with, and deploy classic and state-of-the-art recommendation systems.

The project provides Jupyter notebooks covering five key stages of the recommendation lifecycle:

• Prepare Data: Loading and preparing data for specific algorithms.
• Model: Building models using classical and deep learning algorithms (e.g., ALS, xDeepFM).
• Evaluate: Using offline metrics to evaluate algorithm performance.
• Model Select and Optimize: Hyperparameter tuning and optimization.
• Operationalize: Deploying models in production environments (e.g., on Azure).

The recommenders package includes utilities for dataset loading, model evaluation, and data splitting, along with implementations of several state-of-the-art algorithms.

The recommenders package is organized into several functional submodules:

• datasets: Data loading (e.g., MovieLens) and splitting techniques (Random, Chronological, Stratified).
• evaluation: Metric calculation (RMSE, Precision@K, etc.) for Python and Spark.
• models: Implementations of various algorithms including DeepRec (CASER, DKN, xDeepFM, etc.), LightFM, NCF, SASRec, and more.
• tuning: Hyperparameter tuning utilities.
• utils: High-level utilities for managing constants and framework-specific aspects (GPU, Spark, Jupyter).

The recommenders.models.sasrec module provides implementations for Sequential Self-Attention based recommendation.

• SASRec: The core model is located in recommenders.models.sasrec.model.
• SSE-PT: An implementation found in recommenders.models.sasrec.ssept.
• Supporting components: Use recommenders.models.sasrec.sampler for data sampling and recommenders.models.sasrec.util for utility functions.

The recommenders.models.deeprec module provides a collection of deep learning-based recommendation models, including sequential models, graph-based models, and data processing utilities.

Key components include:

• Base Models: recommenders.models.deeprec.models.base_model.BaseModel and recommenders.models.deeprec.models.sequential.sequential_base_model.SequentialBaseModel for defining custom architectures.
• Sequential Models: Implementations like DKN, A2SVD, Caser, GRU, NextItNet, SLIRec, and SASRec (via ssept).
• Graph/Interaction Models: LightGCN and xDeepFM.
• Iterators: Specialized data loaders for sequential training, such as dkn_iterator, nextitnet_iterator, and sequential_iterator.
• Data Processing: recommenders.models.deeprec.DataModel.ImplicitCF for implicit collaborative filtering data handling.