DOCSEARCHbyAlgolia
Submit Docs

pytorch-widedeep

repository·main·Indexed 23 days ago

https://github.com/jrzaurin/pytorch-widedeep

A PyTorch-based library for building and training Wide & Deep models, optimized for tabular and text data in recommendation and classification tasks. It provides specialized preprocessors (WidePreprocessor, TabPreprocessor), a Trainer for binary and multiclass objectives, and a variety of model components including TabMlp, TabTransformer, HFModel for text, and Vision wrappers for images. It also includes specialized recommendation models such as AutoInt, DeepFM, and DCN V2.

Tokens
51K
Snippets
108
Records
179
Agent score
79%

What's inside pytorch-widedeep

  1. Available callbacks in pytorch-widedeep

    main

    The pytorch-widedeep library provides several callbacks to monitor and control the training process.

    User-selectable callbacks:

    • LRHistory: Tracks learning rate history.
    • ModelCheckpoint: Saves model weights based on specific criteria.
    • EarlyStopping: Stops training when a monitored metric stops improving.
    • RayTuneReporter: Integrates with Ray Tune for reporting.

    Default behavior:

    • The History callback runs by default and automatically saves training metrics into the history attribute of the Trainer instance.

  2. Self-supervised pre-training for tabular data

    main

    The library provides two main routines for self-supervised pre-training of tabular models. These routines allow you to pre-train models on unlabeled data before fine-tuning them on labeled data.

    Supported Models: All tabular models in the library are supported except for TabPerceiver. Support for TabPerceiver is planned for future versions.

    Available Methods:

    1. Encoder-Decoder Architecture (TabNet style): Designed for models that do not use transformer-based architectures or when embeddings have different dimensions. It uses a standard encoder-decoder setup.
    2. Contrastive and Denoising Learning (SAINT style): Designed for transformer-based architectures or when all embeddings must have the same dimension.

    To implement the full workflow (pre-training followed by supervised training), combine these self-supervised trainers with the standard Trainer class.

  3. Use the preprocessing module to prepare data for models

    main

    The preprocessing module provides specialized classes to prepare data for different model components. You should select a preprocessor based on the data mode or model component you are using:

    • Wide component: Use WidePreprocessor.
    • Tabular component: Use TabPreprocessor.
    • Image component: Use ImagePreprocessor.
    • Text component:
      • If not using a Hugging Face model: Use TextPreprocessor.
      • If using a Hugging Face model: Use HFPreprocessor.
    • DIN component: Use DINPreprocessor.

  4. Dimension requirements for metrics

    main

    When using metrics in pytorch-widedeep, ensure your tensors follow these dimension requirements:

    • Regression and Binary Classification: Both predictions (y_pred) and ground truth (y_true) must have the same dimensions: (N_samples, 1).
    • Multiclass Classification: The ground truth (y_true) is expected to be a 1D tensor containing the corresponding class indices.

  5. Use image preprocessors from image_utils

    main

    The pytorch_widedeep.utils.image_utils module provides image preprocessing utilities adapted from Adrian Rosebrock's Deep Learning for Computer Vision. It includes two primary preprocessor classes:

    • SimplePreprocessor: A basic preprocessor for resizing images.
    • AspectAwarePreprocessor: A preprocessor that resizes images while maintaining their original aspect ratio, preventing distortion.

    These classes are intended for use in computer vision pipelines where consistent image dimensions are required for model input.

  6. Understand the fastai_transforms utility module

    main

    The pytorch_widedeep.utils.fastai_transforms module contains a subset of the fastai library's transforms.py module, specifically the Tokenizer and Vocab classes. This module is included to provide necessary text processing capabilities without requiring the full fastai dependency.

    Note that these implementations are derived from an older version of fastai. For comprehensive details on how these transformations work, users are encouraged to consult the official fastai documentation.

  7. How to use custom components in WideDeep

    main

    You can use custom models as components in WideDeep as long as they have an output_dim property that returns the size of the last layer of activations.

    While you don't strictly need to inherit from BaseWDModelComponent, doing so is recommended to avoid typing errors internally. The BaseWDModelComponent simply checks for the existence of the output_dim property.

    Custom components are useful for complex fusion logic, such as combining text, image, and tabular data with a custom head (e.g., a deephead parameter in WideDeep).

  8. Data dimension requirements for losses

    main

    When using losses in pytorch-widedeep, ensure your predictions and ground truth tensors follow these dimensionality rules:

    • Regression and Binary Classification: Both predictions and ground truth must have the same dimensions, typically $(N_{samples}, 1)$.
    • Multiclass Classification: The ground truth is expected to be a 1D tensor containing the corresponding class indices.

  9. Understand the Wide & Deep model components

    main

    The pytorch_widedeep.models module provides the building blocks for constructing Wide & Deep architectures. A complete Wide & Deep model is typically composed of four main components:

    1. wide: A component for memorization (e.g., Wide linear models).
    2. deeptabular: A component for generalization using tabular data (e.g., TabMlp, TabResnet, TabNet, TabTransformer, SAINT, FTTransformer).
    3. deeptext: A component for processing text data (e.g., BasicRNN, AttentiveRNN, HFModel).
    4. deepimage: A component for processing image data (e.g., Vision).

    These components can be used independently or combined using the WideDeep constructor class or a ModelFuser.

  10. Use chunked preprocessors for large datasets

    main

    When your dataset is too large to fit into memory, use the chunked versions of the preprocessors. These are designed to process data in segments to manage memory usage efficiently.

    Available chunked preprocessors:

    • ChunkWidePreprocessor
    • ChunkTabPreprocessor
    • ChunkTextPreprocessor
    • ChunkHFPreprocessor

    Note on Images: There is no ChunkImagePreprocessor. Image processing for large datasets should be handled via the ImageFromFolder class within the load_from_folder module.

  11. Use Bayesian models for uncertainty estimation

    main

    The bayesian_models module provides Bayesian versions of the Wide and TabMlp architectures. These models are designed for scenarios where obtaining a measure of uncertainty is critical. They are based on the methodology described in the paper Weight Uncertainty in Neural Networks.

    Available models:

    • BayesianWide: A Bayesian version of the Wide model.
    • BayesianTabMlp: A Bayesian version of the TabMlp model.

  12. Quick Start: Binary Classification with WideDeep

    main

    This guide demonstrates how to perform binary classification using a WideDeep model (specifically a deeptabular architecture) on the adult census dataset. The workflow involves defining column setups for wide and tabular features, preprocessing data with WidePreprocessor and TabPreprocessor, building the model components, training via the Trainer class, and performing predictions.

    import numpy as np
import torch
from sklearn.model_selection import train_test_split

from pytorch_widedeep import Trainer
from pytorch_widedeep.preprocessing import WidePreprocessor, TabPreprocessor
from pytorch_widedeep.models import Wide, TabMlp, WideDeep
from pytorch_widedeep.metrics import Accuracy
from pytorch_widedeep.datasets import load_adult

# 1. Load and prepare data
df = load_adult(as_frame=True)
df["income_label"] = (df["income"].apply(lambda x: ">50K" in x)).astype(int)
df.drop("income", axis=1, inplace=True)
df_train, df_test = train_test_split(df, test_size=0.2, stratify=df.income_label)

# 2. Define column setup
wide_cols = [
    "education",
    "relationship",
    "workclass",
    "occupation",
    "native-country",
    "gender",
]
crossed_cols = [("education", "occupation"), ("native-country", "occupation")]

cat_embed_cols = [
    "workclass",
    "education",
    "marital-status",
    "occupation",
    "relationship",
    "race",
    "gender",
    "capital-gain",
    "capital-loss",
    "native-country",
]
continuous_cols = ["age", "hours-per-week"]
target = "income_label"
target_values = df_train[target].values

# 3. Preprocess data
wide_preprocessor = WidePreprocessor(wide_cols=wide_cols, crossed_cols=crossed_cols)
X_wide = wide_preprocessor.fit_transform(df_train)

tab_preprocessor = TabPreprocessor(
    cat_embed_cols=cat_embed_cols, continuous_cols=continuous_cols
)
X_tab = tab_preprocessor.fit_transform(df_train)

# 4. Build the model
wide = Wide(input_dim=np.unique(X_wide).shape[0], pred_dim=1)
tab_mlp = TabMlp(
    column_idx=tab_preprocessor.column_idx,
    cat_embed_input=tab_preprocessor.cat_embed_input,
    continuous_cols=continuous_cols,
)
model = WideDeep(wide=wide, deeptabular=tab_mlp)

# 5. Train and validate
trainer = Trainer(model, objective="binary", metrics=[Accuracy])
trainer.fit(
    X_wide=X_wide,
    X_tab=X_tab,
    target=target_values,
    n_epochs=5,
    batch_size=256,
)

# 6. Predict on test
X_wide_te = wide_preprocessor.transform(df_test)
X_tab_te = tab_preprocessor.transform(df_test)
preds = trainer.predict(X_wide=X_wide_te, X_tab=X_tab_te)

# 7. Save and load
trainer.save(path="model_weights", save_state_dict=True)

# Loading example
model_new = WideDeep(wide=wide, deeptabular=tab_mlp)
model_new.load_state_dict(torch.load("model_weights/wd_model.pt"))
trainer_new = Trainer(model_new, objective="binary")
preds_new = trainer_new.predict(X_wide=X_wide, X_tab=X_tab, batch_size=32)