PyTorch Tabular Documentation

PyTorch Tabular provides several methods to transform continuous features to improve model performance. By default, normalize_continuous_features is set to True, which scales features using a StandardScaler.

You can further refine feature distributions using the continuous_feature_transform parameter with the following options:

Parametric Transformations (Gaussian mapping)

These aim to map data to a Gaussian distribution to stabilize variance and minimize skewness:

• yeo-johnson: Works for any distribution.
• box-cox: Can only be applied to strictly positive data.

Non-parametric Transformations (Rank-based)

These use rank transformations to smooth distributions and are less sensitive to outliers, though they may distort correlations:

• quantile_normal: Transforms features to a normal distribution.
• quantile_uniform: Transforms features to a uniform distribution.

PyTorch Tabular uses several specialized configuration classes to manage different stages of the machine learning pipeline. These classes allow you to define settings for data processing, model architecture, training logic, and experiment management.

The core configuration classes are:

• pytorch_tabular.config.DataConfig: Manages data loading, preprocessing, and feature engineering settings.
• pytorch_tabular.config.ModelConfig: Defines the base architecture and parameters for the tabular models.
• pytorch_tabular.config.SSLModelConfig: Configuration specifically for Self-Supervised Learning (SSL) models.
• pytorch_tabular.config.TrainerConfig: Controls the training loop, including epochs, batch size, and device settings.
• pytorch_tabular.config.ExperimentConfig: Manages high-level experiment settings and reproducibility.
• pytorch_tabular.config.OptimizerConfig: Configures the optimization algorithm (e.g., learning rate, weight decay).
• pytorch_tabular.config.ExperimentRunManager: Handles the management and logging of individual experiment runs.

PyTorch Tabular supports a wide variety of architectures for classification, regression, and semi-supervised learning:

Standard Architectures

• FeedForward Network with Category Embedding: A simple FF network using embedding layers for categorical columns.
• Neural Oblivious Decision Ensembles (NODE): A model designed to compete with Gradient Boosting models.
• TabNet: Uses Sparse Attention to model outputs through multiple decision-making steps.
• Mixture Density Networks: A regression model providing probabilistic predictions via Gaussian components.
• AutoInt: Learns feature interactions automatically using self-attentive neural networks.
• TabTransformer: Adapts the Transformer model to create contextual representations for categorical features.
• FT Transformer: A Transformer-based architecture for tabular data.
• Gated Additive Tree Ensemble (GATE): Uses a gating mechanism and an ensemble of differentiable, non-linear decision trees.
• GANDALF: A more efficient, pared-down version of GATE.
• DANETs: Uses Abstract Layers (AbstLay) to group correlative features and generate higher-level semantic abstractions.

Semi-Supervised Learning

• Denoising AutoEncoder: Learns robust feature representations to compensate for noise in datasets.

Self-Supervised Learning (SSL) in PyTorch Tabular follows an encoder-decoder architecture designed to build background knowledge from unlabeled data before fine-tuning on labeled data.

The Workflow

1. Pre-training: Train the model on a large unlabeled dataset to learn feature representations.
2. Fine-tuning: Use the pre-trained model as a backbone for a downstream task (e.g., regression or classification) and train on a smaller labeled dataset.

Model Components

• Embedding Layer: Processes categorical and continuous features into a single tensor.
• Featurizer: Performs representation learning on the embedding layer output to produce learned features.
• Encoder: Compulsory component that learns the representation.
• Decoder: Optional component (defaults to nn.Identity) used to transform the intermediate representation back into a reconstruction (e.g., in Denoising AutoEncoders).

Implementation Requirements

To implement a custom SSL model, you must inherit from SSLBaseModel and implement the following methods:

• embedding_layer (property)
• featurizer (property)
• _setup_loss (sets up the loss function)
• _setup_metrics (sets up metrics)
• calculate_loss (calculates loss)
• calculate_metrics (calculates metrics)
• forward (defines the forward pass)
• featurize (returns learned features from the featurizer)

You can use PyTorch Tabular to train a model, evaluate it on a test set, generate predictions, and manage model persistence (save/load). The workflow involves defining four configuration objects: DataConfig, TrainerConfig, OptimizerConfig, and a specific model configuration (e.g., CategoryEmbeddingModelConfig), then passing them to the TabularModel class.

Key steps in the lifecycle:

1. Configure: Define data columns, training hyperparameters, and model architecture.
2. Initialize: Create a TabularModel instance with the configs.
3. Fit: Call .fit(train=..., validation=...) to train the model.
4. Evaluate: Call .evaluate(test_df) to get performance metrics.
5. Predict: Call .predict(test_df) to get a DataFrame of predictions.
6. Persist: Use .save_model(path) and TabularModel.load_model(path) to save and restore models.

from pytorch_tabular import TabularModel
from pytorch_tabular.models import CategoryEmbeddingModelConfig
from pytorch_tabular.config import (
    DataConfig,
    OptimizerConfig,
    TrainerConfig,
)

data_config = DataConfig(
    target=[
        "target"
    ],  # target should always be a list.
    continuous_cols=num_col_names,
    categorical_cols=cat_col_names,
)
trainer_config = TrainerConfig(
    auto_lr_find=True,  # Runs the LRFinder to automatically derive a learning rate
    batch_size=1024,
    max_epochs=100,
)
optimizer_config = OptimizerConfig()

model_config = CategoryEmbeddingModelConfig(
    task="classification",
    layers="1024-512-512",  # Number of nodes in each layer
    activation="LeakyReLU",  # Activation between each layers
    learning_rate=1e-3,
)

tabular_model = TabularModel(
    data_config=data_config,
    model_config=model_config,
    optimizer_config=optimizer_config,
    trainer_config=trainer_config,
)
tabular_model.fit(train=train, validation=val)
result = tabular_model.evaluate(test)
pred_df = tabular_model.predict(test)
tabular_model.save_model("examples/basic")
loaded_model = TabularModel.load_model("examples/basic")