DOCSEARCHbyAlgolia
Submit Docs

Ensemble-PyTorch Documentation

repository·master·Indexed 21 days ago

https://github.com/torchensemble-community/ensemble-pytorch

A unified framework for improving the performance and robustness of PyTorch deep learning models using ensemble strategies. It provides implementations for Voting, Bagging, Gradient Boosting, Fusion, Snapshot Ensemble, Adversarial Training (via FGSM), and Fast Geometric Ensembling (FGE). The library includes a standardized API with methods such as .fit(), .predict(), .set_optimizer(), and .set_scheduler() to simplify the training and evaluation of ensemble classifiers and regressors.

Tokens
7K
Snippets
11
Records
35
Agent score
74%

What's inside Ensemble-PyTorch

  1. Overview of Ensemble-PyTorch Experiments

    master

    Ensemble-PyTorch includes several experimental configurations used to evaluate the performance of different ensemble methods across various datasets and base estimators. These experiments serve as a benchmark for how different ensemble strategies (like voting, bagging, gradient boosting, and snapshot ensembles) behave depending on the complexity of the dataset and the capacity of the base model.

    Experimental Configurations

    Config NameEstimatorDatasetn_estimators
    LeNet@MNISTLeNet-5MNIST5, 10, 15, 20
    LeNet@CIFAR-10LeNet-5CIFAR-105, 10, 15, 20
    ResNet@CIFAR-10ResNet-18CIFAR-102, 5, 7, 10
    ResNet@CIFAR-100ResNet-18CIFAR-1002, 5, 7, 10

    Hyperparameter Settings used in Experiments

    • CIFAR-10 and CIFAR-100: Data augmentations were applied.
    • LeNet-5: Used Adam optimizer with learning rate 1e-3 and weight decay 5e-4.
    • ResNet-18: Used SGD optimizer with learning rate 1e-1, weight decay 5e-4, and momentum 0.9.

  3. Understand Gradient Boosting ensemble methods

    master

    Gradient Boosting trains base estimators sequentially. Each new estimator $h^m$ attempts to correct the errors of the previous estimators $h^1, dots, h^{m-1}$.

    Key Mechanics:

    • Output: The ensemble output is a weighted sum: $\mathbf{o}i = \sum{m=1}^M \epsilon \mathbf{o}_i^m$, where $\epsilon$ is the shrinkage rate (learning rate).
    • Learning Target: The target for the $m$-th estimator is the negative gradient of the loss function with respect to the accumulated output of previous estimators: $\mathbf{r}_i^m = - \frac{\partial\mathcal{L}(\mathbf{o}_i^{[:m]}, y_i)}{\partial\mathbf{o}_i^{[:m]}}$.
    • Fitting: The $m$-th estimator is fitted using least square regression against this target $\mathbf{r}_i^m$.

    Target Examples:

    • Regression (MSE): $\mathbf{r}_i^m = \mathbf{y}_i - \mathbf{o}_i^{[:m]}$
    • Classification (Cross-Entropy): $\mathbf{r}_i^m = \mathbf{y}_i - \text{Softmax}(\mathbf{o}_i^{[:m]})$ (where $\mathbf{y}_i$ is one-hot encoded).

  5. Understand Adversarial Training ensemble methods

    master

    Adversarial Training improves estimator robustness by training on both original and adversarial samples.

    In Ensemble-PyTorch, AdversarialTrainingClassifier and AdversarialTrainingRegressor contain $M$ independent base estimators (similar to Voting/Bagging). During training, for every sample $\mathbf{x}_i$, an adversarial sample $\mathbf{x}_i^{adv}$ is generated using the Fast Gradient Sign Method (FGSM). The loss function minimized is the sum of the loss on the original sample and the loss on the adversarial sample: $\mathcal{L}(\mathbf{o}_i, y_i) + \mathcal{L}(\mathbf{o}_i^{adv}, y_i)$.

  6. Understand Snapshot Ensemble methods

    master

    Snapshot Ensemble generates an ensemble by training a single base estimator to converge to different local minima $M$ times. At each local minimum, the parameters are saved as a "snapshot."

    To achieve this, it uses a cyclic annealing schedule on the learning rate. The learning rate $\alpha_t$ follows a cosine cycle based on the current iteration $t$ and the total training iterations $T$, ensuring the model traverses different regions of the loss surface.

  7. How Adversarial Training works in ensembles

    master

    Adversarial Training improves ensemble performance by treating adversarial samples (generated via the Fast Gradient Sign Method - FGSM) as augmented training data.

    Warning: Do not use AdversarialTrainingClassifier or AdversarialTrainingRegressor if your base estimator is under-fitted on the dataset, as it may further degrade performance.

    Available classes:

    • torchensemble.adversarial_training.AdversarialTrainingClassifier
    • torchensemble.adversarial_training.AdversarialTrainingRegressor

  9. Performance Insights: ResNet on CIFAR-10 and CIFAR-100

    master

    For more complex architectures like ResNet-18, the ensemble behavior shifts:

    ResNet on CIFAR-10 (Relatively Easy Dataset)

    • Single Estimator Performance: 94% - 95% accuracy.
    • Best Methods: Voting and Snapshot Ensemble are most effective. Snapshot ensemble is particularly efficient when considering training costs.

    ResNet on CIFAR-100 (Hard Dataset)

    • Single Estimator Performance: ~76% accuracy.
    • Best Method: Voting is the most effective.
    • Fusion: Performs poorly.
    • Gradient Boosting: Results were omitted in experiments due to prohibitively long training times.

  10. How Fusion-based ensemble methods work

    master

    In Fusion methods, predictions from all base estimators are aggregated as an average output first. The training loss is then computed based on this averaged output and the ground truth. This loss is back-propagated to all base estimators simultaneously.

    Available classes:

    • torchensemble.fusion.FusionClassifier
    • torchensemble.fusion.FusionRegressor

  11. How Neural Tree Ensembles work

    master

    Neural Tree Ensembles extend voting and gradient boosting by using neural trees as base estimators. Neural trees are differentiable trees that use logistic regression in internal nodes to split samples into child nodes with different probabilities.

    Available classes:

    • torchensemble.voting.NeuralForestClassifier
    • torchensemble.voting.NeuralForestRegressor

  12. How Gradient Boosting ensemble methods work

    master

    Gradient Boosting is a sequential ensemble method. At each iteration, a new base estimator is trained to fit the pseudo residual (calculated using ordinary least squares) between the ground truth and the output of the previously fitted estimators.

    Note on shrinkage_rate: In gradient_boosting classes, the shrinkage_rate argument is equivalent to the learning rate in libraries like XGBoost. This is distinct from the learning rate used by a deep learning parameter optimizer.

    Available classes:

    • torchensemble.gradient_boosting.GradientBoostingClassifier
    • torchensemble.gradient_boosting.GradientBoostingRegressor