DOCSEARCHbyAlgolia
Submit Docs

Captum: Model Interpretability for PyTorch

repository·master·Indexed 26 days ago

https://github.com/meta-pytorch/captum

Captum is a model interpretability and understanding library for PyTorch that provides state-of-the-art algorithms to help researchers and developers understand feature importance and model behavior. It implements a wide range of attribution methods, including gradient-based (Integrated Gradients, DeepLift, GradientSHAP), perturbation-based (Feature Ablation, Occlusion, Shapley Value), and hybrid wrappers like NoiseTunnel, as well as tools for analyzing neuron and layer importance via Conductance.

Tokens
19.9K
Snippets
24
Records
145
Agent score
90%

What's inside Captum

  1. Overview of Captum for model interpretability

    master

    Captum is an open source, extensible library for model interpretability built on PyTorch. It provides state-of-the-art algorithms, such as Integrated Gradients, to help identify which features contribute to a model's output.

    Key Use Cases:

    • Model Developers: Improve and troubleshoot models by identifying feature importance to design better architectures or investigate unexpected outputs.
    • Interpretability Researchers: Implement new algorithms that interact with PyTorch models and benchmark them against existing algorithms in the library.
    • Application Engineers: Troubleshoot production models and provide explanations to end users (e.g., explaining why a specific recommendation was made).

  2. Understand Captum attribution categories

    master

    Captum categorizes its interpretability methods into three distinct groups based on what they evaluate:

    • Primary Attribution: Evaluates the contribution of each individual input feature to the model's output.
    • Layer Attribution: Evaluates the contribution of each neuron within a specific layer to the model's output.
    • Neuron Attribution: Evaluates how much each input feature contributes to the activation of a specific hidden neuron.

    Additionally, Captum provides noise tunnel to smooth the results of any attribution method, and metrics (such as infidelity and sensitivity) to estimate the trustworthiness and goodness of model explanations.

  3. Use Neuron Attribution methods in Captum

    master

    Captum provides several attribution methods specifically designed for analyzing individual neurons. These methods allow you to attribute the activation of a specific neuron to its input features.

    Available Neuron Attribution classes:

    • captum.attr.NeuronGradient
    • captum.attr.NeuronIntegratedGradients
    • captum.attr.NeuronConductance
    • captum.attr.NeuronDeepLift
    • captum.attr.NeuronDeepLiftShap
    • captum.attr.NeuronGradientShap
    • captum.attr.NeuronGuidedBackprop
    • captum.attr.NeuronDeconvolution
    • captum.attr.NeuronFeatureAblation

  4. Apply Noise Tunnel to attribution methods

    master

    Noise Tunnel is a wrapper that can be applied on top of any attribution method to improve stability. It computes attribution multiple times by adding Gaussian noise to the input each time and combines the results using one of the following types:

    • Smoothgrad: Returns the mean of the sampled attributions (approximates smoothing with a Gaussian Kernel).
    • Smoothgrad Squared: Returns the mean of the squared sample attributions.
    • Vargrad: Returns the variance of the sample attributions.

  5. Use Guided Backpropagation and Deconvolution

    master

    Both methods compute gradients of the target output with respect to the input but override ReLU backpropagation to only allow non-negative gradients through:

    • Guided Backpropagation: Applies the ReLU function to the input gradients.
    • Deconvolution: Applies the ReLU function to the output gradients and backpropagates them directly.

    These are primarily used for convolutional networks.

  6. Guidelines for proposing new features and algorithms

    master

    When contributing new features or algorithms to Captum, ensure they meet the following criteria to maintain package quality and maintainability:

    • API Consistency: Maintain similarity with existing Captum APIs to ensure ease of use.
    • Documentation: Provide a summary of the algorithm, example usage, parameter/return descriptions, and known limitations.
    • Testing: Include explicit test cases for test models and support model wrappers like DataParallel, DistributedDataParallel, and JIT.
    • Benchmarking: Test on real models and datasets. Provide performance benchmarking (CPU/GPU runtime) and visual interpretation comparisons against state-of-the-art approaches.
    • Multimodality: Prefer generic algorithms that work across different architectures and input types.
    • Mathematical Soundness: Preference is given to methods that are axiomatic or have strong theoretical justification.

    Process:

    1. Discuss the idea on a GitHub issue (https://github.com/pytorch/captum/issues).
    2. If suitable, provide a design document or proposal explaining the API and structure.
    3. If the contribution is marginal, it may be directed to AWESOME_LIST.MD instead of the core library.

  7. Use Gradient SHAP for attribution

    master
    Gradient SHAP is a gradient-based method to approximate Shapley values. It works by adding Gaussian noise to input samples, selecting random points along the path between the baseline and input, and computing gradients at those points. The resulting attributions approximate the expected value of gradients multiplied by the difference between inputs and baselines, assuming feature independence and a linear explanation model.

  9. Handle functional non-linearities and reused modules

    master

    Most methods work with functional non-linearities (e.g., nn.functional.ReLU). However, methods that rely on hooks during back-propagation require specific configurations:

    • Methods requiring hooks: DeepLift, DeepLiftShap, Guided Backpropagation, and Deconvolution.
    • Requirement: You must use corresponding module activations (e.g., torch.nn.ReLU) initialized in the module constructor rather than functional calls.
    • DeepLift Restriction: Do not reuse modules in the forward function, as this interferes with the propagation of multipliers.

  10. Use DeepLIFT for attribution

    master

    DeepLIFT is a back-propagation based approach that attributes changes in output to changes in inputs relative to a reference (baseline). It uses 'multipliers' to assign blame to specific neurons for the difference in output.

    Note: Currently, Captum only supports the Rescale Rule of DeepLIFT algorithms. The RevealCancel rule is planned for future releases.

  11. Use Lime and LimeBase for attribution

    master

    LIME (Local Interpretable Model-agnostic Explanations) trains an interpretable surrogate model (like a linear model) by sampling data points around an input. Captum provides two variants:

    • LimeBase: A generic framework that returns a representation of the interpretable model (e.g., coefficients).
    • Lime: A specific implementation that provides a consistent API with other perturbation-based algorithms.

  12. Evaluate explanations with Infidelity and Sensitivity metrics

    master

    Use these metrics to quantitatively evaluate the quality of your model explanations:

    • Infidelity: Measures the mean squared error between model explanations in the magnitudes of input perturbations and the predictor function's changes to those perturbations. It is a generalized notion of Sensitivity-n.
    • Sensitivity: Measures how much an explanation changes in response to subtle input perturbations using Monte Carlo sampling. By default, it samples from a sub-space of an L-Infinity ball. Users can modify the radius of the ball and the sampling function.