pytorch-grad-cam

To use ICD augmentation efficiently, you should precompute saliency maps using XAICache to avoid re-calculating Grad-CAM during every training step.

Requirements:

• Your train_loader must yield tuples in the format (image, label, index) so the cache can correctly map saliency maps to specific samples.
• The icd.apply_batch_with_labels method expects uint8 HWC (Height, Width, Channels) input images.

Workflow:

1. Precompute: Use cache.precompute_cache to perform a single pass over the training set.
2. Initialize: Create the ICD object with the model, target layers, and the precomputed cache.
3. Apply: Use icd.apply_batch_with_labels within your training loop.

from pytorch_grad_cam import GradCAM
from bnnr.icd import ICD
from bnnr.xai_cache import XAICache

# 1. Precompute saliency (one pass over the training set)
cache = XAICache("./xai_cache")
cache.precompute_cache(
    model=model,
    train_loader=train_loader,      # yields (image, label, index)
    target_layers=[model.layer4[-1]],
    n_samples=len(train_dataset),
    method="gradcam",
)

# 2. Create the augmentation
icd = ICD(
    model=model,
    target_layers=[model.layer4[-1]],
    cache=cache,
    explainer="gradcam",
    tile_size=16,
    threshold_percentile=50.0,
    fill_strategy="gaussian_blur",
)

# 3. Apply in the training loop (uint8 HWC input)
augmented = icd.apply_batch_with_labels(images_u8, labels, sample_indices=indices)

Vision Transformers (ViT) typically produce outputs in the shape BATCH x N x C (e.g., BATCH x 197 x 192), where the first element is the class token and the remaining elements are spatial patches. To use GradCAM, you must provide a reshape_transform function to the GradCAM constructor to convert these patch sequences into 2D spatial images with channels in the first dimension (similar to CNNs).

Target Layer Selection: Do not select the very last layer if the classification is performed solely on the class token, as the gradients for the spatial patches in that layer will be zero. Instead, choose a layer from a preceding attention block, such as model.blocks[-1].norm1.

# Example reshape_transform for ViT
def reshape_transform(tensor, height=14, width=14):
    # tensor shape: [batch, 197, 192]
    # Remove the class token (index 0) and reshape patches to 2D
    result = tensor[:, 1:, :].reshape(tensor.size(0), height, width, tensor.size(2))

    # Transpose to bring channels to the first dimension: [batch, channels, height, width]
    result = result.transpose(2, 3).transpose(1, 2)
    return result

# Initialize GradCAM with the transform
GradCAM(model=model, target_layers=target_layers, reshape_transform=reshape_transform)

# Recommended target layer
target_layers = [model.blocks[-1].norm1]

Swin Transformers typically produce outputs in the shape BATCH x N x C (e.g., BATCH x 49 x 1024). Unlike ViT, Swin Transformers do not use a dedicated cls_token; therefore, all elements in the sequence represent spatial information.

To use GradCAM, provide a reshape_transform function to the GradCAM constructor to reshape the sequence into a 2D spatial grid and move the channels to the first dimension.

Target Layer Selection: Select a layer from the last block of the last layer, for example: model.layers[-1].blocks[-1].norm1.

# Example reshape_transform for Swin Transformer
def reshape_transform(tensor, height=7, width=7):
    # tensor shape: [batch, 49, 1024]
    result = tensor.reshape(tensor.size(0), height, width, tensor.size(2))

    # Transpose to bring channels to the first dimension: [batch, channels, height, width]
    result = result.transpose(2, 3).transpose(1, 2)
    return result

# Initialize GradCAM with the transform
GradCAM(model=model, target_layers=target_layers, reshape_transform=reshape_transform)

# Recommended target layer
target_layers = [model.layers[-1].blocks[-1].norm1]

Models like Faster R-CNN use a Feature Pyramid Network (FPN) that outputs an OrderedDict of tensors with different spatial sizes. To use CAM, you must implement a reshape_transform that aggregates these tensors into a single consistent shape.

Key implementation details:

• Use torch.nn.functional.interpolate to resize all feature maps to a common target_size (usually the size of the largest/pool layer).
• Use torch.abs() on activations, as FPN activations can be un-bounded and contain negative values.
• Concatenate the resized activations along the channel axis.

def fasterrcnn_reshape_transform(x):
    target_size = x['pool'].size()[-2 : ]
    activations = []
    for key, value in x.items():
        activations.append(torch.nn.functional.interpolate(torch.abs(value), target_size, mode='bilinear'))
    activations = torch.cat(activations, axis=1)
    return activations

To apply Class Activation Maps (CAM) to object detection models like Faster R-CNN, you should use gradient-free methods because object detection outputs (bounding boxes, labels, scores) are typically non-differentiable.

Two primary methods are available:

1. EigenCAM: Extremely fast (requires only one forward pass) but lacks class discrimination (it highlights dominant objects regardless of the target).
2. AblationCAM: A state-of-the-art method that provides better class discrimination by measuring how the output changes when activations are ablated, but it is significantly slower.

To use these with Faster R-CNN, you must provide a reshape_transform to handle the Feature Pyramid Network (FPN) output and a custom target to define what you want to maximize (e.g., bounding box scores or IoU).

from pytorch_grad_cam import AblationCAM, EigenCAM
from pytorch_grad_cam.ablation_layer import AblationLayerFasterRCNN
from pytorch_grad_cam.utils.model_targets import FasterRCNNBoxScoreTarget
from pytorch_grad_cam.utils.reshape_transforms import fasterrcnn_reshape_transform
from pytorch_grad_cam.utils.image import show_cam_on_image

# For AblationCAM (Slower, better discrimination)
targets = [FasterRCNNBoxScoreTarget(labels=labels, bounding_boxes=boxes)]
target_layers = [model.backbone]
cam = AblationCAM(model,
                  target_layers, 
                  use_cuda=torch.cuda.is_available(), 
                  reshape_transform=fasterrcnn_reshape_transform,
                  ablation_layer=AblationLayerFasterRCNN(),
                  ratio_channels_to_ablate=1.0)

# For EigenCAM (Very fast alternative)
cam = EigenCAM(model,
              target_layers, 
              use_cuda=torch.cuda.is_available(), 
              reshape_transform=fasterrcnn_reshape_transform)