DOCSEARCHbyAlgolia
Submit Docs

Coordinate Attention (CA) PyTorch Implementation

repository·main·Indexed 22 days ago

https://github.com/houqb/coordattention

A PyTorch implementation of Coordinate Attention (CA), a lightweight attention mechanism for efficient mobile network design. Designed as a plug-and-play module for architectures like MobileNetV2 and MobileNeXt to augment feature representations. Includes guidance on integration patterns, activation functions, and performance benchmarks for semantic segmentation tasks on Pascal VOC 2012 and Cityscapes.

Tokens
588
Snippets
0
Records
4
Agent score
28%

What's inside coordattention

  1. Coordinate Attention performance in Semantic Segmentation

    main

    Coordinate Attention (CA) provides significant improvements in semantic segmentation tasks compared to ImageNet classification or object detection. This is attributed to CA's ability to capture long-range dependencies with precise positional information, which is highly beneficial for dense prediction tasks.

    Observed Improvements (Pascal VOC 2012 val):

    • MobileNetV2 + CA (Stride 16): 73.32 mIoU (vs 70.84 for baseline)
    • MobileNetV2 + CA (Stride 8): 73.96 mIoU (vs 71.82 for baseline)

    Observed Improvements (Cityscapes val):

    • MobileNetV2 + CA (Stride 8): 74.0 mIoU (vs 71.4 for baseline)

  2. Integrate Coordinate Attention (CA) into existing architectures

    main

    Coordinate Attention (CA) is designed as a feature representation augmentation tool that can be easily plugged into classic building blocks.

    Common integration patterns include:

    • Inverted Residual Blocks (e.g., MobileNetV2): Plug the CA block into the residual structure to enhance feature representation.
    • Sandglass Blocks (e.g., MobileNeXt): Integrate the CA block within the sandglass architecture.

    When applying CA to MobileNeXt, it is observed that adding the attention block after the first depthwise 3x3 convolution yields better performance.

  4. Tips for designing lightweight attention blocks

    main

    When implementing or designing lightweight attention modules using Coordinate Attention, consider the following best practices:

    • Activation Function: Using SiLU activation (referred to as h_swish in the implementation) generally performs better than ReLU6.
    • Directional Attention: Applying attention in either the horizontal or vertical direction alone performs similarly to Squeeze-and-Excitation (SE) attention.
    • MobileNeXt Optimization: For MobileNeXt architectures, placing the attention block after the first depthwise 3x3 convolution is more effective.
    • Softmax Consideration: There is an experimental consideration regarding whether applying a softmax between horizontal and vertical features improves results.