torchvision

The torchvision package provides essential building blocks for computer vision tasks, organized into several key modules:

• transforms: Common image transformations for data augmentation and preprocessing.
• tv_tensors: Specialized tensor types for vision tasks.
• models: Pre-defined model architectures (including pre-trained models).
• datasets: Popular computer vision datasets.
• utils: Utility functions for common tasks.
• ops: Vision-specific operators.
• io: Input/Output operations for images and videos.
• feature_extraction: Tools for extracting features from models.

Similarity learning using triplet loss is a technique used to learn embeddings that differentiate images by measuring distance (e.g., Euclidean distance). This is useful when you have an unknown number of classes or want to learn a distance-based metric between samples, such as face recognition.

In this paradigm, embeddings of the same class should be 'close' to each other, while embeddings of different classes should be 'far' apart.

The libtorchvision library includes custom ops and C++ APIs. These APIs do not come with backward-compatibility guarantees and may change between versions.

Recommendation: For production environments requiring stability, avoid using the C++ APIs directly. Instead, export your models via TorchScript from the Python API, which provides stable backward-compatibility guarantees.

TVTensors are subclasses of torch.Tensor used by torchvision.transforms.v2 to automatically dispatch inputs to the correct lower-level kernels.

Key takeaway: Most users do not need to manipulate TVTensors directly. Instead, you should use the torchvision.transforms.v2 API, which handles these specialized tensor types under the hood to ensure that operations like cropping or flipping are applied correctly to images, bounding boxes, masks, and keypoints simultaneously.

The torchvision.ops module includes operators for pre-processing and post-processing in object detection and segmentation workflows.

Functions:

• batched_nms: Non-maximum suppression for batches.
• masks_to_boxes: Converts segmentation masks to bounding boxes.
• nms: Non-maximum suppression.
• roi_align: Region of Interest (RoI) Align.
• roi_pool: Region of Interest (RoI) Pooling.
• ps_roi_align: Pyramid, Scale, and RoI Align.
• ps_roi_pool: Pyramid, Scale, and RoI Pooling.

Classes:

• FeaturePyramidNetwork: Implements a Feature Pyramid Network (FPN).
• MultiScaleRoIAlign: Multi-scale RoI Align.
• RoIAlign: RoI Align layer.
• RoIPool: RoI Pooling layer.
• PSRoIAlign: PSRoIAlign layer.
• PSRoIPool: PSRoIPool layer.

Unlike standard transforms that operate on individual images, v2.CutMix and v2.MixUp are designed to be applied to entire batches. This is because they combine pairs of images together. You should use these transforms either after the DataLoader (once samples are batched) or as part of a custom collation function.

# Example conceptual usage
# These are applied to batches, not single images
transforms = v2.Compose([
    v2.CutMix(),
    v2.MixUp()
])

batch = transforms(images, labels)

Features in torchvision are classified into three release statuses to help you manage expectations regarding stability and compatibility:

• Stable: Maintained long-term with high documentation coverage and no major performance gaps. Backwards compatibility is generally maintained, though breaking changes may occur with one release's notice.
• Beta: APIs may change based on feedback, performance may still be improving, or operator coverage may be incomplete. While the project commits to moving these to Stable, backwards compatibility is not guaranteed.
• Prototype: Early-stage features for testing and feedback. These are typically not included in standard binary distributions (like PyPI or Conda) and may require run-time flags to access.

Torchvision provides two ways to apply transformations, similar to the torch.nn and torch.nn.functional pattern:

1. Transform Classes (e.g., v2.Resize): These are stateful objects typically used within a v2.Compose pipeline. Random transforms (like v2.RandomCrop) sample parameters automatically each time they are called.
2. Functional APIs (e.g., v2.functional.resize): These are stateless functions in the torchvision.transforms.v2.functional namespace. They are useful for manual control.

Note on Randomness: Because functionals do not perform random sampling themselves, you must use the .get_params() method of the corresponding transform class to sample parameters if you want to apply the same random transformation to multiple inputs (e.g., an image and its bounding box) manually.

Torchvision follows the PyTorch policy regarding breaking changes to ensure stability for users.

• Deprecation Period: Breaking changes require a deprecation period of at least 2 versions.
• Usage: Deprecations should be used sparingly due to their disruptive nature.
• Documentation: All deprecations must clearly indicate their deadline in both the documentation and the warning messages emitted during runtime.

torchvision supports the following image backends for transformations and processing:

• torch tensors
• PIL images:
  • Pillow
  • Pillow-SIMD: A faster drop-in replacement for Pillow using SIMD instructions.

For detailed information on how to use these backends with transforms, refer to the transforms documentation.