DOCSEARCHbyAlgolia
Submit Docs

lightly

repository·master·Indexed 25 days ago

https://github.com/lightly-ai/lightly

A deep learning package and computer vision framework specialized in self-supervised learning (SSL). Lightly provides a modular PyTorch-style API with building blocks for loss functions, projection heads, and augmentation pipelines. It supports a wide range of SSL models including SimCLR, BYOL, DINO, MAE, and ViT-based methods, with built-in support for distributed training via PyTorch Lightning.

Tokens
117.8K
Snippets
316
Records
388
Agent score
86%

What's inside lightly

  1. Overview of LightlySSL

    master
    LightlySSL is a computer vision framework designed for self-supervised learning (SSL). It provides tools for developing and training models without the need for extensive manual labeling. For users requiring advanced features like Docker support or pre-trained models for specific tasks (embedding, classification, detection, and segmentation), a commercial version is available via sales@lightly.ai.

  2. Overview of Lightly features

    master

    Lightly is a modular self-supervised learning framework designed for PyTorch users. Key features include:

    • Modular Architecture: Provides low-level building blocks such as loss functions and model heads.
    • PyTorch-style API: Designed to be intuitive for developers familiar with PyTorch.
    • Custom Backbones: Supports using custom backbone models for self-supervised pre-training.
    • Distributed Training: Built-in support for distributed training via PyTorch Lightning.

  3. Overview of Lightly SSL

    master

    Lightly SSL is a computer vision framework designed for self-supervised learning (SSL). It allows you to train deep learning models using unlabeled datasets, meaning no manual labels are required for the training process.

    Key characteristics:

    • Built on top of PyTorch.
    • Fully compatible with other frameworks like Fast.ai.
    • Designed to help users understand and work with large unlabeled datasets.

  4. Pixio implementation details and configuration

    master

    Pixio is a method based on Masked Autoencoders (MAE) adapted for dense prediction.

    Key Components:

    • Data Augmentations: Uses random resized cropping.
    • Masking: Masks 75% of patches using a coarse granularity. By default, it masks whole grid_size x grid_size blocks (4x4 by default) to prevent trivial reconstruction.
    • Backbone: A standard ViT with multiple class tokens (8 by default, configured via reg_tokens).
    • Decoder: A deep 32-block decoder for pixel reconstruction.
    • Reconstruction Loss: Mean Squared Error (MSE) loss between predicted and normalized pixel values.

    Configuration Notes:

    • Headline Configuration: Uses a 4x4 grid and 8 class tokens.
    • Dense-prediction-optimal Ablation: Uses a 2x2 grid and 4 class tokens.
    • Input Resolution: The reference model uses 256x256 resolution with a patch size of 16 (ensuring the 16x16 patch grid divides evenly into 4x4 blocks).

  5. Use self-supervised learning transforms in lightly.transforms

    master

    The lightly.transforms module provides specialized data augmentation transforms designed for various self-supervised learning (SSL) algorithms. Most transforms are implemented as callable classes that can be integrated into a PyTorch DataLoader.

    Available SSL-specific transforms include:

    • BYOL: byol_transform
    • DINO: dino_transform
    • DenseCL: densecl_transform
    • DetCon: detcon_transform
    • I-BoT: ibot_transform
    • MAE: mae_transform
    • MoCo: moco_transform
    • MSN: msn_transform
    • SimCLR: simclr_transform
    • SimSiam: simsiam_transform
    • SwAV: swav_transform
    • VICReg: vicreg_transform and vicregl_transform

    General purpose transforms available in this module include gaussian_blur, rotation, solarize, and jigsaw.

  6. Understand Lightly SSL core concepts

    master

    Lightly SSL is built around several interacting components used to perform self-supervised learning:

    • Dataset: Accessed via lightly.data.dataset.LightlyDataset. It can be initialized from image/video directories or directly from a torchvision dataset.
    • Transform: Used to create multiple views of an image. You can use lightly.transforms or custom augmentations.
    • Collate Function: Aggregates views into a single batch. Use the default or lightly.data.multi_view_collate.MultiViewCollate.
    • Dataloader: Standard PyTorch DataLoader, but must be passed a lightly.data.dataset.LightlyDataset.
    • Backbone Neural Network: The core architecture (e.g., ResNet, Vision Transformer) that extracts features.
    • Heads: Layers added on top of the backbone to project embeddings into a space where the loss is calculated. Available in lightly.models.modules.heads.
    • Model: A combination of a backbone, one or more heads, and optionally a momentum encoder.
    • Loss: The objective function, found in the lightly.loss module.
    • Optimizer: Any standard PyTorch optimizer.
    • Training: Can be performed using a standard PyTorch training loop or via PyTorch Lightning.
    • Image Embeddings: The compact representations/features learned by the model during training, useful for similarity tasks or data subsetting.
    • Pre-Trained Backbone: The trained backbone can be reused for downstream tasks like classification, object detection, or segmentation.

  8. LeJEPA Architecture and Components

    master

    LeJEPA is a self-supervised learning method that learns image representations by enforcing invariance between multiple augmented views while regularizing projected embeddings with SIGReg (Sketched Isotropic Gaussian Regularization).

    Key components include:

    • Multi-view projections: Uses a shared backbone and projection head for global and local (smaller) views.
    • Invariance loss: Uses mean-squared distance to pull local view projections toward the centroid of global view projections.
    • SIGReg: A sliced, Epps-Pulley based regularizer that drives projected features toward an isotropic Gaussian.
    • Projection head: A multi-layer perceptron with BatchNorm and ReLU (lightly.models.modules.LeJEPAProjectionHead) that maps backbone features into the projection space.

    Note that LeJEPA does not require negative samples, momentum encoders, or stop-gradients, making it different from methods like SimCLR, MoCo, DINO, or BYOL.

  9. Understand Masked Autoencoder (MAE) implementation details

    master

    MAE is a transformer-based self-supervised method that learns image representations by predicting pixel values of masked patches.

    Key technical characteristics:

    • Backbone: Uses a Vision Transformer (ViT). Note that the masking process is incompatible with convolutional-based architectures.
    • Masking Strategy: Applies masking to 75% of input patches (only 25% of tokens are fed to the encoder).
    • Data Augmentation: Minimally relies on handcrafted augmentations; typically only uses random resized cropping.
    • Loss Function: Uses Mean Squared Error (MSE) loss between original and reconstructed pixel values of the masked patches.
    • Evaluation Note: While strong in fine-tuning, MAE models may underperform in shallow evaluations like k-NN or linear evaluation with a frozen backbone.

  10. Use LightlyDataset for image and video data

    master

    The LightlyDataset class provides a uniform interface for creating image and video datasets. It supports all image formats compatible with Pillow (e.g., .jpg, .png, .tiff) and video formats via torchvision and PyAV (.mov, .mp4, .avi).

    Unlabeled Image Datasets

    Pass the path to a directory containing images. Each image is assigned a default label of 0.

    Labeled Image Datasets

    Organize images into subdirectories where each subdirectory name acts as the label. Pass the path to the parent directory to LightlyDataset.

    Video Datasets

    Pass the path to a directory containing video files. The dataset assigns each video frame its video as a label. To use video features, install the extra dependencies: pip install "lightly[video]".

    from lightly.data import LightlyDataset
from lightly.transforms import SimCLRTransform

# Unlabeled images
transform = SimCLRTransform()
dataset = LightlyDataset(input_dir='image_dir/', transform=transform)

# Labeled images (subdirectories = labels)
labeled_dataset = LightlyDataset(input_dir='labeled_image_dir/', transform=transform)

# Videos
video_dataset = LightlyDataset(input_dir='video_dir/', transform=transform)

  12. Perform a clean build of the HTML documentation

    master

    Because the make html-noplot command uses caching, some warnings might not appear after the initial build. To ensure a fresh build and catch all warnings, run a clean build periodically using:

    make clean-html-noplot