labmlai Annotated Deep Learning Paper Implementations

The Feedback Transformer is a PyTorch implementation of the paper 'Accessing Higher-level Representations in Sequential Transformers with Feedback Memory'.

Unlike standard transformers that process tokens in parallel by attending only to the outputs of the previous layer, the Feedback Transformer introduces recurrence by allowing each layer to attend to the outputs of all previous layers from previous steps.

Key Characteristics:

• Training: Significantly slower (5X - 10X) than parallel transformers because tokens must be processed sequentially.
• Inference: Faster during prediction if memory vectors are cached, as the next token can be predicted efficiently.
• Memory Optimization: The implementation can use a weighted sum of outputs from all layers instead of keeping every layer's output, reducing memory usage during caching.

This repository provides a PyTorch implementation of the paper Deep Residual Learning for Image Recognition.

ResNets are designed to solve the degradation problem, where the accuracy of deep neural networks begins to degrade as the number of layers becomes very high. Instead of training layers to learn direct mappings, ResNets train layers as residual functions. This allows for much deeper architectures by ensuring that accuracy increases with depth before eventually saturating.

The labml_nn.optimizers module provides annotated implementations of various deep learning optimizers. You can use these to update model parameters during training. Available implementations include:

• Adam Optimizer: Standard Adam implementation.
• AMSGrad Optimizer: A variant of Adam that addresses convergence issues.
• Adam Optimizer with warmup: Adam with a learning rate warmup schedule.
• Noam Optimizer: Often used in Transformer architectures.
• Rectified Adam (RAdam) Optimizer: Provides a dynamic rectifier to adjust variance of the adaptive learning rate.
• AdaBelief Optimizer: Adapts step size according to the belief in the current gradient.
• Sophia-G Optimizer: A second-order optimizer designed for large-scale training.

WGAN-GP is an implementation of the paper Improved Training of Wasserstein GANs.

In standard Wasserstein GANs (WGAN), the Lipschitz constraint on the discriminator (critic) is typically enforced via weight clipping. However, weight clipping (including L2 norm clipping, weight normalization, or L1/L2 weight decay) can lead to two major issues:

1. Limiting Discriminator Capacity: It restricts the expressive power of the critic.
2. Gradient Instability: It can cause exploding or vanishing gradients, especially when Batch Normalization is not used.

WGAN-GP solves these issues by using a gradient penalty to enforce the Lipschitz constraint, providing a more stable training process without the drawbacks of weight clipping.

This repository provides a PyTorch implementation of the paper 'Playing Atari with Deep Reinforcement Learning'. The implementation includes several advanced reinforcement learning components that can be used together or separately:

• Dueling Network: An architecture modification to the Q-network.
• Prioritized Replay: An enhanced replay buffer mechanism.
• Double Q Network: A method to reduce overestimation bias in Q-learning.

For a complete walkthrough of how these components are integrated into a training loop, you can refer to the provided experiment notebook.

# Reference to the experiment notebook for a full implementation walkthrough
# https://colab.research.google.com/assets/colab-badge.svg

This implementation of Eleuther GPT-NeoX provides tools for model definition, tokenization, checkpoint management, and various usage samples.

Key components include:

• Model definition: The core architecture implementation.
• Tokenizer: Tools for converting text to tokens.
• Checkpoint helpers: Utilities for downloading and loading model checkpoints.
• Utilities: General helper functions.

Common tasks include:

• Generating text: Using the model for inference.
• Fine-tuning: Specifically, fine-tuning biases using pipeline-parallelism.
• Evaluation: Running evaluations, including half-precision models on a single GPU.

This repository provides a PyTorch implementation of Instance Normalization, as described in the paper "Instance Normalization: The Missing Ingredient for Fast Stylization".

Instance normalization is designed to improve style transfer tasks by performing contrast normalization. It addresses the difficulty convolutional networks face when trying to learn contrast normalization independently, ensuring that stylization results are not dependent on the contrast of the input content image.

Graph Attention Networks (GAT) are a PyTorch implementation of the paper Graph Attention Networks. GATs are designed to operate on graph data, where nodes represent entities (e.g., research papers in the Cora dataset) and edges represent connections (e.g., citations).

Key technical concepts:

• Masked Self-Attention: GAT utilizes a mechanism similar to Transformers to allow node embeddings to attend to the embeddings of their connected neighbors.
• Layered Architecture: The model consists of stacked graph attention layers. Each layer takes node embeddings as input and produces transformed embeddings as output.