DOCSEARCHbyAlgolia
Submit Docs

lisenjie757 Knowledge Base

website·Indexed 30 days ago

https://lisenjie757.github.io

A collection of technical notes and deep dives into deep learning and mathematics. Features detailed paper reviews of GNN, GCN, GAN, ResNet, AlexNet, and Transformer, as well as a PyTorch introductory series covering tensor operations, data preprocessing, neural network construction, and autograd. Includes foundational materials on probability theory, information theory, and model compression.

Tokens
11.9K
Snippets
51
Records
83
Agent score
71%

What's inside lisenjie757 Knowledge Base

  1. Overview of the Transformer Architecture

    The Transformer is a neural network architecture based entirely on attention mechanisms, replacing recurrent (RNN) and convolutional (CNN) layers. It uses an encoder-decoder structure to map an input sequence to a continuous representation and then generate an output sequence autoregressively. Key advantages include massive parallelization and reduced training time compared to sequential models.

  2. Overview of Neural Network Model Compression Techniques

    Model compression aims to reduce the parameter count and computational requirements of Deep Neural Networks (DNNs) to enable deployment on resource-constrained edge devices. The primary techniques are categorized into four types:

    1. Compact Models: Designing inherently efficient network structures (e.g., MobileNet, ShuffleNet, SqueezeNet).
    2. Tensor Decomposition: Reducing parameters and computation by decomposing tensors (e.g., CP decomposition, Tucker decomposition, TT decomposition).
    3. Data Quantization: Reducing the precision of weights and activations to lower memory and compute costs (e.g., INT8, INT4, INT2, Binary Neural Networks).
    4. Network Sparsification: Removing redundant weights or activations to create sparse representations (e.g., Pruning, Sparse Matrices, Sparse Matrix Decomposition).

  3. Overview of the Transformer Architecture

    The Transformer is a neural network architecture based entirely on attention mechanisms, replacing recurrent (RNN) and convolutional (CNN) layers. It uses an encoder-decoder structure to map an input sequence to a continuous representation and then generate an output sequence autoregressively. The architecture is highly parallelizable and effective for sequence-to-sequence tasks like machine translation.

  4. Implement Positional Encoding

    Since Transformers lack recurrence or convolution, they require positional encoding to inject information about the relative or absolute position of tokens in the sequence. The encoding has the same dimension as the word embeddings (d_model) and is added to them. It uses sine and cosine functions of different frequencies.
    PE_{(pos, 2i)} = sin(pos / 10000^{2i/d_{model}})
PE_{(pos, 2i+1)} = cos(pos / 10000^{2i/d_{model}})

  5. Understand the motivation for Residual Networks (ResNet)

    ResNet addresses the degradation problem in deep Convolutional Neural Networks (CNNs), where increasing network depth beyond a certain point leads to higher training error rather than improved accuracy. This is caused by the difficulty of optimizers to learn an identity mapping, causing outputs to drift from the optimal solution as depth increases. ResNet solves this by introducing skip connections (residual connections) that pass input information from earlier layers directly to deeper layers. This allows the network to more easily fit identity mappings and prevents the 'forgetting' of low-level feature information from shallow layers, enabling the training of significantly deeper networks without gradient vanishing or explosion.

  6. Inspect PyTorch model structure and parameters

    In PyTorch, you can inspect the architecture of a neural network by printing the model object. This reveals the sequence of layers and their configurations (e.g., in_features and out_features for nn.Linear). To access specific weights or biases of a layer within a nn.Sequential block, use dot notation combined with the index of the layer (e.g., model.linear_relu_stack[0].weight).
    # Example of a model structure output
NeuralNetwork(
  (flatten): Flatten(start_dim=1, end_dim=-1)
  (linear_relu_stack): Sequential(
    (0): Linear(in_features=784, out_features=512, bias=True)
    (1): ReLU()
    (2): Linear(in_features=512, out_features=512, bias=True)
    (3): ReLU()
    (4): Linear(in_features=512, out_features=10, bias=True)
  )
)

# Accessing parameters
print(model.linear_relu_stack[0].weight)
print(model.linear_relu_stack[0].bias)

  7. Apply regularization techniques for Transformer training

    To improve robustness and prevent overfitting, the Transformer architecture employs two primary regularization methods: Residual Dropout and Label Smoothing.
    {
  "residual_dropout": {
    "probability": 0.1,
    "application": "Applied to the output of each sub-layer before residual connection and layer normalization"
  },
  "label_smoothing": {
    "epsilon_ls": 0.1,
    "purpose": "Increases robustness against noise in training data"
  }
}

  8. Use Bottleneck architecture for deep ResNets (50+ layers)

    To reduce computational complexity and parameter count in networks with 50 or more layers, ResNet uses a 'Bottleneck' design. This consists of three layers:

    1. A $1\times1$ convolution to reduce the dimension.
    2. A $3\times3$ convolution (the bottleneck).
    3. A $1\times1$ convolution to restore the original dimension.

  9. Perform partial weight loading for transfer learning (Warm Start)

    When performing transfer learning or loading weights into a model with a slightly different architecture, use strict=False in load_state_dict(). This tells PyTorch to ignore parameters that do not match between the saved state dictionary and the current model structure.
    import torch

# Save weights
torch.save(model.state_dict(), PATH)

# Load weights ignoring non-matching layers
model = Net()
model.load_state_dict(torch.load(PATH), strict=False)

  10. AlexNet Architecture Overview

    AlexNet is a deep convolutional neural network consisting of 8 weighted layers: 5 convolutional layers followed by 3 fully connected layers, ending with a Softmax layer for 1000-class classification. Key architectural features include:

    • ReLU Activation: Used instead of tanh to significantly increase training speed.
    • Overlapping Pooling: Max pooling where the stride is smaller than the pooling window size, which helps reduce error rates.
    • Local Response Normalization (LRN): Applied to the 1st and 2nd layers to improve generalization (though largely replaced by Batch Normalization in modern networks).
    • GPU Parallelization: Designed to be trained across two GPUs, with kernels in layers 2, 4, and 5 only connecting to kernels on the same GPU from the previous layer. Layer 3 and the fully connected layers connect to all kernels/neurons from the previous layer.

  11. Understand the Generative Adversarial Network (GAN) Architecture

    A GAN consists of two neural networks trained simultaneously in an adversarial framework:

    1. Generator (G): Takes a random noise variable $z$ (sampled from $p_z(z)$) as input and attempts to map it to a data point $x$ that mimics the real data distribution $p_{data}(x)$. Its goal is to maximize the probability that the discriminator makes a mistake.
    2. Discriminator (D): Takes a data point $x$ as input and outputs the probability that the data came from the real training set rather than the generator. Its goal is to accurately distinguish between real and synthetic data.

    At convergence, the generator recovers the training data distribution, and the discriminator's probability of distinguishing real from fake becomes 50% ($D(x) = 1/2$).

  12. Implement regularization in Transformer models

    To improve robustness and prevent overfitting, the Transformer architecture employs two primary regularization methods: Residual Dropout and Label Smoothing.
    {
  "residual_dropout": 0.1, // Applied to output of each sub-layer before residual connection and layer norm
  "label_smoothing": 0.1   // Used to make the model more robust to noise in training data
}