DOCSEARCHbyAlgolia
Submit Docs

D2L.ai (Dive into Deep Learning)

repository·master·Indexed 12 days ago

https://github.com/d2l-ai/d2l-en

An interactive, multi-framework deep learning textbook delivered via Jupyter notebooks. It combines mathematical theory with practical, runnable code across PyTorch, TensorFlow, and MXNet to teach deep learning concepts to aspiring applied machine learning scientists.

Tokens
246.7K
Snippets
552
Records
810
Agent score
98%

What's inside D2L

  1. Overview of D2L.ai

    master
    D2L.ai (Dive into Deep Learning) is an interactive, open-source textbook designed to teach deep learning through hands-on practice. The book is composed of Jupyter notebooks that integrate mathematical theory, exposition figures, and runnable code across multiple frameworks. It is intended to serve as a technical resource for aspiring applied machine learning scientists.

  2. Overview of Attention Mechanisms and Transformers

    master

    The Transformer architecture has become the dominant model for nearly all Natural Language Processing (NLP) tasks, replacing traditional Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs). The core innovation is the attention mechanism, which allows a model to selectively focus on different parts of an input sequence at different decoding steps.

    Key concepts covered in this module include:

    • Attention Mechanisms: Moving from compressing inputs into fixed-length vectors to allowing decoders to revisit the entire input sequence.
    • Transformer Architecture: A model that dispenses with recurrent connections entirely, relying on attention to capture relationships between tokens.
    • Foundation Models: Large-scale Transformer-based models (like BERT, GPT, RoBERTa) that are pretrained on massive datasets and then fine-tuned for specific downstream tasks.
    • Vision Transformers (ViT): The application of Transformer architectures to computer vision tasks like image recognition and object detection.

  3. Overview of Single Shot Multibox Detection (SSD)

    master
    Single Shot Multibox Detection (SSD) is an object detection model designed to be simple and fast. It utilizes concepts such as bounding boxes, anchor boxes, and multiscale object detection to identify and locate objects within an image. While SSD is a specific architecture, the design principles and implementation details discussed in this section are applicable to many other object detection models.

  4. Summary of Machine Learning and Deep Learning concepts

    master

    A high-level summary of the core definitions:

    • Machine Learning: The study of how computer systems leverage experience (data) to improve performance at specific tasks, combining statistics, data mining, and optimization.
    • Representational Learning: A class of machine learning that focuses on automatically finding the appropriate way to represent data.
    • Deep Learning: Multi-level representation learning that uses many layers of transformations to replace both shallow models and labor-intensive manual feature engineering.
    • Drivers of Progress: The recent surge in deep learning is driven by the abundance of data (from sensors and the internet) and significant computational progress (primarily through GPUs) and efficient deep learning frameworks.

  5. What is Pointwise Mutual Information (PMI)?

    master

    Pointwise Mutual Information (PMI) measures how much more (or less) likely a specific combination of outcomes $(x, y)$ is compared to what would be expected if they were independent:

    $\textrm{pmi}(x, y) = \log\frac{p_{X, Y}(x, y)}{p_X(x) p_Y(y)}$

    • Large positive PMI: The outcomes occur together much more frequently than random chance.
    • Large negative PMI: The outcomes occur together far less frequently than expected by random chance.

    Mutual information can be interpreted as the average PMI.

  6. What is a Markov Decision Process (MDP)?

    master

    A Markov Decision Process (MDP) is a mathematical framework used to model how the state of a system evolves as different actions are applied. It is used to formulate reinforcement learning problems. An MDP is defined by the tuple $(\mathcal{S}, \mathcal{A}, T, r)$:

    • $\mathcal{S}$ (State Space): The set of all possible states the system can be in (e.g., the coordinates of a robot in a gridworld).
    • $\mathcal{A}$ (Action Space): The set of all actions available to the agent at each state (e.g., "move forward", "turn left").
    • $T$ (Transition Function): A function $T: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to [0,1]$ that defines the probability of reaching state $s'$ given the current state $s$ and action $a$. Mathematically, $T(s, a, s') = P(s' \mid s, a)$. It must satisfy $\sum_{s' \in \mathcal{S}} T(s, a, s') = 1$.
    • $r$ (Reward Function): A function $r: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ that provides a scalar reward when an agent takes action $a$ in state $s$. This is designed by the user to guide the agent toward a goal.

  7. What is Dropout and how does it work?

    master

    Dropout is a regularization technique used to prevent overfitting in neural networks by injecting noise during training. It involves zeroing out a fraction of neurons (nodes) in each layer during each training iteration. This prevents 'co-adaptation', where neurons rely too heavily on specific patterns from previous layers, forcing the network to learn more robust, redundant features.

    To ensure the expected value of the activations remains unchanged (unbiased), dropout uses a 'debiasing' step. If the dropout probability is $p$, each activation $h$ is replaced by $h'$:

    • $h' = 0$ with probability $p$
    • $h' = \frac{h}{1-p}$ otherwise

    Typically, dropout is disabled during testing/inference so that the full capacity of the network is used without normalization. Some researchers use dropout at test time to estimate prediction uncertainty.

  8. What is fine-tuning in transfer learning?

    master

    Fine-tuning is a transfer learning technique used to adapt a model trained on a large source dataset (e.g., ImageNet) to a smaller target dataset (e.g., a specific dataset of chairs).

    This approach is useful when:

    1. The target dataset is too small to train a complex model from scratch without overfitting.
    2. Collecting and labeling a massive dataset for the target task is too expensive or time-consuming.

    By using a pretrained source model, the target model inherits the ability to extract general image features like edges, textures, and shapes, which are applicable even if the specific objects in the target dataset differ from the source.

  9. What is deep learning and how does it differ from traditional machine learning?

    master

    Deep learning is a subset of machine learning focused on models based on many-layered neural networks. The term "deep" refers to the model's ability to learn many layers of transformations.

    Key differentiators from traditional machine learning include:

    • End-to-end training: Instead of assembling a system from individually tuned components (e.g., separate feature engineering and model building), deep learning builds a system and tunes its performance jointly. This replaces manual feature engineering (like Canny edge detectors or SIFT) with automatically tuned filters.
    • Representation learning: Deep learning focuses on automatically finding the appropriate way to represent data through multiple layers of computation.
    • Nonparametric models: As data becomes abundant, deep learning shifts from parametric statistical descriptions (which rely on simplifying assumptions) to nonparametric models that better fit the data.
    • Empiricism: The field has moved toward an empirical approach, embracing nonconvex nonlinear optimization and practical algorithmic progress, often prioritizing performance and scale.

  10. What is Backpropagation Through Time (BPTT)?

    master

    Backpropagation Through Time (BPTT) is the method used to apply backpropagation to Recurrent Neural Networks (RNNs).

    To perform BPTT, the computational graph of the RNN is 'unrolled' (or expanded) one time step at a time. This unrolled structure effectively becomes a feedforward neural network where the same parameters are repeated at every time step.

    Key characteristics:

    • Weight Tying: Because the same parameters appear at each time step, the gradient with respect to a parameter must be summed across all time steps where that parameter occurs.
    • Complexity: For long sequences (e.g., text with >1000 tokens), BPTT faces significant computational challenges (high memory usage) and optimization challenges (numerical instability like vanishing or exploding gradients).

  11. What is Yogi and when should I use it?

    master

    Adam can struggle to converge in certain scenarios where the second moment estimate $\mathbf{s}_t$ fluctuates wildly or updates are sparse, causing the variance estimate to lose track of past values too quickly.

    Yogi is a proposed improvement that modifies the update for the second moment $\mathbf{s}_t$ to be more stable. Instead of a standard EWMA, it uses the sign of the difference between the squared gradient and the current state to control the update magnitude:

    $$\mathbf{s}t \leftarrow \mathbf{s}{t-1} + (1 - \beta_2) \mathbf{g}_t^2 \odot \text{sgn}(\mathbf{g}t^2 - \mathbf{s}{t-1})$$

    This prevents the second moment from blowing up and provides better convergence in high-variance settings.

  12. What is a model in deep learning?

    master

    A model is the computational machinery that ingests data of one type and produces predictions of a possibly different type.

    Deep learning is a specific subset of machine learning characterized by models that consist of many successive transformations of the data chained together from top to bottom. This architecture allows for the processing of complex, high-dimensional data.