DOCSEARCHbyAlgolia
Submit Docs

PyTorch Tutorials

repository·main·Indexed 27 days ago

https://github.com/pytorch/tutorials

A collection of PyTorch tutorials, recipes, and prototype features. The repository includes documentation for beginner, intermediate, and advanced levels, as well as technical guides on C++ autograd operations and building custom SYCL operators using SyclExtension and BuildExtension.

Tokens
101.1K
Snippets
251
Records
505
Agent score
93%

What's inside pytorch-tutorials

  1. Overview of Distributed RPC Framework

    main

    The torch.distributed.rpc package provides tools for building distributed training applications that go beyond the standard DistributedDataParallel (DDP) paradigm. While DDP is designed for replicating models across processes to handle split data, the RPC framework is suitable for more complex scenarios such as:

    1. Reinforcement Learning: Spawning multiple parallel observers that share a single agent, requiring data exchange between observers and the trainer.
    2. Large Model Training: Splitting models across multiple machines that cannot fit on a single GPU.
    3. Parameter Server Architectures: Implementing frameworks where model parameters and trainers reside on different machines.

    Key components of the framework include:

    • RPC and RRef: For sending data between workers and referencing remote data objects.
    • distributed autograd: For executing backward passes in a distributed setting.
    • distributed optimizer: For performing optimizer steps across distributed components.

  2. Overview of PyTorch Distributed Training Methods

    main

    PyTorch provides several methods for distributed training to spread workloads across multiple worker nodes. Choose a method based on your model size and compute requirements:

    • DistributedDataParallel (DDP): Standard approach for data parallelism.
    • Fully Sharded Data Parallel (FSDP2): Shards model parameters, gradients, and optimizer states.
    • Tensor Parallel (TP): Splits individual tensors across multiple GPUs.
    • Device Mesh: Provides a way to manage complex multi-dimensional distributed topologies.
    • Remote Procedure Call (RPC): Enables distributed training via RPC-based communication.
    • Monarch Framework: Uses an actor framework for interactive distributed applications.
    • Custom Extensions: Allows implementing custom ProcessGroup backends using C++ extensions.

  3. Overview of Distributed Checkpoint (DCP)

    main

    PyTorch Distributed Checkpointing (DCP) via torch.distributed.checkpoint enables saving and loading models from multiple ranks in parallel. It is designed to handle scenarios where the number of trainers changes when resuming training by allowing re-sharding across different cluster topologies at load time.

    Key differences from torch.save and torch.load:

    • Multiple Files: Produces multiple files per checkpoint (at least one per rank).
    • In-place Operation: DCP uses pre-allocated storage from the model to load data.
    • Stateful Support: Automatically calls state_dict and load_state_dict methods on objects implementing the torch.distributed.checkpoint.stateful.Stateful protocol.

  4. Access Deep Learning for NLP tutorials

    main

    The beginner_source/nlp tutorials provide a path for developers new to deep learning frameworks to learn NLP programming using PyTorch. These tutorials focus on model implementation rather than data processing, using small-scale synthetic data to demonstrate weight changes during training.

    Prerequisites:

    • Working knowledge of core NLP problems (e.g., part-of-speech tagging, language modeling).
    • Familiarity with neural networks and the backpropagation algorithm (e.g., knowledge of linearities and non-linearities).

  5. Understand Fully Sharded Data Parallel (FSDP) vs DDP

    main

    Comparison of DDP and FSDP

    • DistributedDataParallel (DDP): Each process/worker owns a full replica of the model. Model weights and optimizer states are replicated across all workers. It uses all-reduce to sum gradients.
    • Fully Sharded Data Parallel (FSDP): Shards model parameters, optimizer states, and gradients across all DDP ranks. This reduces the GPU memory footprint per worker, enabling larger models or larger batch sizes, at the cost of increased communication volume.

    FSDP Workflow

    FSDP manages memory by collecting and discarding shards during the training pass:

    1. Constructor: Shards model parameters; each rank keeps only its own shard.
    2. Forward Path:
      • Performs all_gather to collect shards from all ranks to recover the full parameter for the current FSDP unit.
      • Executes forward computation.
      • Discards the collected parameter shards to free memory.
    3. Backward Path:
      • Performs all_gather to recover the full parameter for the current FSDP unit.
      • Executes backward computation.
      • Performs reduce_scatter to sync gradients.
      • Discards parameters.

  6. Use PyTorch Parallelism Modules

    main

    PyTorch provides high-level parallelism modules that compose with existing models:

    • DistributedDataParallel (DDP): For standard data parallelism where the model is replicated on every process.
    • Fully Sharded Data-Parallel Training (FSDP2): For sharding model parameters, gradients, and optimizer states.
    • Tensor Parallel (TP): For splitting individual tensors across multiple devices.
    • Pipeline Parallel (PP): For splitting model layers across different devices.

  7. Understand how PyTorch Tensor Parallel works

    main

    PyTorch Tensor Parallel (TP) and its variant Sequence Parallel (SP) enable efficient training of large-scale Transformer models by sharding computations.

    Core Concepts

    • Tensor Parallel (TP): Shards matrix multiplications in attention and MLP layers (based on Megatron-LM) to distribute compute and memory.
    • Sequence Parallel (SP): A variant of TP that shards the sequence dimension for nn.LayerNorm or RMSNorm layers to reduce activation memory bottlenecks.

    High-Level Workflow

    1. Sharding Initialization:
      • Use parallelize_module to determine the ParallelStyle for each layer.
      • This swaps module parameters to DTensors, which handle sharded computation.
    2. Runtime Forward/Backward:
      • The system automatically runs communication operations (e.g., allreduce, allgather, reduce_scatter) to transform DTensor layouts based on the specified ParallelStyle.
      • Sharded computations are executed for layers like nn.Linear and nn.Embedding to optimize memory and compute.

  8. Learn about ONNX integration in PyTorch

    main

    The PyTorch ONNX tutorials provide guidance on exporting PyTorch models to the Open Neural Network Exchange (ONNX) format. Available tutorials include:

    • Introduction to ONNX: General overview and concepts.
    • Exporting a PyTorch model to ONNX: Basic workflow for simple model exports.
    • Extending the ONNX exporter operator support: How to use the ONNX registry to add support for custom operators.
    • Export a model with control flow to ONNX: Techniques for handling models that contain control flow (e.g., if-statements, loops).

  9. Create Custom Python Operators in PyTorch

    main

    Use the custom operator API to wrap arbitrary Python functions so they behave like native PyTorch operators. This is useful for treating a function as an opaque callable for torch.compile or torch.export, or for adding training support to a Python function.

    Note: If your operation can be expressed as a composition of existing PyTorch operators, you should generally use those instead of creating a custom operator.

    Every custom operator requires:

    1. A stable schema and mutation/aliasing contract.
    2. Validation using torch.library.opcheck.
    3. A fake kernel if the operator returns tensors and must work with torch.compile or torch.export.

  10. Access Intermediate PyTorch Tutorials

    main
    The intermediate tutorials repository provides Python scripts and documentation for advanced PyTorch topics including RNNs, Sequence to Sequence networks, Reinforcement Learning, Distributed Applications, and Spatial Transformer Networks. You can access the full web-based versions of these tutorials at the official PyTorch tutorials website.

  11. Explore PyTorch Recipes

    main
    PyTorch Recipes are bite-sized, actionable examples designed to demonstrate specific PyTorch features. They differ from full-length tutorials by focusing on single, practical tasks such as defining a neural network, using the profiler, or optimizing model performance.

  12. Understand and access Prototype Tutorials and Recipes

    main

    The Prototype Tutorials and Recipes directory contains demonstrations of prototype features in PyTorch.

    Important Considerations:

    • Availability: Prototype features are included in standard binary distributions (PyPI, Conda) but are provided as a technical preview and may change.
    • Production Warning: The PyTorch team does not recommend using prototype features in production pipelines.
    • Accessing Features: Depending on the specific feature, you may need to build from the master branch or use nightly wheels available at pytorch.org. You can also use release wheels from PyPI or Conda.
    • Visibility: These tutorials are intentionally excluded from the official pytorch.org/tutorials website build.