DOCSEARCHbyAlgolia
Submit Docs

DeepSpeed

repository·master·Indexed 12 days ago

https://github.com/deepspeedai/deepspeed

A deep learning optimization library designed for extreme speed and scale in training large-scale models. It features innovations such as ZeRO, 3D-Parallelism, MoE support, and DeepCompile—a compiler-level framework that automatically transforms single-GPU models into optimized multi-GPU training graphs. DeepSpeed provides advanced communication optimizations for AllReduce and AlltoAll, supports PyTorch-compatible backward APIs, and offers low-precision master parameter configurations to reduce memory usage.

Tokens
179.3K
Snippets
468
Records
820
Agent score
98%

What's inside DeepSpeed

  1. Overview of DeepSpeed-VisualChat

    master

    DeepSpeed-VisualChat is an open-source framework designed for multi-round, multi-image interleaved chat experiences. Unlike existing Large Vision-Language Models (LVLMs) that focus on single-image tasks or simultaneous multi-image inputs, DeepSpeed-VisualChat supports continuous dialogues where text and images are interleaved.

    Key features include:

    • Scalability: Leverages DeepSpeed to train large-scale models, such as a 2B visual encoder paired with a 70B LLaMA-2 decoder.
    • Multi-Modal Causal Attention (MMCA): A novel attention mechanism that calculates attention weights independently across different modalities. It enhances causal attention for generative tasks without requiring additional modules or parameters, offering better training data efficiency than standard causal attention.
    • Data Blending: Employs techniques to handle sequential text and image data, overcoming the limitations of existing open-source datasets that lack interleaved formats.

  2. What is DeepSpeed-MII?

    master

    DeepSpeed Model Implementations for Inference (MII) is an open-source Python library designed to provide low-latency, low-cost inference for thousands of deep learning models. It automatically applies system optimizations from DeepSpeed-Inference based on model type, size, batch size, and available hardware.

    Key features include:

    • Automated Optimization: Uses model injection policies to replace standard PyTorch architectures with optimized implementations.
    • Wide Model Support: Supports text generation (BERT, RoBERTa, GPT, OPT, BLOOM), image generation (Stable Diffusion), and more.
    • Performance Gains: Can reduce latency by up to 6x and significantly lower inference costs compared to baseline PyTorch implementations.

  3. Overview of the DeepSpeed4Science Initiative

    master

    DeepSpeed4Science is an initiative by the Microsoft DeepSpeed team designed to accelerate scientific discovery by developing AI system technologies tailored for the unique complexities of scientific models. While standard DeepSpeed focuses on general LLM training, inference, and compression, DeepSpeed4Science creates specialized techniques for domains like climate science, drug design, biology, molecular dynamics, and materials discovery.

    The initiative aims to become a unified software platform and code repository for sharing advanced AI technologies that support scientific research, following Microsoft's 'AI for Good' commitment.

  4. Overview of DeepSpeed Model Compression Library

    master

    DeepSpeed Compression is a library designed for researchers and practitioners to compress deep learning models. It aims to deliver faster inference speeds, smaller model sizes, and reduced compression costs.

    Key features include:

    • State-of-the-art techniques: Built-in methods for various compression tasks.
    • Composable methods: Supports combining different compression techniques (e.-g., quantization and pruning) for synergistic effects.
    • Optimized Inference Engine: An end-to-end approach that improves the computational efficiency of the resulting compressed models.
    • Advanced Pipelines: Supports specialized pipelines like ZeroQuant (for zero-cost quantization) and XTC (for extreme compression).

  5. What is DeepSpeed-FastGen?

    master
    DeepSpeed-FastGen is an inference system framework designed for easy, fast, and affordable inference of Large Language Models (LLMs). It utilizes a technique called Dynamic SplitFuse to handle the challenges of serving LLMs at scale, offering higher effective throughput compared to other state-of-the-art systems like vLLM. It is particularly effective at combining prompt processing and decoding (token generation) in a single forward pass.

  6. What is DeepCompile and how to use it

    master

    DeepCompile is a compiler-level optimization framework for distributed training that transforms standard single-GPU model implementations into optimized multi-GPU training graphs. It eliminates the need for manual distributed logic by automatically applying parameter sharding, communication scheduling, and memory-aware execution at the compiler Intermediate Representation (IR) level.

    Key Features:

    • Automatic Parallelization: Rewrites single-GPU computation graphs into distributed versions (supporting ZeRO, FSDP, etc.) without changing model code.
    • Profile-Guided Optimization: Uses runtime profiling (memory usage, latency) to dynamically schedule computation and communication, improving overlap and avoiding bottlenecks.
    • Zero Code Changes: Works with standard implementations, such as those from the Hugging Face model hub.

    Requirements:

    • DeepSpeed version >= 0.16.6.

  7. Overview of DeepSpeed Compression

    master

    DeepSpeed Compression is a library designed for researchers and practitioners to compress deep learning models. It aims to deliver faster speeds, smaller model sizes, and significantly reduced compression costs.

    Key features include:

    • State-of-the-art compression techniques: Achieves faster compression with better model quality and lower costs.
    • End-to-end efficiency: Includes a highly optimized inference engine to improve the computation efficiency of compressed models.
    • Composable methods: Supports the synergistic composition of multiple built-in compression methods alongside system optimizations.
    • Seamless pipeline: Provides an easy-to-use pipeline for efficient deep learning model inference.

  8. Overview of DeepSpeed Chat

    master

    DeepSpeed Chat is an end-to-end pipeline designed for easy, fast, and affordable Reinforcement Learning from Human Feedback (RLHF) training of ChatGPT-like models at all scales. It addresses the complexity and high resource requirements of traditional RLHF by providing:

    1. Easy Training and Inference: Scripts to execute the full three-step InstructGPT training process starting from Hugging Face models, along with inference APIs for conversational testing.
    2. DeepSpeed-RLHF Pipeline: A comprehensive implementation of the three InstructGPT stages:
      • a) Supervised Fine-Tuning (SFT)
      • b) Reward Model Fine-Tuning
      • c) RLHF (Reinforcement Learning with Human Feedback) It also supports data abstraction and blending from multiple sources.
    3. DeepSpeed-RLHF System (DeepSpeed-HE): A hybrid engine that seamlessly switches between inference and training modes. It leverages DeepSpeed-Inference optimizations (like tensor parallelism and high-performance Transformer kernels) during inference, and DeepSpeed memory optimization techniques (like ZeRO and LoRA) during RLHF training.

    DeepSpeed-HE is optimized for high throughput and scalability, capable of training models with hundreds of billions of parameters across multiple nodes and GPUs, or even training models like OPT-13B on a single GPU.

  9. Overview of DeepSpeed Chat for RLHF Training

    master

    DeepSpeed Chat is an end-to-end Reinforcement Learning from Human Feedback (RLHF) pipeline designed to train ChatGPT-like models efficiently and affordably. It addresses the limitations of standard pre-training and fine-tuning pipelines by providing a system optimized for the complex InstructGPT training stages.

    Key capabilities include:

    • Easy-to-use Training and Inference: A single script can take a pre-trained Huggingface model through all three stages of InstructGPT training (SFT, Reward Model Fine-tuning, and RLHF) and provides an inference API for testing.
    • DeepSpeed-RLHF Pipeline: Replicates the InstructGPT training steps: (a) Supervised Fine-tuning (SFT), (b) Reward Model Fine-tuning, and (c) Reinforcement Learning with Human Feedback (RLHF). It supports data abstraction and blending from multiple sources.
    • DeepSpeed-RLHF System (DeepSpeed-HE): A unified Hybrid Engine that seamlessly transitions between inference and training modes. It leverages DeepSpeed-Inference optimizations (like tensor-parallelism and high-performance transformer kernels) for generation, and DeepSpeed training optimizations (like ZeRO and LoRA) for RL training.

  10. Overview of DeepSpeed ZeRO++ for LLM Training

    master

    DeepSpeed ZeRO++ is an optimization system designed to dramatically accelerate the training of Large Language Models (LLMs) and chat models by reducing communication overhead. It addresses limitations in standard ZeRO optimization, particularly in scenarios where:

    1. Small per-GPU batch sizes: When training across many GPUs relative to the global batch size, frequent communication becomes a bottleneck.
    2. Low-bandwidth clusters: When training on hardware with limited inter-node network bandwidth.

    ZeRO++ achieves up to a 4x reduction in total communication volume compared to standard ZeRO by combining quantization and data remapping techniques without affecting model quality. This leads to:

    • Higher Throughput: Up to 2.2x throughput improvement when per-GPU batch sizes are small.
    • Bandwidth Efficiency: Allows low-bandwidth clusters to achieve throughput comparable to clusters with 4x higher bandwidth.
    • RLHF Acceleration: When integrated with DeepSpeed-Chat, it can speed up the generation phase of RLHF (Reinforcement Learning from Human Feedback) by up to 2x and the training phase by up to 1.3x.

  11. Overview of DeepSpeed Monitoring capabilities

    master

    DeepSpeed's Monitor module allows you to log training metrics and performance details to various backends, including TensorBoard, Weights & Biases (WandB), Comet, or simple CSV files.

    DeepSpeed automatically logs several metrics depending on your configuration. Common metrics include:

    • Train/Samples/train_loss: The training loss.
    • Train/Samples/lr: The learning rate.
    • Train/Samples/loss_scale: The loss scale (requires fp16 enabled).
    • Train/Samples/elapsed_time_ms_step: The optimizer step time (requires flops_profiler.enabled or wall_clock_breakdown).
    • Train/Samples/elapsed_time_ms_forward: Forward pass duration (requires flops_profiler.enabled or wall_clock_breakdown).
    • Train/Samples/elapsed_time_ms_backward: Backward pass duration (requires flops_profiler.enabled or wall_clock_breakdown).

  12. Overview of DeepSpeed Autotuning

    master

    DeepSpeed Autotuning is a feature designed to automatically discover the optimal DeepSpeed configuration (such as ZeRO optimization stages and micro-batch sizes) to maximize training throughput and hardware utilization.

    Instead of manually hand-tuning parameters—which is time-consuming and hardware-dependent—the Autotuner uses model information, system information, and heuristics to explore system knobs that affect compute and memory efficiency. It requires no changes to your existing training code; you only need to modify your DeepSpeed configuration JSON and use a specific flag with the DeepSpeed launcher.