DOCSEARCHbyAlgolia
Submit Docs

SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning)

repository·main·Indexed 12 days ago

https://github.com/modelscope/ms-swift

A comprehensive framework for the full lifecycle of large language models (LLMs) and multimodal models, covering training, alignment, quantization, inference, and deployment. It supports lightweight training methods like LoRA, QLoRA, and DoRA, as well as distributed training via DeepSpeed, FSDP, and Megatron. Key capabilities include Supervised Fine-Tuning (SFT), RLHF (DPO, GRPO, PPO), pre-training, and model evaluation.

Tokens
293.6K
Snippets
606
Records
923
Agent score
90%

What's inside SWIFT

  1. Overview of ms-swift capabilities

    main

    ms-swift (Scalable lightWeight Infrastructure for Fine-Tuning) is a comprehensive framework for large language models (LLMs) and multi-modal large language models (MLLMs). It supports the entire lifecycle including:

    • Training: Pre-training, fine-tuning (SFT), human alignment (RLHF), and specialized tasks like Embedding, Reranker, and sequence classification.
    • Inference & Deployment: Accelerated inference using engines like vLLM, SGLang, and LMDeploy.
    • Evaluation: Model evaluation using EvalScope as the backend.
    • Quantization: Support for GPTQ, AWQ, BNB, and FP8 techniques.

    Key Features:

    • Model Support: 600+ text models (e.g., Qwen, DeepSeek, Llama) and 400+ multi-modal models (e.g., Qwen-VL, Llava, InternVL).
    • Lightweight Fine-tuning: Supports LoRA, QLoRA, DoRA, LLaMAPro, and more.
    • Reinforcement Learning: Built-in support for the GRPO algorithm family (GRPO, DAPO, GSPO, etc.).
    • Distributed Training: Supports DDP, DeepSpeed (ZeRO2/3), FSDP, and Megatron-style parallelism (TP, PP, CP, EP, etc.).
    • Hardware: Compatible with NVIDIA GPUs (A10/A100/H100, RTX), CPU, MPS, and Ascend NPU.

  2. Overview of SWIFT training capabilities

    main

    SWIFT supports a wide range of training methods including Pre-training, Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF) algorithms (like PPO, DPO, KTO, GRPO, etc.), and specialized tasks like Embedding, Reranker, and Sequence Classification.

    Most methods support Full-Parameter training, LoRA, QLoRA, DeepSpeed, Multi-Machine, and Multimodal capabilities, though specific combinations may vary (e.g., PPO does not currently support Multimodal training).

  3. Distillation training methods in swift

    main

    swift provides three primary distillation training methods. They share a common teacher infrastructure but differ in how the signal is computed and passed to the student.

    MethodSignal pathHow to enableDescription
    GKDDirect loss (Path A)--rlhf_type gkdTeacher divergence as loss backprop; supports full-vocab / top-k divergence
    OPD-RLRL advantage (Path B)--rlhf_type grpo + teacherTeacher log-ratio injected into GRPO advantage; can combine with task rewards
    OPSDPath A or BProvide teacher_promptSingle-model self-distillation where the teacher input includes privileged info (e.g. reference solution)

    Teacher Sources

    • --teacher_model: A separate frozen teacher model loaded during training.
    • --teacher_model_server: An external teacher service (e.g., vLLM started via swift deploy) accessed via API. This avoids loading the teacher on training GPUs.
    • Self-distillation:
      • For LoRA: If --teacher_model equals --model, the base model is used as a fixed teacher via disable_adapter().
      • Dynamic: Without --teacher_model or --teacher_model_server, the student's current weights serve as a dynamic teacher (in GKD for all batches; in GRPO only when data includes teacher_prompt).

  4. Megatron GRPO Supported Features

    main

    Megatron GRPO provides high-performance training capabilities with the following support:

    • Training Modes: Full parameter training and LoRA fine-tuning.
    • Parallelism Strategies: Supports Context Parallelism (CP), Pipeline Parallelism (PP), Tensor Parallelism (TP), and Expert Parallelism (EP).
    • Inference Acceleration: Supports vLLM in both colocate and server modes.
    • Model Support: Compatible with LLMs and MLLMs (Multi-modal Large Models) within Megatron Swift.
    • Algorithm Support: Covers most features available in standard swift GRPO.

  5. Supported Tuners in SWIFT

    main

    SWIFT supports a wide variety of tuners designed to reduce training parameters or enhance accuracy. Supported tuners include:

    • Parameter-Efficient Fine-Tuning (PEFT): LoRA, LoRA+, LLaMA PRO, GaLore/Q-GaLore, Liger Kernel, LISA, UnSloth, LongLoRA, Adapter, Side, Res-Tuning, and various tuners provided by the Hugging Face peft library (e.g., AdaLoRA, DoRA, Fourierft).
    • Specialized Tuners: SCEdit (for image diffusion), NEFTune (noisy embeddings), and Vision Prompt Tuning.

  6. Understand the SWIFT directory structure and modules

    main

    The SWIFT repository is organized into several functional modules that developers can extend or utilize:

    • arguments: Definitions for CLI arguments (e.g., SftArguments, RLHFArguments).
    • cli: The CLI mechanism and entry points. Commands like swift sft ... are equivalent to python swift/cli/main.py sft ... or python swift/cli/sft.py ....
    • config: Configuration files for DeepSpeed and FSDP2.
    • dataloader: Implementations for data loading, including shard and dispatcher modes.
    • dataset: Modules for data preprocessing, packing, and streaming. Built-in datasets are in dataset/dataset and dataset/data. See Custom-dataset.md for custom datasets.
    • infer_engine: Inference engine implementations supporting backends like transformers, vllm, sglang, and lmdeploy.
    • megatron: Megatron-SWIFT implementation.
    • model: Model loading and registration. See Custom-model.md and MLLM-Registration.md for details.
    • pipelines: Main execution pipelines for swift sft/rlhf/infer (e.g., sft_main, rlhf_main, infer_main).
    • rlhf_trainers: Trainer implementations for RLHF algorithms like GRPO, GKD, DPO, KTO, and RM.
    • rollout: Sampling implementation for the rollout process in RL algorithms.
    • rewards: Implementation of reward functions for RL algorithms; supports custom reward logic.
    • template: Implementation and registration of conversation templates, including logic for converting messages to input_ids and data_collator logic.
    • trainers: Trainer implementations for Pre-training, SFT, Embedding, Reranker, and Sequence Classification tasks.
    • ui: Web UI implementation for training and inference via swift web-ui.

  7. Megatron GRPO Feature Support

    main

    Megatron GRPO supports the following capabilities:

    • Training Modes: Full parameter training and LoRA fine-tuning.
    • Parallelism Strategies: Context Parallelism (CP), Pipeline Parallelism (PP), Tensor Parallelism (TP), and Expert Parallelism (EP).
    • Inference Acceleration: vLLM colocate mode and server mode.
    • Model Support: Compatible with LLMs and MLLMs (multimodal large models) within the Megatron Swift ecosystem.
    • Algorithm Support: Covers most features available in standard Swift GRPO.

  9. Quick Start with ms-swift

    main

    ms-swift is a comprehensive framework for fine-tuning, deploying, and evaluating large language models (LLMs) and multi-modal models. It supports over 600 text models and 400 multi-modal models, including popular series like Qwen, Llama, DeepSeek, and InternLM.

    Key Capabilities

    • Training: Supports SFT, DPO, KTO, RL (including GRPO family), and more. Includes lightweight fine-tuning (LoRA, QLoRA, etc.) and distributed training (DeepSpeed, Megatron, FSDP).
    • Inference & Deployment: Accelerates inference using vLLM, SGLang, and LMDeploy. Provides OpenAI-compatible interfaces.
    • Quantization: Supports AWQ, GPTQ, FP8, and BNB for model quantization.
    • Multi-modal: Supports mixed-modality training (text, image, video, audio) with packing techniques.
    • Hardware: Compatible with NVIDIA GPUs (A10/A100/H100, RTX), CPU, MPS, and Ascend NPU.

  11. Training capabilities overview

    main

    SWIFT supports a wide range of training methods including Pre-training, Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF) methods like GRPO, GKD, PPO, DPO, KTO, CPO, SimPO, and ORPO. It also supports training for Embedding, Reranker, and Sequence Classification models.

    Key features across these methods include:

    • Parameter-efficient training: Support for Full Parameter, LoRA, and QLoRA.
    • Scalability: Support for DeepSpeed (Zero2/Zero3) and Multi-node training.
    • Modality: Extensive support for Multimodal models across most training methods.

  12. What is GKD (Generalized Knowledge Distillation)

    main

    GKD (Generalized Knowledge Distillation) is a training method used to transfer knowledge from a teacher model to a student model. It achieves distillation by calculating the Jensen-Shannon Divergence (JSD) loss between the output distributions of the two models.

    Megatron GKD Features:

    • Training Modes: Supports Full Parameter training and LoRA fine-tuning.
    • Parallelism Strategies: Supports Context Parallel (CP), Pipeline Parallel (PP), Tensor Parallel (TP), and Expert Parallel (EP).
    • Model Support: Compatible with LLMs and MLLMs within Megatron-SWIFT.
    • Teacher Offload: Supports offloading the teacher model to CPU to save GPU memory.
    • Online Generation: Supports on-policy generation for the student model using vLLM.