ERNIE Documentation

FP8 QAT is a method designed to enable efficient training of ERNIE 4.5 models (such as the 300B models) with significantly reduced hardware requirements.

Key Benefits

• Resource Efficiency: Enables SFT (Supervised Fine-Tuning) full-parameter tuning of 300B models using only 16 Hopper 80G GPUs. This represents approximately 17% of the hardware resources typically required for traditional BF16 mixed-precision training.
• Inference Performance: Supports tensor-wise static W8A8 FP8 inference without requiring quantization calibration. This method achieves a 1.17x speedup compared to block-wise dynamic FP8 quantization inference.

Memory Management Strategy

To minimize GPU memory consumption, the approach employs the following strategy:

• Model Parameters: Stored in FP8.
• Optimizer Moments and Gradients: Use BF16.
• Optimizer States: All optimizer states are offloaded to pinned memory to further reduce GPU memory usage.

WINT8MIX LoRA is a parameter-efficient fine-tuning (PEFT) method designed to reduce the memory footprint required to train large ERNIE 4.5 models (such as the 300B model). It combines Weight Only INT8 quantization with LoRA (Low-Rank Adaptation).

Key Benefits

• Reduced Resource Requirements: Enables training of the 300B model with significantly fewer GPUs.
• Sequence Length Capabilities:
  • 8K sequence training: Requires 4×80G GPUs.
  • 32K sequence training: Requires 8×80G GPUs.

Technical Approach

• LoRA: Freezes original weights and introduces small, trainable low-rank matrices into transformer layers.
• Mixed Quantization: Leverages the sparse characteristics of Mixture-of-Experts (MoE) models:
  • INT4 quantization: Applied to linear weights in expert modules.
  • INT8 quantization: Applied to linear weights in all other components.
• WINT (Weight Only Quantization): Utilizes channelwise quantization and specialized acceleration operators to improve training speed without accuracy degradation.

Follow these steps to prepare your environment using Docker and PaddlePaddle on x86 Ascend NPU machines.

1. Pull the Docker image:

   docker pull ccr-2vdh3abv-pub.cnc.bj.baidubce.com/device/paddle-npu:cann80RC1-ubuntu20-x86_64-gcc84-py310

2. Start the container: Ensure you mount the necessary Ascend drivers and device paths.

   sudo docker run -it --name paddle_test   \#
   -v /home/:/home/  \#
   -v `pwd`:/workspace  \#
   -w /workspace  \#
   --privileged --network=host --shm-size=128G  \#
   -v /ssd1/dataset:/workspace/dataset  \#
   -v /usr/local/Ascend/driver:/usr/local/Ascend/driver  \#
   -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi  \#
   -v /usr/local/dcmi:/usr/local/dcmi  \#
   -e ASCEND_RT_VISIBLE_DEVICES="0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15"  \#
   registry.baidubce.com/device/cann80RC1-ubuntu20-x86_64-gcc84-py310 /bin/bash

3. Install PaddlePaddle and PaddleCustomDevice:

   python -m pip install paddlepaddle==3.1.0a0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/
   python -m pip install paddle-custom-npu==3.1.0 -i https://www.paddlepaddle.org.cn/packages/stable/npu/

4. Install project requirements:

   cd ERRNIEKit
   pip install -r requirements/gpu/requirements.txt

Follow these steps to prepare your environment using Docker and install the necessary PaddlePaddle XPU components.

1. Pull the Docker image:

   docker pull ccr-2vdh3abv-pub.cnc.bj.baidubce.com/device/paddle-xpu:ubuntu20-x86_64-gcc84-py310

2. Start the container: Use the following command to start a container with privileged access, host networking, and XPU device mapping. Replace $(pwd) with your actual project path and xxx with your dataset path.

   docker run -it --privileged=true --net host --shm-size '256gb' --device=/dev/xpu0:/dev/xpu0 --device=/dev/xpu1:/dev/xpu1 --device=/dev/xpu2:/dev/xpu2 --device=/dev/xpu3:/dev/xpu3 --device=/dev/xpu4:/dev/xpu4 --device=/dev/xpu5:/dev/xpu5 --device=/dev/xpu6:/dev/xpu6 --device=/dev/xpu7:/dev/xpu7 --device=/dev/xpuctrl:/dev/xpuctrl --name paddle-xpu-dev -v $(pwd):/work -w=/work -v xxx ccr-2vdh3abv-pub.cnc.bj.baidubce.com/device/paddle-xpu:ubuntu20-x86_64-gcc84-py310 /bin/bash

3. Install PaddlePaddle-XPU and custom operators:

   # Install the core framework
   python -m pip install paddlepaddle-xpu==3.3.0.dev20251016 -i https://www.paddlepaddle.org.cn/packages/nightly/xpu-p800/
   
   # Install XPU custom operators for acceleration
   wget https://bj.bcebos.com/v1/klx-paddlelite/paddle_whl/paddle_kl3/daily_output/20251014/paddle_xpu-0.0.1-py3-none-any.whl
   python -m pip install paddle_xpu-0.0.1-py3-none-any.whl

4. Install project requirements:

   pip install -r requirements/gpu/requirements.txt
   python -m pip install -e .

Supervised Fine-Tuning (SFT) adjusts all model parameters using high-quality annotated data. It is ideal for precision-critical specialized tasks.

Full-Parameter SFT

To perform full-parameter SFT, use the erniekit train command with the appropriate configuration file.

Parameter Efficient Fine-tuning (LoRA)

LoRA (Low-Rank Adaptation) reduces resource requirements by only adjusting a small number of new parameters. To enable LoRA, add fine_tuning: LoRA to your training configuration.

Example: Full-Parameter SFT (8K Seq Length) on ERNIE-4.5-0.3B

# download model from huggingface
huggingface-cli download baidu/ERNIE-4.5-0.3B-Paddle --local-dir baidu/ERNIE-4.5-0.3B-Paddle
# 8K Sequence Length, SFT
erniekit train examples/configs/ERNIE-4.5-0.3B/sft/run_sft_8k.yaml

Example: SFT-LoRA (8K Seq Length) on ERNIE-4.5-0.3B

# download model from huggingface
huggingface-cli download baidu/ERNIE-4.5-0.3B-Paddle --local-dir baidu/ERNIE-4.5-0.3B-Paddle
# 8K Sequence Length, SFT-LoRA
erniekit train examples/configs/ERNIE-4.5-0.3B/sft/run_sft_lora_8k.yaml

Start the Supervised Fine-Tuning process using the erniekit train command. You can control GPU visibility using the CUDA_VISIBLE_DEVICES environment variable.

Command Syntax:

CUDA_VISIBLE_DEVICES=0 \
erniekit train <config_yaml_path> \
        model_name_or_path=<model_path_or_name> \
        train_dataset_path=<path_to_jsonl>

Scaling Training with Multiple GPUs: If you increase the number of GPUs by a factor of N, you can adjust hyperparameters to maintain the same effective batch size (sample_num = G*B*A):

1. Keep sample_num constant: Reduce packing_size and gradient_accumulation_steps such that the product of their reduction factors equals N.
2. Scale Learning Rate: Increase learning_rate by a factor of N.

CUDA_VISIBLE_DEVICES=0 \
erniekit train examples/configs/PaddleOCR-VL/sft/run_ocr_vl_sft_16k.yaml \
        model_name_or_path=PaddlePaddle/PaddleOCR-VL \
        train_dataset_path=./ocr_vl_sft-train_Bengali.jsonl \