DeBERTa Documentation

The following table shows the developer performance of DeBERTa models on specific SuperGLUE tasks: COPA, RTE, and ReCoRD (reporting both F1 and Exact Match scores).

During evaluation, you can export the model to ONNX format using the --export_ort_model flag. You can also generate quantized models by controlling the --fp16 flag.

Export Options

• Standard ONNX export: Use --export_ort_model True.
• Quantized model export: Use --fp16 False --export_ort_model True.

Output File Naming

Exported models are saved in the output folder with the following naming convention based on the <prefix>:

• If fp16 is True: <prefix>__onnx_fp16.bin
• If fp16 is False: <prefix>__onnx_fp32.bin and <prefix>__onnx_qt.bin

You can install DeBERTa as a pip package using the following command:

pip install deberta

To train an ELECTRA-like model using the Replaced Token Detection task, use the rtd.sh script. Available DeBERTaV3 configurations include:

• deberta-v3-xsmall: 9M backbone parameters (12 layers, 256 hidden size) + 32M embedding parameters (128k vocabulary).
• deberta-v3-base: 81M backbone parameters (12 layers, 768 hidden size) + 96M embedding parameters (128k vocabulary).
• deberta-v3-large: 288M backbone parameters (24 layers, 1024 hidden size) + 128M embedding parameters (128k vocabulary).

You can perform continuous training on DeBERTaV3-X models by using the -continue suffix. You must check the rtd.sh script to specify the initialization models for both the generator and the discriminator.

Common checkpoint sources for continuous training include Hugging Face repositories for deberta-v3-xsmall, deberta-v3-small, and deberta-v3-large.

To run experiments on the RACE (Reading Comprehension) dataset, follow these steps:

1. Download the dataset from http://www.cs.cmu.edu/~glai1/data/race/.
2. Execute the provided shell script run_race.sh to start the experiment.

If you have pulled the repository source code, you can install the required dependencies using pip:

1. Run pip3 install -r requirements.txt from the root directory.
2. Navigate to experiments/glue/ to run GLUE experiments.

You can run fine-tuning experiments for various GLUE benchmark tasks using the provided shell scripts in the experiments/glue/ directory. The scripts take the model size/variant as an argument.

Example commands for the Large model:

• MNLI xxlarge v2: experiments/glue/mnli.sh xxlarge-v2
• QQP large: experiments/glue/qqp.sh large
• CoLA large: experiments/glue/cola.sh

Note: Results may vary based on GPU models, drivers, CUDA versions, precision (FP16 vs FP32), and random seeds.

You can run pre-configured SiFT experiments for DeBERTa by executing the provided shell scripts in the experiments/glue/ directory.

Examples:

• For a base model: experiments/glue/mnli.sh base-sift
• For an xxlarge-v2 model: experiments/glue/mnli.sh xxlarge-v2-sift

Docker is the recommended way to run DeBERTa as all dependencies are pre-built in the bagai/deberta image.

To run GLUE experiments with Docker:

1. Pull the code.
2. Run ./run_docker.sh.
3. Execute bash commands located under /DeBERTa/experiments/glue/ within the container.