DOCSEARCHbyAlgolia
Submit Docs

AraBERT Documentation

repository·master·Indexed 20 days ago

https://github.com/aub-mind/arabert

A repository of state-of-the-art Arabic language models, including AraBERT (encoder-based), AraGPT2 (decoder-based), and AraELECTRA (discriminator-based). It provides tools for text preprocessing via ArabertPreprocessor, model weights hosted on HuggingFace, and scripts for pre-training, fine-tuning, and evaluating models across various NLP tasks such as Question Answering, Sequence Tagging, and machine-generated text detection.

Tokens
5.9K
Snippets
22
Records
32
Agent score
73%

What's inside AraBERT

  1. How to add a new task to AraELECTRA

    master

    To implement a new downstream task:

    1. Create a new class that implements finetune.task.Task.
    2. For standard tasks, inherit from:
      • finetune.classification.classification_tasks.ClassificationTask (Classification)
      • finetune.qa.qa_tasks.QATask (Question Answering)
      • finetune.tagging.tagging_tasks.TaggingTask (Sequence Tagging)
    3. Register the new task in finetune.task_builder.py.
    4. Use run_finetuning.py as usual.

  2. Pre-train an ELECTRA model

    master

    Run run_pretraining.py to start the pre-training process. If training is interrupted, re-running the command with the same arguments will resume from the last checkpoint.

    Arguments:

    • --data-dir: Directory containing pre-training data (expects pretrain_tfrecords and vocab.txt subdirectories) and model weights.
    • --model-name: Name for the model. Weights are saved to <data-dir>/models/<model-name>.
    • --hparams (optional): A JSON dict or path to a JSON file containing hyperparameters (see configure_pretraining.py for details).

  3. Test AraGPT2 models using Hugging Face transformers

    master

    You can use AraGPT2 for text generation using the transformers library. Note that the model class you import depends on the model size:

    • For AraGPT2-base and AraGPT2-medium: Use transformers.GPT2LMHeadModel.
    • For AraGPT2-large and AraGPT2-mega: Use arabert.aragpt2.grover.modeling_gpt2.GPT2LMHeadModel as a direct replacement for the standard transformers class. These models follow the grover architecture and are compatible with transformers v4.x.

    It is recommended to use ArabertPreprocessor to clean your input text before passing it to the model.

    from transformers import GPT2TokenizerFast, pipeline
# For base and medium
from transformers import GPT2LMHeadModel
# For large and mega
from arabert.aragpt2.grover.modeling_gpt2 import GPT2LMHeadModel

from arabert.preprocess import ArabertPreprocessor

MODEL_NAME='aubmindlab/aragpt2-base'
arabert_prep = ArabertPreprocessor(model_name=MODEL_NAME)

model = GPT2LMHeadModel.from_pretrained(MODEL_NAME)
tokenizer = GPT2TokenizerFast.from_pretrained(MODEL_NAME)
generation_pipeline = pipeline("text-generation", model=model, tokenizer=tokenizer)

text=""
text_clean = arabert_prep.preprocess(text)

# Generation with decoding settings
result = generation_pipeline(
    text_clean,
    pad_token_id=tokenizer.eos_token_id,
    num_beams=10,
    max_length=200,
    top_p=0.9,
    repetition_penalty=3.0,
    no_repeat_ngram_size=3
)[0]['generated_text']

  5. How to use AraBERT models

    master

    AraBERT is designed to be compatible with existing BERT codebases. The primary difference is in the tokenization.py file, where the _is_punctuation function is modified to support the + symbol and [ and ] characters.

    Depending on the version you choose, your preprocessing workflow will differ:

    1. AraBERTv1 and AraBERTv2: These versions require pre-segmentation. You must use the ArabertPreprocessor to split prefixes and suffixes (using the Farasa Segmenter logic) before passing the text to the tokenizer.
    2. AraBERTv0.1 and AraBERTv0.2: These versions do not require pre-segmentation and can be used with raw text.
    from transformers import AutoTokenizer, AutoModel
from arabert.preprocess import ArabertPreprocessor

model_name = "aubmindlab/bert-base-arabertv2"
arabert_tokenizer = AutoTokenizer.from_pretrained(model_name)
arabert_model = AutoModel.from_pretrained(model_name)

arabert_prep = ArabertPreprocessor(model_name=model_name)

text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
text_preprocessed = arabert_prep.preprocess(text)
# Output: "و+ لن نبالغ إذا قل +نا إن هاتف أو كمبيوتر ال+ مكتب في زمن +نا هذا ضروري"

tokens = arabert_tokenizer.tokenize(text_preprocessed)
# Output: ['و+', 'لن', 'نبال', '##غ', 'إذا', 'قل', '+نا', 'إن', 'هاتف', 'أو', 'كمبيوتر', 'ال+', 'مكتب', 'في', 'زمن', '+نا', 'هذا', 'ضروري']

  6. Download TensorFlow 1.x models

    master

    PyTorch, TF2, and TF1 models are available on HuggingFace under the aubmindlab username. To download a TensorFlow 1.x model specifically, use wget with the following pattern:

    wget https://huggingface.co/aubmindlab/MODEL_NAME/resolve/main/tf1_model.tar.gz

    Replace MODEL_NAME with the specific model you require.

  7. Fine-tune AraGPT2 using TensorFlow 1.15.4

    master

    For custom training or fine-tuning using the repository's native code (optimized for GPUs and TPUs via TPUEstimator), follow these two steps:

    1. Create Training TFRecords

    Use create_pretraining_data.py to convert raw text files into TFRecords. The input file should contain documents/articles separated by an empty line.

    2. Run Pretraining/Fine-tuning

    Execute run_pretraining.py with the required hyperparameters and paths to your Google Storage (GS) buckets.

    # Step 1: Create TFRecords
python create_pretraining_data.py \
 --input_file=<RAW TEXT FILE with documents/article sperated by an empty line> \
 --output_file=<OUTPUT TFRecord> \
 --tokenizer_dir=<Directory with the GPT2 Tokenizer files>

# Step 2: Fine-tuning
python3 run_pretraining.py \
 --input_file="gs://<GS_BUCKET>/pretraining_data/*" \
 --output_dir="gs://<GS_BUCKET>/pretraining_model/" \
 --config_file="config/small_hparams.json" \
 --batch_size=128 \
 --eval_batch_size=8 \
 --num_train_steps= \
 --num_warmup_steps= \
 --learning_rate= \
 --save_checkpoints_steps= \
 --max_seq_length=1024 \
 --max_eval_steps= \
 --optimizer="lamb" \
 --iterations_per_loop=5000 \
 --keep_checkpoint_max=10 \
 --use_tpu=True \
 --tpu_name=<TPU NAME> \
 --do_train=True \
 --do_eval=False

  8. Install farasapy for text segmentation

    master

    To use AraBERT v1 and v2 with text segmentation (pre-segmentation), you must install the farasapy library via pip.

    pip install farasapy
    pip install farasapy

  9. Use the AraGPT2 Machine-Generated Text Detector

    master

    The AraGPT2 detector is a model trained to identify machine-generated text from long passages, achieving a 99.4% F1-Score. To use it, preprocess your text with ArabertPreprocessor and then use a transformers pipeline with the aubmindlab/aragpt2-mega-detector-long model.

    from transformers import pipeline
from arabert.preprocess import ArabertPreprocessor

processor = ArabertPreprocessor(model="aubmindlab/araelectra-base-discriminator")
pipe = pipeline("sentiment-analysis", model = "aubmindlab/aragpt2-mega-detector-long")

text = " "
text_prep = processor.preprocess(text)
result = pipe(text_prep)
# Example output: [{'label': 'machine-generated', 'score': 0.9977743625640869}]

  10. Continue pre-training from released checkpoints

    master

    To extend training of a released AraELECTRA model:

    1. Set --model-name to the directory of the downloaded model (e.g., --model-name electra_small).
    2. Set num_train_steps in --hparams to the desired total steps (e.g., "num_train_steps": 4010000 to add 10,000 steps to a model already trained for 4e6 steps).
    3. Adjust the learning rate to account for linear decay: new_lr = original_lr * (total_steps_after) / (steps_added).
    4. For ELECTRA-Small, you must include "generator_hidden_size": 1.0 in your --hparams.

  11. Create a pre-training dataset for ELECTRA

    master

    Use build_pretraining_dataset.py or build_arabert_pretraining_data.py to convert raw text dumps into ELECTRA pre-training examples.

    Arguments:

    • --corpus-dir: Directory containing raw text files (documents separated by empty lines).
    • --vocab-file: Path to the wordpiece vocabulary file.
    • --output-dir: Destination directory for the generated ELECTRA examples.
    • --max-seq-length: Number of tokens per example (default: 128).
    • --num-processes: Number of parallel processes (default: 1).
    • --blanks-separate-docs: If True (default), blank lines indicate document boundaries.
    • --do-lower-case/--no-lower-case: Whether to lowercase input text (default: True).

  12. Download AraBERT model weights from HuggingFace

    master

    AraBERT models (including PyTorch, TF2, and TF1 versions) are hosted on HuggingFace under the aubmindlab organization. You can download specific model weights using wget by replacing MODEL_NAME with the desired model identifier.

    wget https://huggingface.co/aubmindlab/MODEL_NAME/resolve/main/tf1_model.tar.gz