BanglaBERT

Pretrained model checkpoints are available on the Huggingface model hub. The following variants are provided:

• BanglaBERT: The standard pretrained model.
• BanglishBERT: A model variant for Banglish (Bangla written in Roman script).
• BanglaBERT (small): A lightweight version that can be fine-tuned with as little as 4 GB VRAM.
• BanglaBERT (large): A larger, more powerful variant.

Critical Requirement: These models were pretrained using a specific normalization pipeline. To achieve optimal results when adapting the model for new tasks, you must normalize your text units using the csebuetnlp/normalizer pipeline before tokenization.

# Available models on Huguggingface:
# - csebuetnlp/banglabert
# - csebuetnlp/banglishbert
# - csebuetnlp/banglabert_small
# - csebuetnlp/banglabert_large

The question_answering.py script accepts input data files in two ways:

1. Directory mode: Use the --dataset_dir <path> flag. The directory must contain files prefixed with train, validation, and test (e.g., train.json, validation.json, test.json).
2. Individual file mode: Use specific flags for each split: --train_file <path>, --validation_file <path>, and --test_file <path>.

The token classification script requires input data in jsonl format (one JSON object per line).

By default, each line must contain the following keys:

• tokens: A list of input tokens.
• tags: A list of classification labels/tags corresponding to each token.

If your dataset uses different keys, you can map them using the --tokens_key <key_name> and --tags_key <key_name> flags when running token_classification.py.

{
  "tokens": ["Hello", "world"],
  "tags": ["O", "O"]
}

To finetune the model on a single GPU, use the token_classification.py script. You can provide data by pointing to a directory containing files prefixed with train, validation, and test using the --dataset_dir flag, or by specifying individual files using --train_file, --validation_file, and --test_file.

$ python ./token_classification.py \
    --model_name_or_path "csebuetnlp/banglabert" \
    --dataset_dir "sample_inputs/" \
    --output_dir "outputs/" \
    --learning_rate=2e-5 \
    --warmup_ratio 0.1 \
    --gradient_accumulation_steps 2 \
    --weight_decay 0.1 \
    --lr_scheduler_type "linear"  \
    --per_device_train_batch_size=16 \
    --per_device_eval_batch_size=16 \
    --max_seq_length 512 \
    --logging_strategy "epoch" \
    --save_strategy "epoch" \
    --evaluation_strategy "epoch" \
    --num_train_epochs=3 \
    --do_train --do_eval

To use the BanglaBERT models and scripts, you need to clone the repository and set up a specific Conda environment with the required versions of Python and PyTorch.

Follow these steps:

1. Clone the repository.
2. Create a Conda environment with the specified dependencies.
3. Activate the environment.
4. Run the setup.sh script to install remaining requirements.

Note: Always use the newly created environment to run the scripts provided in this repository.

$ git clone https://github.com/csebuetnlp/banglabert
$ cd banglabert/
$ conda create python==3.7.9 pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.0 cudatoolkit=10.2 -c pytorch -p ./env
$ conda activate ./env # or source activate ./env (for older versions of anaconda)
$ bash setup.sh 

The sequence_classification.py script supports csv, tsv, and jsonl (one JSON per line) formats.

Default Column/Key Names

Single sequence classification:

• sentence1: The input sequence.
• label: The classification label (optional for test files).

Double sequence classification:

• sentence1: The first input sequence.
• sentence2: The second input sequence.
• label: The classification label (optional for test files).

Customizing Column/Key Names

If your data uses different headers, use the following flags to map them:

• --sentence1_key <key_name>
• --sentence2_key <key_name>
• --label_key <key_name>

BanglaBERT supports several downstream task categories. You can find specific implementation code in the corresponding directories within the repository:

• Sequence Classification:
  • Single sequence: Document classification, Sentiment classification, Emotion classification, etc.
  • Double sequence: Natural Language Inference (NLI), Paraphrase detection, etc.
• Token Classification: Token tagging tasks like Named Entity Recognition (NER) and Parts of Speech (PoS) Tagging.
• Question Answering: Extractive Question Answering and Open-domain Question Answering.

To calculate metrics on a test set or perform inference on raw data using a previously trained model, use the --do_predict flag. Provide the path to your trained model via --model_name_or_path.

$ python ./token_classification.py \
    --model_name_or_path <path/to/trained/model> \
    --dataset_dir "sample_inputs/" \
    --output_dir "outputs/" \
    --per_device_eval_batch_size=16 \
    --overwrite_output_dir \
    --do_predict

To calculate metrics on a test set or perform inference on raw data using a previously trained model, use the --do_predict flag. Use --overwrite_output_dir to ensure the output directory is refreshed.

$ python ./question_answering.py \
    --model_name_or_path <path/to/trained/model> \
    --dataset_dir "sample_inputs/" \
    --output_dir "outputs/" \
    --per_device_eval_batch_size=16 \
    --overwrite_output_dir \
    --do_predict

To calculate metrics on a test set or perform inference on raw data using a previously trained model, use the --do_predict flag. Ensure you provide the path to your trained model via --model_name_or_path.

$ python ./sequence_classification.py \
    --model_name_or_path <path/to/trained/model> \
    --dataset_dir "sample_inputs/single_sequence/jsonl" \
    --output_dir "outputs/" \
    --per_device_eval_batch_size=16 \
    --overwrite_output_dir \
    --do_predict