Indic BERT v1

IndicGLUE is a benchmark for NLU performance in Indian languages. It includes the following tasks:

• News Category Classification: Predict the genre of a news article.
• Named Entity Recognition (NER): Recognize entities and their types in a sequence.
• Headline Prediction: Predict the correct headline for a news article from candidates.
• Wikipedia Section Title Prediction: Predict the correct title for a Wikipedia section.
• Cloze-style Question Answering (WCQA): Predict a masked entity from a list of candidates.
• Cross-lingual Sentence Retrieval (XSR): Retrieve a translation of a sentence from a set of candidates in a different language.

The easiest way to use Indic BERT is through the Hugging Face transformers library. You can load the pre-trained tokenizer and model using AutoTokenizer and AutoModel.

To preserve accents (vowel matras / diacritics) during tokenization, set the keep_accents parameter to True.

# Required dependencies:
# pip3 install transformers
# pip3 install sentencepiece

from transformers import AutoModel, AutoTokenizer

# Standard loading
tokenizer = AutoTokenizer.from_pretrained('ai4bharat/indic-bert')
model = AutoModel.from_pretrained('ai4bharat/indic-bert')

# Loading with accent preservation
tokenizer = AutoTokenizer.from_pretrained('ai4bharat/indic-bert', keep_accents=True)

Since the project uses the Hugging Face transformers library, you can load models from a local directory. To do this, your directory must contain the following three files:

1. config.json: The configuration file in Hugging Face format.
2. tok.model: The tokenizer model file (e.g., SentencePiece or WordPiece).
3. pytorch_model.bin: The PyTorch binary containing the model parameters.

To run evaluation or fine-tuning with these local files, use the following command structure:

python3 -m fine_tune.cli --model <path to the directory containing pytorch_model.bin> --tokenizer_name <path to the tokenizer file> --config_name <path to the config file> --dataset <dataset name> --lang <iso lang code> --iglue_dir <base path to indic glue dir> --output_dir <output dir>

You can download Indic BERT models in both TensorFlow checkpoint and PyTorch binary formats via:

1. Direct Download: Google Storage Link
2. Hugging Face: ai4bharat/indic-bert

You can use the fine-tuning CLI to train on any model available on the Hugging Face Hub by providing its shortcut name to the --model argument. This is useful for testing different architectures or pre-trained weights with the Indic-BERT fine-tuning pipeline.

If you are using a Google Colab notebook, you can pass these arguments within the argvec list in the provided notebook script.

python3 -m fine_tune.cli --model <HF name*> --dataset <dataset name> --lang <iso lang code> --iglue_dir <base path to indic glue dir> --output_dir <output dir>

To run the Indic BERT code on a local VM, clone the repository and install the required dependencies via requirements.txt.

git clone https://github.com/AI4Bharat/indic-bert
cd indic-bert
sudo pip3 install -r requirements.txt

Use the albert.run_race module for the RACE benchmark. You can also use --albert_hub_module_handle to load from TF-Hub.

Key arguments:

• --max_seq_length: Typically 512.
• --max_qa_length: Typically 128.
• --learning_rate: Recommended 1e-5 for v2 models.

python -m albert.run_race \
  --albert_config_file=... \
  --output_dir=... \
  --train_file=... \
  --eval_file=... \
  --data_dir=...\
  --init_checkpoint=... \
  --spm_model_file=... \
  --max_seq_length=512 \
  --max_qa_length=128 \
  --do_train \
  --do_eval \
  --train_batch_size=32 \
  --eval_batch_size=8 \
  --learning_rate=1e-5 \
  --train_step=12000 \
  --warmup_step=1000 \
  --save_checkpoints_steps=100

To run experiments on a TPU, you must update your .bashrc with specific environment variables and install pytorch-xla.

1. Set Environment Variables:

export PYTHONPATH="${PYTHONPATH}:/usr/share/tpu/models:<path to this repo"
export PYTHONIOENCODING=utf-8
export TPU_IP_ADDRESS="<TPU Internal Address"
export TPU_NAME="grpc://$TPU_IP_ADDRESS:8470"
export XRT_TPU_CONFIG="tpu_worker;0;$TPU_IP_ADDRESS:8470"
export LD_LIBRARY_PATH="/usr/local/lib"

2. Install pytorch-xla:

curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
sudo python3 pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev

If your model is currently in TensorFlow format, you must convert it to a PyTorch binary (pytorch_model.bin) before it can be used with the fine-tuning CLI. Use the transformers-cli convert command.

Note: You must modify the --model_type and filenames in the command below to match your specific model architecture (e.g., change albert to bert if necessary).

MODEL_DIR=$1

# modify model_type and filenames accordingly
transformers-cli convert --model_type albert \
  --tf_checkpoint $MODEL_DIR/tf_model \
  --config $MODEL_DIR/config.json \
  --pytorch_dump_output $MODEL_DIR/pytorch_model.bin

Datasets for fine-tuning can be downloaded from the Indic-GLUE website. For example, to download and extract the wiki-cloze dataset, navigate to the indic-glue directory and use wget and tar.

% cd indic-glue
!wget https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/wiki-cloze.tar.gz
!tar -xaf wiki-cloze.tar.gz
% cd ..