AI4Bharat-IndicNLP Corpus

To replicate the training and evaluation results reported in the IndicNLP paper, you must install the following tools and packages:

1. FastText: https://github.com/facebookresearch/fastText
2. MUSE: https://github.com/facebookresearch/MUSE
3. Python Dependencies: Install the required packages listed in the repository's requirements.txt file.

To train a new morphanalyzer, you must first extract a vocabulary from an existing embeddings file and then run the morfessor-train command.

1. Extract vocabulary from embeddings file

Use zcat, tail, and cut to isolate the first column (the words) from the embedding vectors file, skipping the header.

2. Train Morfessor

Run the morfessor-train command using the extracted vocabulary file as the training data source.

## extract vocabulary from embedings file
zcat $embedding_vectors_path |  \
    tail -n +2 | \
    cut -f 1 -d ' '  > $vocab_file_path

## train morfessor 
morfessor-train -d ones \
        -S $model_file_path \
        --logfile  $log_file_path \
        --traindata-list $vocab_file_path \
        --max-epoch 10

To evaluate embeddings on the IIIT-H Word Similarity Database, download the database from here and run the wordsim.py script.

python scripts/word_similarity/wordsim.py \
	<embedding_file_path> \
	<word_sim_db_path> \
	<max_vocab>

To evaluate embeddings on the Facebook word analogy dataset, you must first add the MUSE root directory to your PYTHONPATH. Then, use the word_analogy.py script.

export PYTHONPATH=$PYTHONPATH:$MUSE_PATH

python  scripts/word_analogy/word_analogy.py \
    --analogy_fname <analogy_fname> \
    --embeddings_path <embedding_file_path> \
    --lang 'hi' \
    --emb_dim 300 \
    --cuda

The IndicNLP project provides a news article classification dataset for 9 languages (Bengali, Gujarati, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu).

1. Download the dataset: indicnlp-news-articles.tgz
2. Run evaluation: Use the txtcls.py script with the following arguments:

• --emb_path: Path to your trained embeddings.
• --data_dir: Path to the downloaded classification data.
• --lang: The language code (e.g., 'hi', 'bn').

python3 scripts/txtcls.py --emb_path <path> --data_dir <path> --lang <lang code>

You can evaluate embeddings on several public datasets (ACTSA, BBC News, IIT Patna Product/Movie Reviews, INLTK Headlines, and Bengali News) using the standard test/validation/test splits provided by AI4Bharat.

1. Download the splits: classification_public_datasets.tgz
2. Run evaluation: Use the txtcls.py script.

Arguments:

• --emb_path: Path to your trained embeddings.
• --data_dir: Path to the downloaded public datasets.
• --lang: The language code.

python3 scripts/txtcls.py --emb_path <path> --data_dir <path> --lang <lang code>

You can train word embeddings on the IndicNLP corpora using the FastText skipgram command. Ensure you have the $FASTTEXT_HOME environment variable set and your mono-lingual corpus path defined in $mono_path.

$FASTTEXT_HOME/build/fasttext skipgram \
	-epoch 10 -thread 30 -ws 5 -neg 10    -minCount 5 -dim 300 \
	-input $mono_path \
	-output $output_emb_prefix 

If you use any resources from the AI4Bharat-IndicNLP Corpus, please cite the following BibTeX entry:

@article{kunchukuttan2020indicnlpcorpus,
    title={AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages},
    author={Anoop Kunchukuttan and Divyanshu Kakwani and Satish Golla and Gokul N.C. and Avik Bhattacharyya and Mitesh M. Khapra and Pratyush Kumar},
    year={2020},
    journal={arXiv preprint arXiv:2005.00085},
}