DOCSEARCHbyAlgolia
Submit Docs

Indic NLP Catalog

repository·master·Indexed 20 days ago

https://github.com/ai4bharat/indicnlp_catalog

A collaborative repository of NLP datasets, models, benchmarks, and tools for languages of the Indian subcontinent. It features resources such as the Universal Language Contribution API (ULCA), large-scale datasets like IndicCorp and Samanantar, specialized libraries including the Indic NLP Library and iNLTK, and evaluation benchmarks like IndicGLUE and IndicNLG Suite. The catalog covers monolingual corpora, parallel translation and transliteration data, NER, QA, and summarization datasets, as well as pre-trained language models like IndicBERT and IndicBART.

Tokens
9.1K
Snippets
9
Records
26
Agent score
21%

What's inside indicnlp_catalog

  1. Overview of Featured Indic NLP Resources

    master

    The catalog highlights several key trends and resources in Indian language NLP:

    • Universal Language Contribution API (ULCA): A standard API and open scalable data platform (part of the Bhasini mission) for discovering and uploading Indian language datasets and models.
    • Large-scale Datasets: Examples include IndicCorp (9B tokens), Samanantar (50M parallel sentence pairs), Naamapadam (5.7M NER sentences), HiNER (100k NER sentences), and Aksharantar (26M transliteration pairs).
    • High Language Coverage: Resources like Aksharantar (21 languages), FLORES-200 (27 languages), and IndoWordNet (18 languages) are expanding coverage across most constitutional Indian languages.
    • Low-resource Support: Increasing focus on languages like Bodo, Kangri, and Khasi.
    • Key Contributors: Major groups include AI4Bharat, BUET CSE NLP, KMI, L3Cube, iNLTK, and IIT Patna.

  2. Find Monolingual Corpora for Indian Languages

    master

    The catalog provides several sources for monolingual text corpora used for training and evaluating NLP models for Indian languages:

    • AIBharat IndicCorp: Large-scale corpora.
      • v1: 8.9 billion tokens from 12 Indian languages (including Indian English).
      • v2: 20 billion tokens from 22 Indian languages (including Indian English).
    • Common Crawl based:
      • OSCAR Corpus: Processed CommonCrawl.
      • WMT Common Crawl Dumps: Raw, noisy text (2012-2016).
      • CC-100 Corpus: Facebook CommonCrawl extracted data, including romanized corpora for some Indian languages.
    • Other notable sources:
      • Wikipedia Dumps
      • LDCIL Monolingual Corpus
      • IIT Bombay Hindi Monolingual Corpus
      • Lot Of Indic Tweets Corpus: Large datasets for Telugu (7.9M) and Hindi (17.6M).
      • DNLP-Tel Telugu Corpus: 280M tokens and 23M sentences.
      • Kangri_corpus: 181,552 sentences for the Himachali language Kangri.

  3. Contribute to the Indic NLP Catalog

    master

    The Indic NLP Catalog is a collaborative project to build a comprehensive catalog of NLP datasets, models, and resources for Indian subcontinent languages. You can contribute by raising a pull request or an issue to add new resources.

    To propose a new entry, use the following format:

    [Resource Name](URL)

    Include a small, informative description of the dataset and links to any relevant papers, articles, or websites. Mention your name to be included in the CONTRIBUTORS.md list.

  4. Speech Models

    master

    Pre-trained speech models for Indic languages include:

    • AI4Bharat IndicWav2Vec: Multilingual pre-trained models for 40 Indian languages based on Wav2Vec 2.0.
    • Vakyansh CLSRIL-23: Pre-trained wav2vec2 models trained on 10,000 hours of speech data across 23 Indic languages.
    • Bengali Wav2Vec2: A specific wav2vec2-large-xlsr model trained on ~50 hours of OpenSLR Bengali data.
    - [AI4Bharat IndicWav2Vec](https://ai4bharat.iitm.ac.in/indic-wav-2-vec): Multilingual pre-trained models for 40 Indian languages based on Wav2Vec 2.0.
- [Vakyansh CLSRIL-23](https://github.com/Open-Speech-EkStep/vakyansh-models): Pretrained wav2vec2 model trained on 10,000 hours of Speech data in 23 Indic Languages [(documentation)](https://open-speech-ekstep.github.io/) [(experimentation platform)](https://github.com/Open-Speech-EkStep/vakyansh-wav2vec2-experimentation).
- [arijitx/wav2vec2-large-xlsr-bengali](https://huggingface.co/arijitx/wav2vec2-large-xlsr-bengali): Pretrained wav2vec2-large-xlsr trained on ~50 hrs(40,000 utterances) of OpenSLR Bengali data. Test WER 32.45% without LM.

  5. Find Parallel Transliteration Corpora

    master

    Resources for script conversion and transliteration tasks:

    • AI4Bharat Aksharantar: Largest publicly available transliteration dataset for 21 Indic languages (26M Indic-English pairs).
    • Dakshina Dataset: Text in both Latin and native scripts for 12 South Asian languages (~300k word pairs, 120k sentence pairs).
    • BrahmiNet Corpus: 110 language pairs mined from ILCI.
    • Xlit-Crowd: Hindi-English Transliteration Corpus via crowdsourcing.
    • AI4Bharat StoryWeaver Xlit Dataset: Transliteration datasets for Hindi, Maithili, and Konkani.

  7. Word Embedding Models

    master

    The catalog lists several resources for word embeddings, including FastText-based models for specific Indian languages and multilingual options:

    • AI4Bharat IndicFT: Fast-text embeddings for 11 Indian languages.
    • FastText variants: CommonCrawl+Wikipedia and Wikipedia pretrained vectors.
    • Polyglot: Multilingual embeddings.
    • EM-FT: FastText embeddings for Manipuri.
    • Sanskrit-Hindi-MT: Specific FastText embeddings for Sanskrit and Hindi.
    • UoM-Sinhala Sentiment Analysis: FastText 300 model for Sinhala.
    - [AI4Bharat IndicFT](https://ai4bharat.iitm.ac.in/indic-ft): Fast-text word embeddings for 11 Indian languages.
- [FastText CommonCrawl+Wikipedia](https://fasttext.cc/docs/en/crawl-vectors.html)
- [FastText Wikipedia](https://fasttext.cc/docs/en/pretrained-vectors.html)
- [Polyglot](https://sites.google.com/site/rmyeid/projects/polyglot)
- [EM-FT](http://catalog.elra.info/en-us/repository/browse/ELRA-W0316/): The first FastText word embedding available for Manipuri language trained on 1,880,035 Manipuri sentences.
- [Sanskrit-Hindi-MT](https://github.com/priyanshu2103/Sanskrit-Hindi-Machine-Translation): The FastText embeddings for Sanskrit is available [here](https://drive.google.com/file/d/1k5INFw9oaxV7yoWRg0qscmcFrOHVhdzW/view?usp=sharing) and for Hindi [here](https://drive.google.com/file/d/1Md9N7Ux2P9JCky1_9RgL2KjXRGb_lpXj/view?usp=sharing). 
- [UoM-Sinhala Sentiment Analysis- FastText 300](https://github.com/LahiruSen/sinhala_sentiment_anlaysis_tallip#word-embedding-models): The FastText word embedding model for Sinhala language. Described in [this paper](https://arxiv.org/pdf/2011.07280.pdf).

  8. Major Indic Language NLP Repositories

    master

  9. Indic NLP Libraries and Tools

    master

    A collection of specialized toolkits for Indian language processing:

    • Indic NLP Library: Python library for tokenization, sentence splitting, normalization, script conversion, and transliteration.
    • pyiwn: Python interface to IndoWordNet.
    • Indic-OCR: OCR specifically for Indic scripts.
    • CLTK: Toolkit for classical languages, including Sanskrit support.
    • iNLTK: Provides out-of-the-box support for various Indic NLP tasks.
    • Sanskrit Coders Indic Transliteration: Tools for script conversion and romanization.
    • Smart Sanskrit Annotator: Annotation tool for Sanskrit.
    • BNLP: Bengali language processing toolkit (tokenization, embedding, POS tagging, NER).
    • CodeSwitch: Support for code-mixed data (e.g., Hindi and Nepali) including LID, POS, NER, and sentiment analysis.
    • IndIE: Open Information Extraction (triple extractor) for Hindi (with conjectured support for Tamil, Telugu, and Urdu).
    • Hindi-BenchIE: Triple evaluation tool for Hindi sentences.

  11. Language Identification (LID) Models

    master

    For language identification tasks, you can use the NLLB-200 model, which supports 200 languages, including 27 Indic languages.

    - [NLLB-200](https://github.com/facebookresearch/fairseq/tree/nllb#lid-model): LID for 200 languages including 27 Indic languages.

  12. Indic NLP Evaluation Benchmarks

    master

    Standardized benchmarks for evaluating NLP models on Indian languages:

    • AI4Bharat IndicGLUE: NLU benchmark for 11 languages.
    • AI4Bharat IndicNLG Suite: NLG benchmark for 11 languages covering biography generation, summarization, headline generation, paraphrase generation, and question generation.
    • GLUECoS: Hindi-English code-mixed benchmark covering LID, POS, NER, Sentiment Analysis, QA, and NLI.
    • AI4Bharat Text Classification: Compilation of classification datasets for 10 languages.
    • WAT 2021 Translation Dataset: Standard train/test sets for English-to-10-Indian-languages translation.