BNLP: Bengali Natural Language Processing Toolkit

The LanguageDetector class provides high-performance language identification using FastText. It automatically downloads the required model on its first use unless configured otherwise.

Key capabilities:

• Single detection: Detect the primary language and confidence score.
• Top-K predictions: Retrieve multiple possible language candidates.
• Bengali check: Quickly verify if text is Bengali.
• Batch processing: Detect languages for a list of strings efficiently.
• Mixed language detection: Identify proportions of different languages in code-mixed text (e.g., Bengali-English).

from bnlp import LanguageDetector

detector = LanguageDetector()
result = detector.detect("আমি বাংলায় গান গাই")
print(result.language)      # 'bn'
print(result.confidence)    # ~0.99
print(result.is_bengali)    # True

The project roadmap includes support for modern Bengali transformer models. Developers looking for high-performance embeddings can look for integration with models like BanglaBERT or BanglaT5 (e.g., from sagorsarker/bangla-bert-base or csebuetnlp/csebuetnlp).

Proposed usage pattern:

from bnlp.transformers import BanglaBERT, BanglaT5

model = BanglaBERT()
embeddings = model.encode("আমি বাংলায় গান গাই।")

You can install the BNLP toolkit using pip. It supports Python versions 3.8 through 3.13 on Linux, Windows, and Mac.

To install:

pip install bnlp_toolkit

To upgrade to the latest version:

pip install -U bnlp_toolkit

To upgrade your existing installation to the latest version, use the -U flag with pip.

pip install -U bnlp_toolkit

If you prefer building from source, clone the repository and use setup.py:

git clone https://github.com/sagorbrur/bnlp.git
cd bnlp
python setup.py install

BNLP provides two pretrained BengaliDoc2vec models that are automatically downloaded when initialized. You can choose between a model trained on news articles or one trained on Wikipedia.

To use them, pass the corresponding identifier to the model_path argument of BengaliDoc2vec():

• NEWS_DOC2VEC: Model trained on News Article dataset.
• WIKI_DOC2VEC: Model trained on Wikipedia Dump dataset.

If no model_path is provided, NEWS_DOC2VEC is selected by default.

from bnlp import BengaliDoc2vec

# Use the news-trained model
model_key = "NEWS_DOC2VEC"
bn_doc2vec = BengaliDoc2vec(model_path=model_key)

To use the language detection features (which rely on FastText), you must install the bnlp_toolkit with the langdetect extra.

pip install bnlp_toolkit[langdetect]

To use FastText features in BNLP, you must install the fasttext library. You can do this via standard pip or by using the BNLP toolkit extra:

pip install fasttext==0.9.2
# OR
pip install bnlp_toolkit[fasttext]

Note for Windows users: You may need to follow specific instructions for building the fasttext python wrapper from source.

You can load your own trained Doc2Vec model by passing the local file path to the model_path argument in BengaliDoc2vec().

Note: Ensure that any associated .npy model files are kept in the same directory as the .pkl model file.

from bnlp import BengaliDoc2vec

own_model_path = "own_directory/own_doc2vec_model.pkl"
bn_doc2vec = BengaliDoc2vec(model_path=own_model_path)

document = "আপনার টেক্সট"
vector = bn_doc2vec.get_document_vector(document)

When initializing LanguageDetector, you can pass several configuration parameters:

• threshold: Set a custom confidence threshold.
• model_path: Provide a path to a specific FastText model file (e.g., lid.176.ftz).
• auto_download: Set to False to prevent automatic model downloading.

You can also use detector.get_language_name(lang_code) to retrieve the full name of a language code.

from bnlp import LanguageDetector

# Custom threshold
detector = LanguageDetector(threshold=0.7)

# Custom model path
detector = LanguageDetector(model_path="/path/to/lid.176.ftz")

# Disable auto-download
detector = LanguageDetector(auto_download=False)

# Get language name
print(detector.get_language_name('bn'))  # 'Bengali'