DOCSEARCHbyAlgolia
Submit Docs

hazm

repository·master·Indexed 23 days ago

https://github.com/roshan-research/hazm

A comprehensive Python toolkit for Persian Natural Language Processing (NLP) version 0.12.1. It provides tools for normalization (including InformalNormalizer), tokenization, stemming, lemmatization, POS tagging, chunking, dependency parsing, and sequence labeling. The library supports word and sentence embeddings and includes a variety of specialized corpus readers for loading Persian datasets. It integrates with Hugging Face for automatic downloading of pretrained models.

Tokens
13.9K
Snippets
27
Records
106
Agent score
79%

What's inside hazm

  1. Use Corpus Readers to load Persian datasets

    master

    Hazm provides a collection of specialized reader classes to facilitate loading popular Persian corpora into a format suitable for NLP tasks. These readers automate the parsing and preprocessing of raw data from various sources, saving time during the data ingestion phase of machine learning or pattern extraction workflows.

    Note that these readers are provided as utility tools to facilitate developer work and are not considered part of the core Hazm library logic.

    Available readers include:

    • hamshahri_reader
    • mirastext_reader
    • quran_reader
    • bijankhan_reader
    • dadegan_reader
    • universal_dadegan_reader
    • degarbayan_reader
    • persica_reader
    • persian_plain_text_reader
    • peykare_reader
    • sentipers_reader
    • tnews_reader
    • treebank_reader
    • verbvalency_reader
    • wikipedia_reader
    • mizan_reader
    • ner_reader
    • naab_reader
    • arman_reader
    • faspell_reader

  2. Use hazm.embedding for Persian word and sentence embeddings

    master

    The hazm.embedding module provides tools for generating vector representations (embeddings) for Persian text. It includes classes for individual word embeddings, sentence embeddings, and processing entire corpora.

    Available components:

    • WordEmbedding: For generating embeddings for individual words.
    • SentEmbedding: For generating embeddings for single sentences.
    • SentenceEmbeddingCorpus: For generating embeddings for a collection of sentences (corpus).

  3. Use pretrained models from Hugging Face

    master
    Hazm supports automatic downloading of pretrained models (such as POS Tagger, Chunker, and Embeddings) from Hugging Face. When initializing a model class, provide the repo_id and model_filename to trigger the download and caching process.

  4. Load pretrained models for Hazm

    master

    Advanced tasks like POS tagging, Chunking, and Dependency Parsing require pretrained models. You can load them in two ways:

    1. Automatic Loading (Hugging Face Hub)

    Provide a repo_id and model_filename to automatically download and cache the model from the Hugging Face Hub.

    2. Manual Loading (Offline)

    Download the model files manually and provide the local file path to the model parameter in the class constructor.

    from hazm import POSTagger

# Automatic Loading from Hugging Face
tagger = POSTagger(
    repo_id="roshan-research/hazm-postagger", 
    model_filename="pos_tagger.model"
)

# Manual Loading from a local path
tagger = POSTagger(model="path/to/your/pos_tagger.model")

  6. Install Hazm

    master

    To install the latest version of Hazm, you need Python 3.12 or higher. Run the following command:

    pip install hazm

    To enable automatic downloading and caching of pretrained models from Hugging Face, you must also install the huggingface-hub package:

    pip install huggingface-hub
    pip install hazm
pip install huggingface-hub

  9. Customizing feature generation with data_maker

    master

    Both SequenceTagger and IOBTagger allow you to provide a custom data_maker function during initialization. This function determines how tokens are converted into feature dictionaries for the CRF model.

    Default behaviors:

    • SequenceTagger uses data_maker (default) which extracts basic features: word, is_first, is_last, is_num, prev_word, and next_word.
    • IOBTagger uses iob_data_maker which includes POS tags: word, is_first, is_last, is_num, prev_word, next_word, pos, prev_pos, and next_pos.

  10. Use SpacyChunker for high-performance chunking

    master

    The SpacyChunker is a specialized implementation of Chunker that leverages the spaCy library. It supports GPU acceleration and can be initialized from a local spaCy model or a Hugging Face repository.

    Note: To use this, you must install hazm[all] to ensure spacy is available.

    Key Features:

    • GPU Support: Use using_gpu=True and specify gpu_id to accelerate processing.
    • Batch Processing: parse_sents supports a batch_size parameter for efficient processing of large datasets.

  12. Install dependencies for Named Entity Recognition (NER)

    master

    The HazmNER class requires spacy to function. If you encounter an ImportError when initializing HazmNER, install the full hazm package with spaCy support using the following command:

    pip install hazm[all]