DadmaTools Persian NLP Library

When using the Pipeline, the resulting doc is a spaCy Doc object. You can access standard spaCy attributes and custom DadmaTools extensions directly through the object.

Standard spaCy attributes:

• doc.text, doc.ents, doc.sents
• token.pos_, token.lemma_, token.dep_, token.head

Custom DadmaTools extensions:

• token._.kasreh: Access Kasreh Ezafe information.
• doc._.lang: Access the language attribute.
• doc._.sentiment: Access sentiment analysis results.
• doc._.spellchecker: Access spellchecker results.

# Access token information
for token in doc:
    print(f"Token: {token.text}, POS: {token.pos_}, Lemma: {token.lemma_}, Dep: {token.dep_")

# Access entities
for ent in doc.ents:
    print(f"Entity: {ent.text}, Type: {ent.label_}")

# Access custom attributes
print(f"Language: {doc._.lang}")
print(f"Sentiment: {doc._.sentiment}")

# Use spaCy utilities
from spacy import displacy
displacy.serve(doc, style="dep")

You can interact with Persian word embeddings using the dadmatools.embeddings module. This allows you to retrieve embedding information, load specific models, and perform vector operations like finding nearest neighbors, calculating similarity, or generating embeddings for text.

Workflow

1. Inspect available models: Use get_all_embeddings_info() to see all supported models.
2. Get model metadata: Use get_embedding_info(name) to get details about a specific model.
3. Load a model: Use get_embedding(name) to load the embedding object.
4. Perform operations: Once loaded, you can access word vectors, retrieve the vocabulary, find nearest neighbors, check similarity, or embed full text strings.

from dadmatools.embeddings import get_embedding, get_all_embeddings_info, get_embedding_info
from pprint import pprint

# 1. Inspect all available embeddings
pprint(get_all_embeddings_info())

# 2. Get info for a specific model
embedding_info = get_embedding_info('glove-wiki')

# 3. Load the embedding model
word_embedding = get_embedding('glove-wiki')

# 4. Perform operations
# Get vector for a specific word
print(word_embedding['سلام'])

# Get the vocabulary
vocab = word_embedding.get_vocab()

# Find top 10 nearest neighbors for a word
print(word_embedding.top_nearest("زمستان", 10))

# Calculate similarity between two words
print(word_embedding.similarity('کتب', 'کتاب'))

# Generate embedding for a full text string
print(word_embedding.embedding_text('امروز هوای خوبی بود'))

DadmaTools provides an easy way to load popular Persian NLP datasets. Each dataset is provided as an iterator (e.g., .train, .test, .dev, or .data) and can be used in loops or as generators.

Supported datasets include:

• Named Entity Recognition (NER): PersianNER, ARMAN, Peyma
• Textual Entailment: FarsTail
• Spell Checking: FaSpell
• Text Classification: PersianNews
• Universal Dependency: PerUDT
• Text Summarization: PnSummary
• Sentiment Classification: SnappfoodSentiment
• Text Translation (eng-fa): TEP
• Corpus: WikipediaCorpus, PersianTweets

from dadmatools.datasets import FarsTail, SnappfoodSentiment, Peyma, PerUDT, PersianTweets, PnSummary

# Initialize a dataset
farstail = FarsTail()

# Access splits (e.g., train, test, dev)
print(len(farstail.train))
print(next(farstail.train))

# Loop over items in a split
snappfood_sa = SnappfoodSentiment()
for i, item in enumerate(snpfood_sa.test):
    print(item['comment'], item['label'])

DadmaTools supports glove, fasttext, and word2vec formats. To use fasttext-based embeddings, you must first install the fasttext library via pip install fasttext.

Supported embeddings:

• glove-wiki: GloVe algorithm trained on Wikipedia
• fasttext-commoncrawl-bin: FastText algorithm trained on CommonCrawl (binary format)
• fasttext-commoncrawl-vec: FastText algorithm trained on CommonCrawl (vector format)
• word2vec-conll: Word2Vec algorithm trained on Persian CoNLL17 corpus

Once loaded, you can perform vector lookups, find nearest neighbors, calculate similarity, or generate embeddings for entire text strings.

from dadmatools.embeddings import get_embedding, get_all_embeddings_info, get_embedding_info

# Get info on all available embeddings
print(get_all_embeddings_info())

# Load a specific embedding (e.g., glove-wiki)
word_embedding = get_embedding('glove-wiki')

# Get vector for a word
vector = word_embedding['سلام']

# Get vocabulary
vocab = word_embedding.get_vocab()

# Useful operations
print(word_embedding.top_nearest("زمستان", 10))
print(word_embedding.similarity('کتب', 'کتاب'))
print(word_embedding.embedding_text('امروز هوای خوبی بود'))

The dadmatools.datasets module provides programmatic access to several publicly available Persian NLP datasets. Each dataset class typically exposes splits such as .train, .test, .dev, or .data.

Datasets are designed to be iterable, similar to generators. You can access individual items using next() or loop over them directly. Some datasets also provide an .info attribute for metadata.

Available Datasets

from dadmatools.datasets import FarsTail, SnappfoodSentiment, Peyma, PerUDT, PersianTweets, PnSummary

# Accessing splits and iterating
farstail = FarsTail()
print(len(farstail.train))  # Get length of train split
print(next(farstail.train)) # Get next item

# Accessing dataset info
pn_summary = PnSummary()
print(pn_summary.info)

# Looping over test split
snappfood_sa = SnappfoodSentiment()
for i, item in enumerate(snpfood_sa.test):
    print(item['comment'], item['label'])

# Accessing specific fields (e.g., lemma in PerUDT)
perudt = PerUDT()
for token_list in perudt.dev:
    print(token_list[0]['lemma'])

# Accessing NER tags in Peyma
peyma = Peyma()
print(next(peyma.data)[0]['tag'])

# Accessing corpus data
tweets = PersianTweets()
print(len(tweets.data))
print(next(tweets.data))

For advanced NLP tasks like Part-of-Speech (POS) tagging, dependency parsing, constituency parsing, and Named Entity Recognition (NER), use the Pipeline class from dadmatools.pipeline_v1.language.

When you initialize a Pipeline with specific tasks, it automatically loads the necessary components (like the lemmatizer, POS tagger, and tokenizer).

Workflow:

1. Define the required tasks as a comma-separated string (e.g., 'pos,dep,cons,ner').
2. Initialize the Pipeline.
3. Pass text to the pipeline object to receive a spaCy Doc object.
4. Use language.to_json() to export analysis results to a dictionary format.
5. Access NER results via the doc._.ners attribute.

import dadmatools.pipeline_v1.language as language

# Define tasks: pos (part-of-speech), dep (dependency), cons (constituency), ner (named entity recognition)
pips = 'pos,dep,cons,ner'
nlp = language.Pipeline(pips)

# Inspect the loaded pipeline components
print(nlp.analyze_pipes(pretty=True))

# Process text - returns a spaCy Doc object
doc = nlp('از قصهٔ کودکیشان که می‌گفت، گاهی حرص می‌خورد!')

# Export analysis to JSON-compatible dictionary
dictionary = language.to_json(pips, doc)
print(dictionary)

# Access NER results specifically
print(doc._.ners)