DOCSEARCHbyAlgolia
Submit Docs

Slovnet

repository·master·Indexed 42 days ago

https://github.com/natasha/slovnet

A lightweight, high-performance Python library for Russian NLP tasks, including Named Entity Recognition (NER), morphology tagging, and syntax parsing. Optimized for CPU inference, it provides a smaller footprint than BERT-based models while maintaining competitive accuracy. Features include pre-trained models for news articles and tools for training custom NER models via distillation and synthetic data generation.

Tokens
14.3K
Snippets
44
Records
52
Agent score
54%

What's inside Slovnet

  1. Overview of Slovnet NLP models

    master

    Slovnet is a Python library providing deep-learning based NLP models specifically for the Russian language. It is designed to be lightweight and fast on CPUs, making it suitable for practical applications where large BERT models might be too slow or resource-intensive.

    Key features include:

    • NER (Named Entity Recognition): High speed (~25 news articles/sec) and small size (~30 MB), with accuracy only 1-2% lower than BERT SOTA.
    • Morphology and Syntax: High accuracy comparable to large SOTA BERT models, but significantly smaller (~30 MB) and much faster (~500 sentences/sec) on CPU.

  2. Understand Slovnet evaluation metrics

    master

    Slovnet is evaluated based on both quality metrics (like F1, UAS, LAS) and production-critical performance parameters. When choosing a model for your application, consider these four metrics:

    • init: The time (in seconds) between system launch and the first response. Lower is better for rapid scaling and testing.
    • disk: The file size (in MB) of all required artifacts (weights, embeddings, binaries, vocabs). Lower is better for compact deployments.
    • ram: The average CPU/GPU RAM usage (in MB).
    • speed: The processing throughput, measured in items per second (it/s). For NER, this is news articles (~1KB); for Morphology and Syntax, this is sentences per second.

  3. How the map and __call__ methods work in Slovnet annotators

    master

    Slovnet annotators (NER, Morph, Syntax) provide two primary ways to process data:

    1. __call__(text): A convenience method that processes a single string. Internally, it calls map with a list containing that one item.
    2. map(list_of_items): The primary method for high-throughput processing. It takes a list of items (e.g., a list of strings for NER, or a list of token lists for Morph/Syntax) and returns an iterator of the same size containing the markup objects.

    Batching: Internally, map processes items in batches of batch_size. The default is 8. Increasing batch_size improves CPU utilization but requires more RAM.

  4. Train a small NER model on synthetic data

    master

    This step trains a lightweight NER model using the large synthetic dataset generated in the previous step.

    1. Configure slovnet/scripts/05_ner/main.py

    Set the following variables in the script:

    • CUSTOM_TUNING = True
    • TAGS: The exact same tags used in the BERT training step.
    • CUSTOM_TEXTS: The filename of your large synthetic dataset (e.g., big-synthetic-dataset.jl.gz).
    • ID: The desired package name (e.g., slovnet_ner_custom_tags).

    2. Prepare and run slovnet/scripts/05_ner/main.ipynb

    • Dataset: Place your zipped synthetic dataset (.jl.gz) in the 05_ner/data/ directory.
    • Embeddings: Download the navec embeddings (navec_news_v1_1B_250K_300d_100q.tar) into the 05_ner/navec/ directory.
    • Output: Save the resulting encoder.pt, ner.pt, and shape.pt files into the 05_ner/model/ directory.
    # Configuration in slovnet/scripts/05_ner/main.py
CUSTOM_TUNING = True
TAGS = ['TAG1', 'TAG2']
CUSTOM_TEXTS = 'big-synthetic-dataset.jl.gz'
ID = 'slovnet_ner_custom_tags'

  6. Install development dependencies and run notebooks

    master

    To run the notebooks located in /slovnet/scripts, you must first install the development dependencies and the package itself in editable mode from the repository root.

    # Navigate to the repository root folder first
pip3 install -r requirements/dev.txt
pip3 install -e .
    pip3 install -r requirements/dev.txt
pip3 install -e .

  7. Perform Named Entity Recognition (NER) with Slovnet

    master

    Use the NER class to identify entities like locations (LOC), persons (PER), and organizations (ORG) in text.

    Workflow:

    1. Load a Navec model (e.g., navec_news_v1_1B_250K_300d_100q.tar).
    2. Load the NER model using NER.load().
    3. Link the models using ner.navec(navec).
    4. Call the annotator directly on text using ner(text) to get a markup object, or use .map() for batch processing.

    To visualize the results in an ASCII format, you can use ipymarkup.show_span_ascii_markup.

    from navec import Navec
from slovnet import NER
from ipymarkup import show_span_ascii_markup as show_markup

text = 'Your text here...'

navec = Navec.load('navec_news_v1_1B_250K_300d_100q.tar')
ner = NER.load('slovnet_ner_news_v1.tar')
ner.navec(navec)

markup = ner(text)
show_markup(markup.text, markup.spans)

  8. Train a large BERT NER model on custom data

    master

    This step involves training a large BERT-based NER model using a small, manually annotated dataset.

    1. Configure slovnet/scripts/02_bert_ner/main.py

    Set the following variables in the script:

    • CUSTOM_TUNING = True
    • TAGS: A list of your custom NER tags.
    • CUSTOM_TEXTS: The filename of your custom dataset (e.g., custom-dataset.jl.gz).

    2. Prepare and run slovnet/scripts/02_bert_ner/main.ipynb

    • Dataset Format: Annotate 1K-2K texts (~1000 symbols each) and combine them into a .jl file. Zip this file as .jl.gz and place it in the 02_bert_ner/data/ directory.
    • Pretrained Assets: Download the encoder.pt, vocab.txt, and emb.pt files into the 02_bert_ner/bert/ directory.
    • Output: After training, save the resulting encoder.pt and ner.pt files into the 02_bert_ner/model/ directory.
    # Configuration in slovnet/scripts/02_bert_ner/main.py
CUSTOM_TUNING = True
TAGS = ['TAG1', 'TAG2']
CUSTOM_TEXTS = 'custom-dataset.jl.gz'

  9. Perform Morphological Analysis with Slovnet

    master

    The Morph annotator provides morphological tags (e.g., NOUN, ADJ, VERB along with features like Case, Gender, Number) for tokens.

    Important: The Morph annotator processes tokenized text. You should use a tool like razdel to first split your text into sentences and then into tokens.

    Batch Processing: Use the .map() method to process a list of tokenized sentences. This method accepts a batch_size parameter (default is 8) to optimize CPU/RAM usage. .map() returns an iterator over markup objects.

    from razdel import sentenize, tokenize
from navec import Navec
from slovnet import Morph

# 1. Prepare tokenized chunks using razdel
chunk = []
for sent in sentenize(text):
    tokens = [_.text for _ in tokenize(sent.text)]
    chunk.append(tokens)

# 2. Load and configure Morph
navec = Navec.load('navec_news_v1_1B_250K_300d_100q.tar')
morph = Morph.load('slovnet_morph_news_v1.tar', batch_size=4)
morph.navec(navec)

# 3. Process
markup = next(morph.map(chunk))
for token in markup.tokens:
    print(f'{token.text:>20} {token.tag}')

  10. Access Slovnet documentation and technical deep dives

    master

    Detailed technical documentation and explanations regarding the underlying technology of Slovnet (such as distillation and quantization) are available in Russian.

    Key resources include:

    • An article covering distillation and quantization techniques used in Slovnet.
    • A video presentation from the Datafest 2020 talk covering the Slovnet section.

  11. Perform Syntactic Parsing with Slovnet

    master

    The Syntax annotator identifies dependency relations between tokens in a sentence.

    Workflow:

    1. Segment text into sentences and tokens using razdel.
    2. Load Navec and Syntax models.
    3. Link them via syntax.navec(navec).
    4. Use syntax.map(chunk) to process the tokenized sentences.

    Data Format: To convert the output into a format compatible with visualization tools like ipymarkup, you must map the token.head_id and token.id to 0-based indices. Note that you should skip the root node and self-loops (where source == target).

    from ipymarkup import show_dep_ascii_markup as show_markup
from razdel import sentenize, tokenize
from navec import Navec
from slovnet import Syntax

# 1. Prepare tokenized chunks
chunk = []
for sent in sentenize(text):
    tokens = [_.text for _ in tokenize(sent.text)]
    chunk.append(tokens)

# 2. Load and configure Syntax
navec = Navec.load('navec_news_v1_1B_250K_300d_100q.tar')
syntax = Syntax.load('slovnet_syntax_news_v1.tar')
syntax.navec(navec)

# 3. Process and convert to CoNLL-style indices for visualization
markup = next(syntax.map(chunk))
words, deps = [], []
for token in markup.tokens:
    words.append(token.text)
    source = int(token.head_id) - 1
    target = int(token.id) - 1
    if source > 0 and source != target:
        deps.append([source, target, token.rel])

show_markup(words, deps)

  12. Generate a large synthetic dataset using a BERT NER model

    master

    Once a large BERT NER model is trained, use it to automatically label a much larger corpus to create a synthetic dataset for training a smaller, more efficient model.

    Steps for slovnet/scripts/02_bert_ner/infer.ipynb

    1. Prepare Data: Gather a large dataset of 500K-1000K texts (~1000 symbols each).
    2. Setup: Ensure the vocab.txt pretrained tokens are in the 02_bert_ner/bert/ directory.
    3. Inference: Configure the model trained in the previous step and iterate through your texts, feeding them in small chunks (~1000 symbols each) to generate the markup.