Razdel: Russian Language Tokenization

Domain Optimization

razdel is a rule-based system optimized for Russian news and fiction. It is designed to mimic segmentation found in datasets like SynTagRus, OpenCorpora, GICRYA, and RNC. It may perform less accurately on social media, scientific articles, or legal documents.

Error Measurement

Performance is measured by the absolute number of errors per 1000 tokens/sentences. An error is counted if a split is either missing or extra compared to the ground truth (etalon).

Substring Objects

Both tokenize and sentenize return Substring objects. These objects are useful because they preserve the exact character offsets (start, end) from the original input string, allowing you to map tokens back to their original positions.

Install razdel using pip. The library supports Python 3.7+ and PyPy 3.

$ pip install razdel

Use razdel.tokenize to split a string into individual tokens (words, numbers, punctuation). The function returns a list of Substring objects. Each Substring object contains the start index, end index, and the text of the token.

from razdel import tokenize

tokens = list(tokenize('Кружка-термос на 0.5л (50/64 см³, 516;...)'))
# To get just the text strings:
text_list = [_.text for _ in tokens]
print(text_list)
# Output: ['Кружка-термос', 'на', '0.5', 'л', '(', '50/64', 'см³', ',', '516', ';', '...', ')']

Use razdel.sentenize to split a block of text into a list of sentences. Like tokenize, it returns a list of Substring objects representing the segments of the original text.

from razdel import sentenize

text = '''
... - "Так в чем же дело?" - "Не ра-ду-ют".
И т. д. и т. п. В общем, вся газета
'''

sentences = list(sentenize(text))
# Output: [Substring(1, 23, '- "Так в чем же дело?"'), Substring(24, 40, '- "Не ра-ду-ют".'), ...]