RUAccent Python Library

repository·main·Indexed 41 days ago

https://github.com/den4ikai/ruaccent

A Python library for the automatic placement of accents in Russian text using neural networks and dictionaries. It features the RUAccent class for processing text via process_all(), a specialized process_yo() method for 'ё' homographs, and configurable model sizes ranging from 'tiny' to 'big_poetry'. The library supports CPU and CUDA acceleration and includes utilities for context-aware sentence splitting.

Tokens
2.8K
Snippets
11
Records
13
Agent score
52%

What's inside RUAccent

  1. Install RUAccent via pip or Git

    main

    You can install the ruaccent library using standard pip or directly from the GitHub repository.

    # Using pip
    pip install ruaccent
    
    # Using Git
    pip install git+https://github.com/Den4ikAI/ruaccent.git
  2. Process text with RUAccent

    main

    To use RUAccent, instantiate the RUAccent class, call load() to initialize the models, and then use process_all(text) to apply accents to a string.

    from ruaccent import RUAccent
    
    accentizer = RUAccent()
    # Initialize with specific settings
    accentizer.load(omograph_model_size='turbo3.1', use_dictionary=True, tiny_mode=False)
    
    text = 'на двери висит замок.'
    # Process the text
    print(accentizer.process_all(text))
  3. Configure RUAccent with the load() method

    main

    The load() method initializes the accentuation engine. You can configure model size, dictionary usage, custom dictionaries, hardware acceleration, and working directories.

    Hardware Requirements:

    • Minimum 512 MB RAM is required for stable operation (using the tiny omograph model).
    • For CUDA support, you must install onnxruntime-gpu and have CUDA configured on your system.
    accentizer.load(
        omograph_model_size='turbo2', 
        use_dictionary=True, 
        custom_dict={}, 
        device="CPU", 
        workdir=None
    )
  4. Reference: load() parameters

    main

    The following parameters are available for the load() method:

    ParameterTypeDescription
    omograph_model_sizestrSelects the model size. Available options: tiny, tiny2, tiny2.1, turbo2, turbo3, turbo3.1, turbo, big_poetry.
    use_dictionaryboolIf True, loads the full dictionary (requires more RAM). If False, all accents are placed by the neural network.
    custom_dictdictA dictionary for custom accentuation variants. Format: {'word': 'w+ord with acc+ent'}.
    devicestrHardware device to use. Options: "CPU" or "CUDA".
    workdirstrThe directory path where models are downloaded.
    tiny_modeboolIf True, disables the rule-based pipeline and some models, and does not load the accent dictionary.
  5. Process 'yo' (ё) homographs with `process_yo`

    main

    The process_yo method specifically handles the disambiguation of words involving the letter 'ё' (yo). It uses a specialized model to determine if a word should be pronounced with 'ё' or 'е' based on context.

    from ruaccent import RUAccent
    
    accentuator = RUAccent()
    accentuator.load()
    
    text = "Все были рады встрече."
    # This will resolve whether 'все' is 'всë' or 'все'
    result = accentuator.process_yo(text)
    print(result)
  6. Initialize the RUAccent class

    main

    To use RUAccent, instantiate the RUAccent class and then call the load method to download necessary models and dictionaries from Hugging Face and prepare the environment.

    By default, load will download models to the directory where the package is installed. You can specify a workdir to store models elsewhere and tiny_mode=True to use a lighter version of the accentuation logic.

    Parameters for load:

    • omograph_model_size (str): Size of the omograph model. Options include: 'big_poetry', 'medium_poetry', 'small_poetry', 'turbo', 'turbo2', 'turbo3', 'turbo3.1', 'tiny', 'tiny2', 'tiny2.1'. Default is 'turbo2'.
    • use_dictionary (bool): Whether to use the full dictionary for accentuation. Default is False.
    • custom_dict (dict): A dictionary to provide custom accentuation mappings.
    • custom_homographs (dict): A dictionary to provide custom homograph mappings.
    • device (str): The device to run models on (e.g., 'CPU', 'cuda'). Default is 'CPU'.
    • repo (str): The Hugging Face repository ID to download assets from. Default is 'ruaccent/accentuator'.
    • workdir (str, optional): The directory where models and dictionaries will be stored.
    • tiny_mode (bool): If True, uses a lighter mode (skips some heavy processing and uses accents_nn.json.gz). Default is False.
    from ruaccent import RUAccent
    
    accentuator = RUAccent()
    accentuator.load(
        omograph_model_size="turbo2",
        use_dictionary=True,
        custom_dict={"слово": "сл+ово"},
        device="CPU"
    )
  7. Accentuate Russian text with `process_all`

    main

    The process_all method is the primary entry point for accentuating text. It handles sentence splitting, omograph resolution (disambiguating words with multiple pronunciations), and applying stress marks (e.g., +).

    Parameters:

    • text (str): The input Russian text to process.
    • skip_regex (str, optional): A regular expression string. Any text matching this pattern will be skipped and left unaccentuated. This is useful for protecting specific entities or technical terms.

    Return Value:

    Returns the processed string with stress marks inserted.

    from ruaccent import RUAccent
    
    accentuator = RUAccent()
    accentuator.load()
    
    text = "Мама мыла раму."
    # You can also skip specific patterns
    result = accentuator.process_all(text, skip_regex=r'\d+')
    print(result)
  8. Determine sentence boundaries with is_sentence_end

    main

    The is_sentence_end function is a low-level heuristic engine used to decide how to handle a potential sentence boundary between a left string (the text before the dot) and a right string (the text after the dot).

    It returns one of three integer constants:

    • 0 (JOIN): The dot is part of an abbreviation or context that requires the text to stay together. Do not split.
    • 1 (MAYBE): The dot might be a boundary, but it depends on the following character (e.g., if the next word starts with a lowercase letter or a digit, it might not be a split).
    • 2 (SPLIT): The dot is a definitive sentence boundary.

    Parameters:

    • left (str): The text preceding the potential boundary.
    • right (str): The text following the potential boundary.
    • shortenings (Set[str]): Set of words that trigger MAYBE.
    • joining_shortenings (Set[str]): Set of words that trigger JOIN.
    • paired_shortenings (Set[Tuple[str, str]]): Set of word pairs that trigger MAYBE.
  9. Split text into sentences with split_sentences

    main

    The split_sentences function provides a basic sentence splitting mechanism using a regular expression (SENTENCE_SPLITTER). It is less sophisticated than split_by_sentences as it does not account for abbreviations or context-aware joining logic; it simply finds patterns ending in ., ?, !, or ….

    Returns:

    • List[str]: A list of stripped strings found by the regex pattern.
    from ruaccent.text_split import split_sentences
    
    text = "Hello! How are you?"
    sentences = split_sentences(text)
    # Output: ['Hello!', 'How are you?']
  10. Split text into sentences with split_by_sentences

    main

    The split_by_sentences function is the primary utility for segmenting a string into a list of individual sentences. It uses internal heuristics to distinguish between actual sentence boundaries and abbreviations or shortenings that end in dots.

    By default, it uses predefined sets of shortenings, joining_shortenings, and paired_shortenings to decide whether a dot indicates a split or a continuation.

    Parameters:

    • text (str): The input text to be split.
    • shortenings (Set[str]): A set of words that, when followed by a dot, suggest a potential sentence boundary (returns MAYBE). Defaults to SHORTENINGS.
    • joining_shortenings (Set[str]): A set of words that, when followed by a dot, strongly suggest the sentence should NOT be split (returns JOIN). Defaults to JOINING_SHORTENINGS.
    • paired_shortenings (Set[Tuple[str, str]]): A set of word pairs that, when appearing consecutively across a dot, suggest a potential boundary (returns MAYBE). Defaults to PAIRED_SHORTENINGS.

    Returns:

    • List[str]: A list of stripped sentence strings.
    from ruaccent.text_split import split_by_sentences
    
    text = "Привет! Как дела? Я учу Python."
    sentences = split_by_sentences(text)
    # Output: ['Привет!', 'Как дела?', 'Я учу Python.']
  11. Reference: Default shortening sets

    main

    The library uses these predefined sets to handle Russian and English abbreviations during sentence splitting.

    JOINING_SHORTENINGS = {
        'mr', 'mrs', 'ms', 'dr', 'vs', 'англ', 'итал', 'греч', 'евр', 'араб', 'яп', 'слав', 'кит',
        'тел', 'св', 'ул', 'устар', 'им', 'г', 'см', 'д', 'стр', 'корп', 'пл', 'пер', 'сокр', 'рис'
    }
    
    SHORTENINGS = {
        'co', 'corp', 'inc', 'авт', 'адм', 'барр', 'внутр', 'га', 'дифф', 'дол', 'долл', 'зав', 'зам', 'искл',
        'коп', 'корп', 'куб', 'лат', 'мин', 'о', 'обл', 'обр', 'прим', 'проц', 'р', 'ред', 'руб', 'рус', 'русск',
        'сан', 'сек', 'тыс', 'эт', 'яз', 'гос', 'мн', 'жен', 'муж', 'накл', 'повел', 'букв', 'шутл', 'ед'
    }
    
    PAIRED_SHORTENINGS = {('и', 'о'), ('т', 'е'), ('т', 'п'), ('у', 'е'), ('н', 'э')}