DOCSEARCHbyAlgolia
Submit Docs

nlprule

repository·main·Indexed 20 days ago

https://github.com/bminixhofer/nlprule

A fast, low-resource Natural Language Processing and Error Correction library written in Rust, with Python bindings. It supports English, German, and Spanish, providing grammatical error correction, sentence segmentation, POS tagging, lemmatization, and chunking. The library includes a Tokenizer for linguistic analysis and a Rules system for generating and applying grammatical suggestions.

Tokens
12K
Snippets
37
Records
51
Agent score
70%

What's inside nlprule

  1. Install and use nlprule in Rust

    main

    nlprule is a high-performance Rust library. For the best experience, use the nlprule-build crate in your build.rs to compile the necessary language binaries during the build process.

    Dependency Setup

    Add both nlprule and nlprule-build to your Cargo.toml. Important: The versions of nlprule and nlprule-build must be identical.

    Build Script (build.rs)

    Use nlprule_build::BinaryBuilder to build and validate the language binaries (e.g., for "en") and place them in the OUT_DIR.

    Application Code

    In your main application, use the tokenizer_filename! and rules_filename! macros to locate the compiled binaries in the OUT_DIR. You can then load them using Tokenizer::from_reader and Rules::from_reader.

    // Cargo.toml
[dependencies]
nlprule = "0.6.4"

[build-dependencies]
nlprule-build = "0.6.4"

// build.rs
fn main() -> Result<(), nlprule_build::Error> {
    println!("cargo:rerun-if-changed=build.rs");
    nlprule_build::BinaryBuilder::new(
        &["en"],
        std::env::var("OUT_DIR").expect("OUT_DIR is set"),
    )
    .build()?
    .validate()
}

// src/main.rs
use nlprule::{Rules, Tokenizer, tokenizer_filename, rules_filename};

fn main() {
    let mut tokenizer_bytes = include_bytes!(concat!(env!("OUT_DIR"), "/", tokenizer_filename!("en")));
    let mut rules_bytes = include_bytes!(concat!(env!("OUT_DIR"), "/", rules_filename!("en")));

    let tokenizer = Tokenizer::from_reader(&mut tokenizer_bytes).unwrap();
    let rules = Rules::from_reader(&mut rules_bytes).unwrap();

    let corrected = rules.correct("She was not been here since Monday.", &tokenizer);
    assert_eq!(corrected, "She was not here since Monday.");
}

  2. Install and use nlprule in Python

    main

    You can use nlprule in Python for grammatical error correction and text processing.

    Installation

    Install via pip:

    pip install nlprule

    Basic Usage

    To perform error correction and suggestions, you must first load a Tokenizer and then load Rules using that tokenizer.

    Error Correction and Suggestions

    • rules.correct(text): Returns a corrected version of the input string.
    • rules.suggest(text): Returns an iterator of suggestion objects containing the error span, replacements, and a message.

    Tokenization

    Use tokenizer.pipe(text) to process text into sentences and tokens. Each token provides access to its text, span, tags (POS), lemmas, and chunks.

    from nlprule import Tokenizer, Rules

tokenizer = Tokenizer.load("en")
rules = Rules.load("en", tokenizer)

# Correct text
print(rules.correct("He wants that you send him an email."))

# Get suggestions
for s in rules.suggest("She was not been here since Monday."):
    print(s.start, s.end, s.replacements, s.source, s.message)

# Tokenization pipeline
for sentence in tokenizer.pipe("A brief example is shown."):
    for token in sentence:
        print(token.text, token.span, token.tags, token.lemmas, token.chunks)

  3. Identify rules with Category, Group, and Index

    main

    The rule hierarchy is represented by three main structs:

    • Category: The top-level identifier.
    • Group: A sub-division within a category. Created by calling Category::join("group_name").
    • Index: A specific rule within a group. Created by calling Group::join(index_number).

    Identifiers are case-insensitive when used for matching.

  4. Use IncompleteToken to build a complete Token

    main

    An IncompleteToken<'t> is a temporary structure used during the sentence construction phase. It contains a Word, a Span, and metadata like is_sentence_end and has_space_before.

    When you call into_token(), the following happens automatically:

    1. The token is converted to a Token<'t>.
    2. A default WordData is added with a SpecialPos::None tag.
    3. If no other POS tags exist, a SpecialPos::Unknown tag is added.
    4. If is_sentence_end was true, a SpecialPos::SentEnd tag is added.
    // Conceptual conversion
let incomplete_token: IncompleteToken = ...;
let token: Token = incomplete_token.into_token();

  5. Understand the Sentence lifecycle: IncompleteSentence to Sentence

    main

    In nlprule, text processing follows a lifecycle where a sentence starts as an IncompleteSentence and is eventually converted into a frozen Sentence.

    • IncompleteSentence<'t>: Represents a sentence where tokens may still be undergoing processing (e.g., tagging). It holds a reference to the original text and a Tagger.
    • Sentence<'t>: A completed sentence where all token information is set and frozen.

    To transition from an incomplete state to a complete one, use the into_sentence() method. This consumes the IncompleteSentence and produces a Sentence by converting all IncompleteTokens into Tokens.

    // Conceptual flow
let incomplete: IncompleteSentence = ...;
let sentence: Sentence = incomplete.into_sentence();

  6. Use owned data types for long-lived structures

    main

    Most core types in nlprule (like WordId, PosId, Word, Token) are designed with lifetimes ('t) to refer to original text for performance. However, if you need to store these results in long-lived structures (like rule test results or databases) that outlive the original text buffer, use the owned module.

    • owned::WordId
    • owned::PosId
    • owned::WordData
    • owned::Word
    • owned::Token

    Most referential types provide a to_owned_*() method to perform this conversion.

    // Converting a referential token to an owned one
let owned_token = token.to_owned_token();

  7. How to select and manipulate rules using Selectors

    main

    Rules in nlprule are organized in a three-layer hierarchy: Category -> Group -> Index. You can use a Selector to target specific rules or entire groups of rules for operations like enabling or disabling them using the Rules::select_mut method.

    There are three ways to create a Selector:

    1. Using Structs: Construct Category, Group, or Index and convert them into a Selector using .into().
    2. Using String Syntax: Use a slash-separated string (e.g., "category/group/index") and convert it using .try_into().
    3. Chaining: Use the .join() method on existing identifiers to traverse the hierarchy.

    Note: Comparisons for Category and Group are case-insensitive.

    use nlprule::{Tokenizer, Rules, rule::id::Category};
use std::convert::TryInto;

let tokenizer = Tokenizer::new("path/to/en_tokenizer.bin")?;
let mut rules = Rules::new("path/to/en_rules.bin")?;

// 1. Select via struct hierarchy
// Disable rules named "confusion_due_do" in category "confused_words"
rules
    .select_mut(
        &Category::new("confused_words")
            .join("confusion_due_do")
            .into(),
    )
    .for_each(|rule| rule.disable());

// 2. Select via category level (disables all rules in that category)
disables all grammar rules
rules
    .select_mut(&Category::new("grammar").into())
    .for_each(|rule| rule.disable());

// 3. Select via string syntax (slash-separated)
// Enable rules using the string path
rules
    .select_mut(&"confused_words/confusion_due_do".try_into()?)
    .for_each(|rule| rule.enable());
# Ok::<(), nlprule::Error>(())

  8. Understand the Tagger's dictionary structure

    main

    The Tagger is built from a dictionary where each line associates a word with one or more pairs of (lemma, POS tag).

    Example dictionary format:

    actualize   actualize   VB
actualize   actualize   VBP
actualized  actualize   VBD
actualized  actualize   VBN
actualizes  actualize   VBZ
actualizing actualize   VBG
actually    actually    RB

    Internally, the Tagger uses:

    • A POS bimap: Maps POS tag strings to 16-bit IDs.
    • A Word bimap: Maps known words to 32-bit IDs.
    • A Tags map: Associates a WordId with a list of (lemma_id, pos_id) pairs.

  9. How the Tokenizer pipeline works

    main

    The Tokenizer follows a specific sequence of operations to transform raw text into structured linguistic data:

    1. Sentencization: The text is split into segments using the internal sentencizer rules.
    2. Tokenization: Each segment is split into individual tokens based on whitespace and language-specific rules.
    3. Tagging: Tokens are assigned lemmas and part-of-speech (POS) tags via a Tagger.
    4. Chunking: A Chunker (if present) identifies noun/verb phrases and grammatical cases.
    5. Multiword Tagging: A MultiwordTagger (if present) handles complex multi-token units.
    6. Disambiguation: DisambiguationRules are applied to resolve ambiguities in the initial tagging (e.g., determining if a word is a noun or a verb based on context).

  10. Configure Unification constraints for rules

    main

    A Unification can be attached to a Rule or DisambiguationRule to invalidate a match if certain conditions are not met.

    • It uses a mask (a vector of Option<bool>) to determine which groups in the match graph are subject to the unification.
    • It uses filters (a vector of Vec<PosFilter>) to define the required properties for those tokens.
    • If the unification is negated, the match is kept only if the filters do not match. Otherwise, it is kept only if they do match.

  11. Understand the difference between Rule and DisambiguationRule

    main

    In nlprule, there are two primary types of rules:

    1. Rule (Grammar Rule): Used for finding errors and providing corrections. When a match is found, it produces Suggestions (e.g., "Did you mean 'doesn't'?"). It is primarily used for error detection and correction.

    2. DisambiguationRule: Used for refining the linguistic analysis of a sentence. Instead of suggesting a replacement to a user, it changes the internal metadata (like Part-of-Speech tags or lemmas) of the tokens to resolve ambiguity (e.g., determining if 'have' is a verb or a noun based on context).

  12. Initialize Grammatical Rules in Python

    main

    The Rules class manages grammatical rules and provides suggestions for correcting text. It requires a Tokenizer instance to function.

    1. From a language code: Downloads the rules binary and uses the provided Tokenizer.
    2. From a local file path: Loads rules from a specific .bin file.

    Note: When loading by language code, the Tokenizer must be passed as an argument.

    import nlprule

tokenizer = nlprule.Tokenizer.load("en")

# Method 1: Load rules for a language
rules = nlprule.Rules.load("en", tokenizer)

# Method 2: Load from a local binary file
rules = nlprule.Rules("/path/to/rules.bin", tokenizer)