DOCSEARCHbyAlgolia
Submit Docs

Simplemma: Multilingual Lemmatizer for Python

repository·main·Indexed 42 days ago

https://github.com/adbar/simplemma

A fast, zero-dependency Python library for lemmatization, tokenization, and sentence splitting supporting 54 languages. It provides tools for language detection, vocabulary checking, and memory-efficient processing via configurable dictionary backends like DefaultDictionaryFactory, TrieDictionaryFactory, and StreamDictionaryFactory.

Tokens
16.7K
Snippets
64
Records
84
Agent score
52%

What's inside Simplemma

  1. Overview of Simplemma capabilities

    main

    Simplemma provides several text processing tasks through a unified entry point system. The core capabilities include:

    • Lemmatization: Reducing words or entire texts to their base form using lemmatize() or text_lemmatizer().
    • Tokenization: Splitting text into individual tokens using simple_tokenizer().
    • Sentence Splitting: Breaking text into sentences using split_sentences().
    • Language Detection: Identifying the language of a text via langdetect() or checking if text matches a specific language using in_target_language().
    • Vocabulary Checking: Verifying if a word exists in the dictionary using is_known().

  2. How the rulebuilder recipe works

    main

    The rulebuilder tool uses a four-step pipeline to generate high-precision lemmatization rules. You can run the entire process via python -m training.rulebuilder <lang> or drive the steps manually for fine-grained control.

    1. mine(lang): Identifies candidate (suffix, target) cells that meet the initial precision/support threshold.
    2. trim_by_mass(cells, 0.70): Keeps only the highest-firing cells covering 70% of the firing mass, dropping the low-value tail to speed up subsequent refinement.
    3. refine(cells, dictionary): Builds rules and iteratively drops cells that are imprecise or become under-supported once more general rules intercept their tokens.
    4. subsume(rules, dictionary): Removes redundant rules (e.g., rules that are already covered by an earlier or more general group) to ensure a clean, non-overlapping set.
    # Run the full pipeline for a specific language
python -m training.rulebuilder <lang>

  3. Understand Simplemma's casing heuristics

    main

    Simplemma applies specific casing heuristics during the lemmatization process to ensure accuracy. It follows three main rules:

    1. Sentence-initial lowering: A capital letter at the start of a sentence is treated as positional rather than lexical. Simplemma lowers these characters before performing a word lookup to ensure the lemma is correctly identified.
    2. Proper noun protection: The library uses per-language gates to identify and spare probable proper nouns from being lowered.
    3. Acronym preservation: Words that are entirely in ALL-CAPS are treated as acronyms and are kept verbatim without being lowered.

  4. Use StreamDictionaryFactory for low-memory lemmatization

    main

    The StreamDictionaryFactory is the backend implementation used when low_memory=True is specified. It is designed for environments with limited RAM by reading the shipped front-coded dictionary streams directly from disk instead of loading the entire dictionary into memory.

    Key characteristics:

    • Low Memory Footprint: Avoids building a full dictionary in RAM.
    • Zero Dependencies: Uses only the Python standard library.
    • No On-Disk Cache: Does not require or create additional cache files on disk.
    # Example conceptual usage (implementation detail depends on the caller)
# Setting low_memory=True triggers the use of StreamDictionaryFactory
import simplemma

# When initializing a language or strategy with low_memory=True,
# simplemma uses the StreamDictionaryFactory internally.

  6. Understand Simplemma accuracy caveats

    main

    Because Simplemma operates without morphosyntactic information (part-of-speech tagging), there are inherent limitations:

    • Ambiguity: The algorithm cannot always distinguish between different grammatical roles of the same word (e.g., a word that is both a noun and a verb). It may choose one valid form over another.
    • Morphological limits: Diminutive forms or highly specific variations not present in the model data may remain unchanged.
    • Participles vs Adjectives: It may struggle to disambiguate past participles from verb-derived adjectives.
    • Frequent words: Some short, frequent words (pronouns, conjunctions) might require manual post-processing.

    To improve coverage for morphologically-rich languages, you can use the greedy argument, which triggers a decomposition algorithm that acts similarly to a stemmer.

  7. Use the DefaultStrategy lemmatization pipeline

    main

    The DefaultStrategy is the standard lemmatization pipeline used by simplemma unless you define a custom one. It follows a specific sequence of operations to decompose and lemmatize words:

    1. Dictionary lookup: Attempts to find the word in a pre-defined dictionary.
    2. Decomposition: If not found, it applies a series of decomposition strategies in order: clitic, apostrophe, hyphen, prefix, affix, and morpheme.
    3. Rules: Applies per-language specific rules.

    You can customize the behavior of this strategy using the dictionary_factory or by enabling the low_memory flag to reduce the memory footprint.

    # Note: This strategy is used by default when calling lemmatize()
# but can be referenced via:
from simplemma.strategies.default import DefaultStrategy

  8. Use custom classes for advanced lemmatization and language detection

    main

    While simple functions are available for basic tasks, you can instantiate Lemmatizer and LanguageDetector classes directly for fine-grained control. Both classes accept a lemmatization_strategy argument, which must implement the LemmatizationStrategy protocol.

    Core components include:

    • Lemmatizer: Handles the lemmatization process.
    • LanguageDetector: Handles language detection.
    • DefaultStrategy: A composite strategy that combines multiple underlying strategies.
    • DictionaryLookupStrategy: A strategy that performs lookups in a dictionary.
    • DictionaryFactory: Used to build dictionaries for lookup strategies.
    from simplemma import LanguageDetector, Lemmatizer
from simplemma.strategies import DefaultStrategy

# Example of manual instantiation
strategy = DefaultStrategy()
lemmatizer = Lemmatizer(lemmatization_strategy=strategy)
detector = LanguageDetector(lemmatization_strategy=strategy)

  9. Compare Simplemma dictionary backends

    main

    Simplemma provides three dictionary backends that allow you to trade memory usage for speed. The choice depends on whether you prioritize peak RAM, load time, or lookup speed.

    BackendPeak RAMLoad timeUncached lookupCached lookupExtra dependency
    DefaultDictionaryFactory~175 MB~0.6 sfastest (baseline)fastest (baseline)none
    TrieDictionaryFactory~30 MB~1 ms warm¹~2.5× slower~1.2× slowermarisa-trie
    StreamDictionaryFactory~50 MB~0.6 s~18× slower~6× slowernone

    ¹ Note on TrieDictionaryFactory: The first use of a language builds its trie from the shipped dictionary, which takes a few seconds and briefly requires as much memory as DefaultDictionaryFactory. It then caches the result on disk. If your machine lacks the memory to perform this initial build, you can build the trie on a different machine with the same CPU architecture and copy the cache directory over.

    Key Trade-offs:

    • DefaultDictionaryFactory: Fastest performance, but highest memory usage. Memory usage compounds with every additional language loaded.
    • TrieDictionaryFactory: Lowest steady-state footprint, but requires the marisa-trie dependency and has a high initial memory spike during the first build.
    • StreamDictionaryFactory: Very low memory usage and no extra dependencies, but significantly slower lookups.

  10. Install Simplemma

    main

    Install the core library via pip. Note that Python 3.10 or later is required for the current version. For older Python versions, use simplemma==1.1.2 (for 3.8/3.9) or simplemma==1.0.0 (for 3.6/3.7).

    To achieve the lowest possible memory usage, install the marisa-trie extra. Alternatively, you can use the low_memory=True flag in the API to use a dependency-free low-memory mode.

    # Standard installation
pip install simplemma

# Update simplemma
pip install -U simplemma

# Install cutting-edge version from GitHub
pip install git+https://github.com/adbar/simplemma

# Install with marisa-trie for lowest memory usage
pip install simplemma[marisa-trie]

  11. Reduce RAM usage by configuring dictionary cache size

    main

    To conserve memory, you can limit the number of language dictionaries held in the cache. This is achieved by passing a cache_max_size to the DefaultDictionaryFactory, wrapping it in a DefaultStrategy, and passing that strategy to your Lemmatizer or LanguageDetector instances.

    By default, the cache size is 8. Setting a lower value like 5 will reduce the memory footprint at the cost of potentially more frequent dictionary reloads.

    from simplemma import LanguageDetector, Lemmatizer
from simplemma.strategies import DefaultStrategy
from simplemma.strategies.dictionaries import DefaultDictionaryFactory

# Configure a factory with a limited cache size to save RAM
strategy = DefaultStrategy(dictionary_factory=DefaultDictionaryFactory(cache_max_size=5))

# Use the custom strategy in a Lemmatizer
lemmatizer = Lemmatizer(lemmatization_strategy=strategy)
print(lemmatizer.lemmatize('doughnuts', lang='en'))

# Use the custom strategy in a LanguageDetector
detector = LanguageDetector('la', lemmatization_strategy=strategy)
print(detector.proportion_in_target_languages("opera post physica posita (\u03c4\u own\u03b1 \u03bc\u03b5\u03c4\u03b1 \u03c4\u03b1 \u03c6\u03c5\u03c3\u03b9\u03ba\u03ac)"))

  12. Use Wikidata as a primary source for languages

    main

    For languages where Wiktionary coverage is poor but Wikidata is strong (e.g., Malayalam ml), use Wikidata as the base word list instead of an OOV fill.

    1. Add the language's Wikidata QID to LANGUAGE_QIDS in wikidata_lexemes.py.
    2. Extract data directly into training/lists/<code>.txt (not training/fill/) using wikidata_lexemes.py.
    3. Build the dictionary using _build_dictionary with listpath="lists".

    Example extraction:

    python3 -m training.wikidata_lexemes ml training/data/wikidata/latest-lexemes.json.gz training/lists/ml.txt