DOCSEARCHbyAlgolia
Submit Docs

LemmInflect: English Lemmatization and Inflection

repository·master·Indexed 42 days ago

https://github.com/bjascob/lemminflect

A Python module for English lemmatization and inflection using a hybrid approach of dictionaries and neural networks to handle out-of-vocabulary words. It can be used as a standalone library via functions like getLemma() and getInflection(), or as an extension for spaCy (version 2.0+) to add ._.lemma() and ._.inflect() methods to Token objects.

Tokens
5K
Snippets
20
Records
27
Agent score
52%

What's inside LemmInflect

  1. Understanding AUX and MODAL verb types

    master

    LemmInflect handles the distinction between auxiliary verbs and modal verbs, which can vary between tag sets (Penn Treebank vs. UPOS).

    • AUX verbs: These include common auxiliaries like have (has, had), be (is, was, been), and do (does, did).
    • MODAL verbs: These include modal auxiliaries such as can, could, may, might, will, would, shall, should, must, ought, and dare.

    Depending on your tagger, these may be labeled as verbs, but LemmInflect recognizes these specific categories to provide accurate inflections.

  2. Obtain source data for CreateEnglishWordSet.py

    master

    The CreateEnglishWordSet.py script requires English word lists. These are standard in Ubuntu 18.04 via the wamerican and wbritish packages (and larger variants like wamerican-large, -huge, or -insane).

    If you are not on a Debian-based distribution, you can download and extract the raw text files from the Debian package repository: https://packages.debian.org/stable/text/wamerican

  3. Use LemmInflect as a library

    master

    You can use lemminflect by importing specific functions to convert inflected words to their base forms. The Lemmatizer class is implemented as a singleton that is instantiated automatically upon the first call to any lemminflect method.

    Commonly used functions include:

    • getLemma(word, upos, lemmatize_oov=True): Returns a single lemma based on a Universal Dependencies (UPOS) tag.
    • getAllLemmas(word, upos=None): Returns a dictionary of all possible lemmas, keyed by their UPOS tags.
    • getAllLemmasOOV(word, upos): Uses the rules-based system (instead of dictionary lookup) to find lemmas for Out-Of-Vocabulary words.
    • isTagBaseForm(tag): Checks if a Penn Treebank tag is already in its lemma form. It is best practice to call this before getLemma to avoid errors when lemmatizing a word that is already a lemma.
    from lemminflect import getLemma, getAllLemmas, getAllLemmasOOV, isTagBaseForm

# Get a single lemma for a specific UPOS
getLemma('watches', upos='VERB')  # ('watch',)

# Get all possible lemmas across different UPOS tags
getAllLemmas('watches')  # {'NOUN': ('watch',), 'VERB': ('watch',)}

# Get lemmas for Out-Of-Vocabulary words using rules
getAllLemmasOOV('xatches', 'NOUN')  # {'NOUN': ('xatch',)}

# Check if a Penn Tag is already a base form
isTagBaseForm('JJ')  # True

  5. Install LemmInflect via pip

    master

    Install LemmInflect using pip3. The only external requirement is numpy for the neural network matrix math. The library is compatible with Python 3 and works on Linux, Windows, and macOS.

    pip3 install lemminflect

  6. Obtain source data for BWCorpToUnigrams.py

    master

    The BWCorpToUnigrams.py script uses the Billion Word Corpus.

    1. Download the data: Use wget to download the archive:
    wget http://statmt.org/wmt11/training-monolingual.tgz
    1. Extract English portions: The archive is ~9.9GB and contains multiple languages. To extract only the English shuffled news files (which total ~14GB), use:
    tar --extract -v --file training-monolingual.tgz --wildcards training-monolingual/news.20??.en.shuffled
    1. Configure the path: You must modify the lemminflect.config.bwcorp_dir setting to point to the corpus location.
    wget http://statmt.org/wmt11/training-monolingual.tgz
tar --extract -v --file training-monolingual.tgz --wildcards training-monolingual/news.20??.en.shuffled

  8. Use LemmInflect as a spaCy extension

    master

    LemmInflect automatically sets up an extension for spaCy when it is imported. This allows you to access lemmas directly from spaCy Token objects using the ._.lemma() property.

    When using the extension, if the word's Penn tag indicates it is already in lemma form, the original word is returned directly.

    import lemminflect
import spacy

nlp = spacy.load('en_core_web_sm')
doc = nlp('I am testing this example.')

# Access the lemma via the spaCy extension
print(doc[2]._.lemma())  # 'test'

  9. Setup prior to running lexicon build scripts

    master

    Before running the scripts in scripts/01_BuildLexicon/, you must prepare the environment to handle temporary and project-related data files.

    1. Create the data directory: The scripts use a directory named data_repo to store temporary files. This directory is defined in lemminflect/config.py and is located one level up from the config.py file. You must create this directory manually before running any setup scripts.
    2. Use a local library instance: These scripts are designed to run against a local download of the lemminflect library rather than an installed package. They use sys.path.insert(0, '../..') to force the use of the local source. If you are using a different directory structure, you may need to modify the paths in lemminflect/config.py so the scripts can correctly locate the data files.

  10. Use the LemmInflect Inflection API

    master

    The Inflection class (accessed via lemminflect functions) converts words from their base form (lemma) to a specified inflection type. It uses a combination of dictionary lookups and rule-based inflections (including neural-network models).

    Important: The library methods expect the input word to be a lemma. If your word is not in its base form, you must first call the lemmatizer to get the base form. When using the spaCy extension, lemmatization is handled automatically.

    from lemminflect import getInflection, getAllInflections, getAllInflectionsOOV

# Get a specific inflection using a PennTreebank tag
getInflection('watch', tag='VBD')
# Output: ('watched',)

# Get all possible inflections for a lemma
getAllInflections('watch')
# Output: {'NN': ('watch',), 'NNS': ('watches', 'watch'), ...}

# Get all inflections limited to a specific Universal Dependencies POS tag
getAllInflections('watch', upos='VERB')

# Get inflections for Out-Of-Vocabulary (OOV) words using the rules system
getAllInflectionsOOV('xxwatch', upos='NOUN')

  11. Download and extract raw BWCorpus data

    master

    To obtain the raw, unprocessed text data from the BWCorpus, you can download the training monolingual dataset and extract specific shuffled news files using wget and tar.

    wget http://statmt.org/wmt11/training-monolingual.tgz
tar --extract -v --file training-monolingual.tgz --wildcards training-monolingual/news.20??.en.shuffled

  12. Build resources and models for development

    master

    The scripts directory contains the tools required to build the resources that drive the runtime system. These scripts must be run in a specific order, indicated by the numbering of the directories and the numerical prefixes of the scripts themselves.

    Prerequisites

    To run these build scripts, you must have the following libraries installed:

    • nltk
    • keras
    • A Keras backend (e.g., tensorflow)

    Setup Steps

    1. Prepare Data Directory: Create a data_repo directory or a link to one in the main project directory.
    2. Download Corpora: Download the necessary corpora required for building resources (refer to the 01_BuildLexicon/README.txt for specific details).
    3. Configure Paths: Review and update lemmatizer/config.py. This file defines the locations where data is read and written. You will likely need to update the source paths for the Gutenberg and Billion Word Corpora to match your local environment.