PyATE: Python Automated Term Extraction

The pyate package is organized into several specialized submodules for term extraction and linguistic analysis. Depending on your task, you will interact with one or more of the following:

• pyate.basic: Core basic functionality.
• pyate.combo_basic: Combinatorial basic functionality.
• pyate.cvalues: Likely related to constant values or specific linguistic values used in the package.
• pyate.term_extraction: Low-level term extraction logic.
• pyate.term_extraction_pipeline: High-level pipelines for automated term extraction.
• pyate.term_extractor: The primary interface for extracting terms.
• pyate.weirdness: Tools for calculating 'weirdness' or statistical anomalies in text.

PyATE (PYthon Automated Term Extraction) is a Python library that implements various automated term extraction (ATE) algorithms. It leverages spaCy for Part-of-Speech (POS) tagging to identify potential terms.

Supported algorithms include:

• C-Value
• Basic
• Combo Basic
• Weirdness
• Term Extractor (using spaCy POS tagging)

The pyate package implements five distinct algorithms for automatic term recognition. Each algorithm has different requirements for parameters like general corpora, weights, and smoothing.

Note: The technical_corpus parameter is required for all algorithms.

Supported Algorithms

• C-Value: Uses a threshold-based approach.
• Basic: A simplified version of Combo Basic focusing on term length, frequency, and containment.
• Combo Basic: A generalization of Basic that includes a third factor for candidate terms contained within the candidate.
• Weirdness: Compares term frequency in a technical domain against a general domain.
• Term Extractor: A comprehensive extraction algorithm.

To perform basic term extraction, import one of the implemented algorithms from pyate. According to research (Astrakhantsev 2016), combo_basic is considered the most precise of the available algorithms. The functions take a string (or corpus) as input and return a Pandas Series where the index contains the extracted terms and the values represent their respective termhood scores.

To get the most relevant terms, you can sort the resulting Pandas Series in descending order.

from pyate import combo_basic

string = """Your text content here..."""

# Returns a Pandas Series of terms and their termhood scores
print(combo_basic(string).sort_values(ascending=False))

Warning: This model only works with spaCy v2.

This model is trained on the ACL dataset and can achieve high performance for term extraction. It is not included in the standard pyate installation and must be installed separately.

Installation:

pip install https://github.com/kevinlu1248/pyate/releases/download/v0.4.2/en_acl_terms_sm-2.0.4.tar.gz

Usage:

import spacy

nlp = spacy.load("en_acl_terms_sm")
doc = nlp("Hello world, I am a term extraction algorithm.")
print(doc.ents)

If you use PyATE in published research, please notify the author at kevinlu1248@gmail.com and cite the software using the following formats:

Lu, Kevin. (2021, June 28). kevinlu1248/pyate: Python Automated Term Extraction (Version v0.5.3). Zenodo. http://doi.org/10.5281/zenodo.5039289

Or using BibTeX:

@software{pyate,
	title        = {kevinlu1248/pyate: Python Automated Term Extraction},
	author       = {Lu, Kevin},
	year         = 2021,
	month        = {Jun},
	publisher    = {Zenodo},
	doi          = {10.5281/zenodo.5039289}
}

The pyate project supports a high-performance spaCy model for term extraction. This model is not included with the base pyate installation and must be installed separately via pip using the direct URL to the model package.

pip install https://github.com/kevinlu1248/pyate/releases/download/v0.4.2/en_acl_terms_sm-2.0.4.tar.gz

To integrate term extraction into an existing spaCy pipeline, use the TermExtractionPipeline with spaCy's add_pipe method. This allows you to access extracted terms via the ._.combo_basic (or other algorithm) extension on the Doc object.

import spacy
from pyate.term_extraction_pipeline import TermExtractionPipeline

nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("combo_basic")
doc = nlp(string)
print(doc._.combo_basic.sort_values(ascending=False).head(5))

You can install pyate directly from PyPI along with a specific spaCy model using pip. This command installs both the library and the en_core_web_sm-2.2.5 model required for operation.

pip install pyate https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.5/en_core_web_sm-2.2.5.tar.gz

Install PyATE using pip and download the required English spaCy model:

pip install pyate 
spacy download en_core_web_sm

Note on spaCy versions:

• PyATE supports spaCy V3.
• For spaCy V2, use pyate==0.4.3.

To build and install pyate from the source repository, clone the repository using git or the GitHub CLI (gh), then run pip install from within the cloned directory.

git clone https://github.com/kevinlu1248/pyate.git
# or alternatively, using the gh cli
# gh repo clone kevinlu1248/pyate
pip install pyate

To confirm that pyate is installed correctly and to check its version and dependencies, use the pip show command.

pip show pyate