pytextrank

PyTextRank is a Python implementation of the TextRank algorithm designed as a spaCy pipeline extension. It uses graph-based natural language processing (textgraph algorithms) to perform tasks such as:

• Phrase Extraction: Identifying the top-ranked phrases within a text document.
• Extractive Summarization: Providing low-cost summaries of text documents.
• Concept Inference: Helping transform unstructured text into more structured representations.

It implements several textgraph algorithms, including TextRank, PositionRank, Biased TextRank, and TopicRank. It leverages spaCy 3.x for preprocessing (like noun chunking and NER) and uses lemmatization instead of stemming.

PyTextRank provides an implementation of extractive summarization.

While more advanced (and often higher-cost) methods like abstractive summarization (which uses deep learning and knowledge graphs) exist, PyTextRank offers a lower-cost alternative that may better suit specific engineering and policy trade-offs in various use cases.

The quality of phrase ranking in PyTextRank can be improved by enriching the lemma graph before the ranking stage.

Common methods for enrichment include:

• Coreference Resolution: Linking different mentions of the same entity.
• Semantic Relations: Using knowledge graphs or thesauri to infer links between words that are not explicitly linked in the text (e.g., using hyponymy/hypernymy).

Examples of external resources that can be used for this purpose include WordNet and DBpedia.

The TopicRank algorithm identifies key topics in a document through a multi-stage process. It is deployed as a spaCy pipeline component.

Algorithm Steps:

1. Preprocessing: Sentence segmentation, word tokenization, and POS tagging.
2. Candidate extraction: Extracts noun chunks (sequences of nouns and adjectives).
3. Candidate clustering: Uses Hierarchical Agglomerative Clustering (with average linking) based on lemma overlap (similarity threshold > 25%).
4. Candidate ranking: Applies TextRank on a complete graph where topics are nodes, with edge weights favoring topics appearing closer together.
5. Candidate selection: Selects the first occurring keyphrase from each topic to represent it.

Note: TopicRank is not instantiated directly; you must use its factory class.

PyTextRank relates to the field of summarization, which can be categorized into two main approaches:

• extractive summarization: Summarizing source text by identifying a subset of the most important sentences as excerpts and generating them verbatim.
• abstractive summarization: Generating a short, concise summary that captures salient ideas, potentially using new phrases and sentences that do not appear in the original source text.

In the context of the TextRank algorithm used by PyTextRank, two key concepts are used:

• phrase extraction: The process of selecting representative phrases from a document as its characteristic entities (distinct from simple keyword analysis).
• lemma graph: A graph data structure used to represent links among phrases extracted from a source text during the operation of the TextRank algorithm.

PyTextRank integrates directly into the spaCy pipeline. After adding the textrank component to your nlp object, you can process text and access ranked phrases via the doc._.phrases attribute. Each phrase object provides access to its text, rank, frequency count, and associated noun chunks.

import spacy
import pytextrank

text = "Compatibility of systems of linear constraints over the set of natural numbers..."

# Load a spaCy model
nlp = spacy.load("en_core_web_sm")

# Add PyTextRank to the spaCy pipeline
nlp.add_pipe("textrank")

# Process the text
doc = nlp(text)

# Examine the top-ranked phrases
for phrase in doc._.phrases:
    print(phrase.text)
    print(phrase.rank, phrase.count)
    print(phrase.chunks)

If you are contributing to the pytextrank library, you can set up the build environment by installing the development requirements and configuring pre-commit hooks.

1. Install development dependencies: python3 -m pip install -r requirements-dev.txt

2. Install pre-commit hooks: pre-commit install git config --local core.hooksPath .git/hooks/