TextDescriptives

TextDescriptives is a Python package built on top of spaCy designed to extract a wide variety of document-level linguistic metrics. It provides modular spaCy pipeline components that can be added to existing workflows to calculate:

• Descriptive Statistics: Token counts (total, unique), character counts, and statistical measures (mean, median, std dev) for token length, sentence length, and syllables per token.
• Readability: Standard indices including Gunning-Fog, SMOG, Flesch reading ease, Flesch-Kincaid grade, Automated Readability Index, Coleman-Liau, Lix, and Rix.
• Dependency Distance: Mean and standard deviation of the distance between a word and its head, and proportions of adjacent dependency relations.
• POS Proportions: Proportions of all part-of-speech tags in the document.
• Coherence: First- and second-order coherence based on word embedding similarity between sentences.
• Information Theory: Shannon entropy and perplexity.
• Quality: Metrics for filtering low-quality text, including stop word counts, symbol-to-word ratios, ellipsis/bullet point proportions, and repetitious text detection (duplicate lines, paragraphs, or n-grams).

The information_theory component calculates Shannon entropy, perplexity, and length-normalized perplexity. These metrics are used to describe text complexity: higher entropy indicates more complex text, while perplexity measures how well a model predicts the sample.

Metrics available via spaCy extensions:

• {doc/span}._.entropy: Shannon entropy using token.prob as the probability of each token.
• {doc/span}._.perplexity: Perplexity defined as $e^{-H(X)}$.
• {doc/span}._.per_word_perplexity: Perplexity divided by the number of words (length-normalized).

Important Requirement: This component requires a lexeme prop table from spaCy. If the table is not available for your chosen language, a warning will be raised and values will be set to np.nan.

import spacy
from textdescriptives as td

# Load a spaCy model and add the information_theory pipe
nlp = spacy.load("en_core_web_lg")
nlp.add_pipe("textdescriptives/information_theory")

doc = nlp("This is a simple text")

# Access individual metrics
perplexity = doc._.perplexity
entropy = doc._.entropy

# Extract all information theory metrics into a pandas DataFrame
df = td.extract_df(doc)

You can customize the quality requirements by passing a QualityThresholds object to the component. Thresholds are typically defined as tuples representing a range (min, max). Use None for no bound.

Common threshold configuration keys include:

• n_stop_words: (min, max)
• alpha_ratio: (min, max)
• mean_word_length: (min, max)
• doc_length: (min, max)
• symbol_to_word_ratio: A dictionary mapping symbols to ranges, e.g., {"#": (None, 0.1)}
• proportion_ellipsis: (min, max)
• proportion_bullet_points: (min, max)
• contains: A dictionary mapping strings to a boolean (e.g., {"lorem ipsum": False} to fail if present)
• duplicate_line_chr_fraction: (min, max)
• duplicate_paragraph_chr_fraction: (min, max)
• duplicate_ngram_chr_fraction: A dictionary mapping n-gram sizes (as strings) to ranges, e.g., {"5": (None, 0.15)}
• top_ngram_chr_fraction: A dictionary mapping n-gram sizes to ranges
• oov_ratio: (min, max)

import spacy
import textdescriptives as td
from textdescriptives.components.quality import QualityThresholds

nlp = spacy.load("en_core_web_sm")

# Define custom thresholds
thresholds = QualityThresholds(
    n_stop_words=(2, None),
    alpha_ratio=(0.7, None),
    mean_word_length=(3, 10),
    doc_length=(10, 100000),
    symbol_to_word_ratio={"#": (None, 0.1)},
    proportion_ellipsis=(None, 0.3),
    proportion_bullet_points=(None, 0.8),
    contains={"lorem ipsum": False},
    duplicate_line_chr_fraction=(None, 0.2),
    duplicate_paragraph_chr_fraction=(None, 0.2),
    duplicate_ngram_chr_fraction={
        "5": (None, 0.15),
        "6": (None, 0.14),
        "7": (None, 0.13),
        "8": (None, 0.12),
        "9": (None, 0.11),
        "10": (None, 0.1),
    },
    top_ngram_chr_fraction={"2": (None, 0.2), "3": (None, 0.18), "4": (None, 0.16)},
    oov_ratio=(None, 0.2)
)

# Add pipe and apply thresholds
quality_pipe = nlp.add_pipe("textdescriptives.quality")
quality_pipe.set_quality_thresholds(thresholds)

doc = nlp("The world is changed. I feel it in the water.")
print(doc._.passed_quality_check)

To calculate readability metrics, add the textdescriptives/readability pipe to your spaCy pipeline. Once processed, the metrics are available via the ._.readability attribute on the Doc object. You can also use td.extract_df(doc) to convert the results into a pandas DataFrame.

Note on Language Support: The readability component uses the Pyphen hyphenation module. If the language of your text is not supported by Pyphen, a warning will be raised and metrics requiring hyphenation (like Gunning-Fog, SMOG, Flesch, and Flesch-Kincaid) will be set to np.nan.

import spacy
import textdescriptives as td

nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("textdescriptives/readability") 
doc = nlp("The world is changed. I feel it in the water. I feel it in the earth. I smell it in the air. Much that once was is lost, for none now live who remember it.")

# Access metrics as a dictionary
print(doc._.readability)

# Extract metrics to a pandas DataFrame
df = td.extract_df(doc)
print(df)