DOCSEARCHbyAlgolia
Submit Docs

textacy Documentation

repository·main·Indexed 25 days ago

https://github.com/chartbeat-labs/textacy

A Python library for high-level NLP tasks built on top of spaCy. textacy focuses on workflow stages before and after core spaCy processing, providing tools for text preprocessing, information extraction (n-grams, entities, SVO triples, acronyms, keyterms), data augmentation, text analysis (readability and lexical diversity), and the creation of document representations such as co-occurrence networks and sparse matrices.

Tokens
18.1K
Snippets
31
Records
133
Agent score
77%

What's inside textacy

  1. Overview of textacy

    main

    textacy is a Python library designed for natural language processing (NLP) tasks that complement spaCy. While spaCy handles core NLP fundamentals like tokenization, part-of-speech tagging, and dependency parsing, textacy focuses on the workflow stages before and after these core processes.

    Key capabilities include:

    • Text Preprocessing: Cleaning, normalizing, and exploring raw text.
    • Information Extraction: Extracting n-grams, entities, acronyms, keyterms, and SVO (Subject-Verb-Object) triples.
    • Data Loading: Loading prepared datasets containing text and metadata.
    • Text Analysis: Computing readability (e.g., Flesch-Kincaid) and lexical diversity (e.g., Type-Token Ratio) statistics.
    • Similarity & Modeling: Comparing strings/sequences and performing topic modeling (tokenization, vectorization, training, and visualization).
    • spaCy Integration: Accessing and extending spaCy's functionality for single or multiple documents.

  2. Perform data augmentation with textacy.augmentation

    main

    The textacy.augmentation module provides tools for expanding datasets through various text transformation techniques. You can use the Augmenter class to orchestrate transformations, or use individual transform functions directly.

    Available transformation types include:

    • Word-level transforms: substitute_word_synonyms, insert_word_synonyms, swap_words, and delete_words.
    • Character-level transforms: substitute_chars, insert_chars, swap_chars, and delete_chars.

    These tools are useful for increasing the diversity of training data in NLP pipelines.

  3. Calculate document similarity with textacy.similarity

    main

    The textacy.similarity module provides various methods to measure the similarity between documents, tokens, or sequences. These methods are categorized into different approaches based on the underlying algorithm:

    • Edit-based similarity (textacy.similarity.edits): Measures similarity based on the number of edits required to transform one string into another. Includes hamming, levenshtein, jaro, and character_ngrams.
    • Token-based similarity (textacy.similarity.tokens): Measures similarity based on token sets or distributions. Includes jaccard, sorensen_dice, tversky, cosine, and bag.
    • Sequence-based similarity (textacy.similarity.sequences): Focuses on the order of elements, such as matching_subsequences_ratio.
    • Hybrid similarity (textacy.similarity.hybrid): Combines different approaches, such as token_sort_ratio and monge_elkan.

  5. Perform File I/O with textacy.io

    main
    The textacy.io module provides a collection of utility functions for reading and writing various data formats commonly used in NLP workflows, including plain text, JSON, CSV, sparse matrices, and spaCy Doc objects. It also includes utilities for HTTP streaming and file system operations like unzipping and downloading files.

  6. Use textacy.preprocessing for text cleaning and normalization

    main

    The textacy.preprocessing module provides a suite of tools to clean, normalize, and transform raw text before performing NLP tasks. It is organized into several functional submodules:

    • pipeline: For creating reusable preprocessing pipelines.
    • normalize: For standardizing text elements like whitespace, unicode, and punctuation.
    • remove: For stripping unwanted content like HTML tags, accents, or brackets.
    • replace: For substituting specific patterns (e.g., URLs, emails, emojis) with placeholders or other text.

    These tools are designed to work alongside spaCy to prepare text for downstream analysis.

  7. Manage collections of documents with textacy.Corpus

    main

    A textacy.Corpus is an ordered collection of spaCy Doc objects, all processed by the same language pipeline.

    Key Features:

    • Initialization: You can initialize a corpus from a stream of texts, records (text + metadata), or existing Doc objects. You must specify the language or a spaCy language object.
    • Indexing: Supports standard Python indexing (e.g., corpus[-1] or corpus[10:15]).
    • Querying: Supports boolean queries via .get(lambda doc: ...) to filter documents based on attributes like metadata.

    Warning: All data in a Corpus is stored in-memory. The maximum size is limited by your available RAM.

  8. Requirement: spaCy models for textacy

    main

    Most textacy operations require language-specific models from spaCy. You must install these models following the spaCy documentation.

    Important Change: textacy no longer performs automatic language identification to select a model. You must explicitly specify the full model name (e.g., en_core_web_sm) when applying it to text, as aliasing models (e.g., mapping en to en_core_web_sm) is no longer supported by spaCy.

  9. Handle multi-lingual text collections

    main

    A textacy.Corpus is monolingual because it uses a single spaCy language pipeline. If your dataset contains multiple languages, use one of these two strategies:

    Strategy 1: Individual Document Processing

    Iterate over texts and use textacy.make_spacy_doc(text). If the language is unspecified, textacy attempts auto-detection. If the required spaCy model is not installed, an OSError is raised. You can wrap this in a try/except block to skip unsupported languages.

    Strategy 2: Split into Monolingual Corpora

    Use textacy.identify_lang(text) to filter your data into language-specific collections, then instantiate a separate textacy.Corpus for each language using the appropriate spaCy model name.

    # Strategy 1: Skip unsupported languages
>>> for text in texts:
...     try:
...         doc = textacy.make_spacy_doc(text)
...     except OSError:
...         continue
...     # do stuff...

# Strategy 2: Create separate corpora
>>> en_corpus = textacy.Corpus(
...     "en_core_web_sm", data=(
...         text for text in texts
...         if textacy.identify_lang(text) == "en")
... )
>>> es_corpus = textacy.Corpus(
...     "es_core_news_sm", data=(
...         text for text in texts
...         if textacy.identify_lang(text) == "es")
... )

  10. Load built-in datasets with textacy.datasets

    main
    You can access built-in datasets using the textacy.datasets module. For example, the CapitolWords dataset provides a collection of speeches from the U.S. Congressional Record. Use .download() to fetch the data and .records() to iterate through the records. Each record is a Record object containing text and meta (metadata).

  11. Install textacy via pip or conda

    main

    You can install textacy using standard Python package managers.

    To install via pip:

    $ pip install textacy

    To install via conda (using the conda-forge channel):

    $ conda install -c conda-forge textacy

    If you are installing from a downloaded source tarball:

    $ cd path/to/textacy
$ pip install .

  12. Preprocess text using textacy.preprocessing

    main

    Use textacy.preprocessing to clean and normalize text. You can use individual functions like preproc.replace.numbers() or create a reusable pipeline using preproc.make_pipeline(). Common normalization tasks include unicode, quotation_marks, and whitespace normalization.

    from textacy import preprocessing as preproc

# Create a reusable pipeline
preprocessor = preproc.make_pipeline(
    preproc.normalize.unicode,
    preproc.normalize.quotation_marks,
    preproc.normalize.whitespace,
)

# Apply to text
clean_text = preprocessor(raw_text)