clean-text

If you encounter authentication errors when running poetry publish, you must configure your PyPI API token. You can do this via the Poetry configuration command or by setting an environment variable.

# Option 1: Using poetry config
poetry config pypi-token.pypi <your-token>

# Option 2: Using environment variable
export POETRY_PYPI_TOKEN_PYPI=pypi-XXXXXXXXXXXX

Install the base package via pip. Note that the package name is clean-text, not cleantext.

To include the GPL-licensed unidecode package (which provides superior transliteration to ASCII), use the [gpl] extra. If unidecode is not installed, the library will fallback to Python's unicodedata.normalize.

# Install with unidecode (GPL)
pip install clean-text[gpl]

# Install without unidecode
pip install clean-text

# Install with scikit-learn compatibility
pip install clean-text[sklearn]

Use the exceptions argument to protect specific text patterns from being modified. Each entry in the exceptions list must be a regex pattern string. Matches are preserved verbatim (they are not lowered, not transliterated, and remain exactly as they appeared in the input).

from cleantext import clean

# Preserve a literal compound word while removing other punctuation
clean("drive-thru and text---cleaning", no_punct=True, exceptions=["drive-thru"])
# => 'drive-thru and textcleaning'

# Preserve all hyphenated compound words using a regex
clean("drive-thru and pick-up", no_punct=True, exceptions=[r"\w+-\w+"])
# => 'drive-thru and pick-up'

# Multiple exception patterns
clean("drive-thru costs $5", no_punct=True, no_currency_symbols=True,
      exceptions=[r"\w+-\w+", r"\$\d+"])
# => 'drive-thru costs $5'

If you need to re-tag a version (for example, if you forgot to include a fix), you must delete both the local and remote tags before re-tagging.

# Delete local and remote tag
git tag -d vX.Y.Z
git push origin :refs/tags/vX.Y.Z

# Re-tag after fixing
git tag vX.Y.Z
git push origin --tags

To evaluate the performance of the clean_texts() function, you can run the included benchmark script. This script compares sequential processing against parallel processing using different n_jobs configurations.

python benchmarks/bench_clean_texts.py

For integration into machine learning pipelines, use the CleanTransformer class. It provides a scikit-learn compatible API and accepts the same parameters as the clean() function.

from cleantext.sklearn import CleanTransformer

cleaner = CleanTransformer(no_punct=False, lower=False)
cleaner.transform(['Happily clean your text!', 'Another Input'])

To process a list of strings, use clean_texts(). You can enable parallel processing using the n_jobs parameter to leverage Python's multiprocessing.

n_jobs semantics:

• 1 or None: Sequential processing (default, zero overhead).
• -1: Use all available CPU cores.
• -2: Use all cores except one.
• Any positive integer: Use exactly that many workers.
• 0: Raises ValueError.

from cleantext import clean_texts

# Sequential (default)
clean_texts(["text one", "text two", "text three"])

# Use all available CPU cores
clean_texts(["text one", "text two", "text three"], n_jobs=-1)

# Use a specific number of workers
clean_texts(["text one", "text two", "text three"], n_jobs=4)

# Pass all clean() keyword arguments
clean_texts(texts, n_jobs=-1, no_urls=True, lang="de", lower=False)