nlpaug

repository·master·Indexed 26 days ago

https://github.com/makcedward/nlpaug

A Python library for augmenting NLP, audio, and spectrogram data to improve machine learning model performance through synthetic data generation. It provides various Augmenter classes for character-level (e.g., KeyboardAug, OCRAug), word-level (e.g., ContextualWordEmbsAug, SynonymAug), and sentence-level (e.g., AbstSummAug) text augmentation, as well as signal augmentation for audio and spectrograms. The library includes Flow orchestration for sequential or random augmentation pipelines and supports Python 3.12+.

Tokens
10.1K
Snippets
38
Records
62
Agent score
88%

What's inside nlpaug

  1. Overview of nlpaug augmentation components

    master

    The nlpaug library provides tools for augmenting NLP data in machine learning projects. It is built around two core concepts:

    1. Augmenter: The basic element used to perform a specific type of augmentation.
    2. Flow: A pipeline used to orchestrate multiple augmenters together in a sequence.
  2. Overview of nlpaug Augmenters

    master

    The library provides various Augmenter classes categorized by target data type and granularity:

    Textual Augmentation

    • Character level: KeyboardAug (keyboard errors), OcrAug (OCR errors), RandomAug (random operations), SpellingAug (spelling mistakes), SplitAug (splitting words).
    • Word level: AntonymAug (antonyms), ContextualWordEmbsAug (BERT/RoBERTa/XLNet), RandomWordAug (random operations), SynonymAug (synonyms), TfIdfAug (TF-IDF based), WordEmbsAug (word2vec/GloVe/fasttext), BackTranslationAug (translation models), ReservedAug (reserved words).
    • Sentence level: ContextualWordEmbsForSentenceAug (XLNet/GPT2), AbstSummAug (abstractive summarization), LambadaAug (language model based).

    Signal Augmentation

    • Audio: CropAug, LoudnessAug, MaskAug, NoiseAug, PitchAug, ShiftAug, SpeedAug, VtlpAug, NormalizeAug, PolarityInverseAug.
    • Spectrogram: FrequencyMaskingAug, TimeMaskingAug, LoudnessAug.

    Pipeline Orchestration (Flow)

    • Sequential: Apply a list of augmentation functions one after another.
    • Sometimes: Apply augmentation functions randomly.
  3. Requirements for NLPaug 2.0.0

    master

    As of version 2.0.0 (June 2026), the runtime baseline for NLPaug has been upgraded. Ensure your environment meets the following requirements:

    • Python: 3.12 or higher
    • Major Dependencies:
      • transformers 5.9+
      • gensim 4.4+
      • librosa 0.11+
      • NumPy 2.x+
  4. Explore nlpaug augmentation examples

    master

    The nlpaug library provides several specialized augmentation workflows. You can find detailed implementation examples in the following Jupyter notebooks:

    • Audio augmenters: Techniques for augmenting audio data.
    • Textual augmenters: Techniques for augmenting text data.
    • Spectrogram augmenters: Techniques for augmenting spectrograms.
    • Custom augmenter: How to implement and use your own augmentation logic.
    • TF-IDF model training: Using TF-IDF for augmentation-related tasks.
    • Flow: Managing augmentation flows.
  5. Download pretrained assets for WordEmbsAug

    master

    If using WordEmbsAug with word2vec, GloVe, or fasttext, you must download the pretrained assets first using DownloadUtil.

    from nlpaug.util.file.download import DownloadUtil
    DownloadUtil.download_word2vec(dest_dir='.')
    DownloadUtil.download_glove(model_name='glove.6B', dest_dir='.')
    DownloadUtil.download_fasttext(model_name='wiki-news-300d-1M', dest_dir='.')
  6. Install nlpaug feature extras

    master

    Install optional dependencies for specific augmentation types using bracket notation:

    • Transformers: pip install "nlpaug[transformers]"
    • NLTK: pip install "nlpaug[nltk]"
    • Word Embeddings: pip install "nlpaug[word-embs]"
    • Audio: pip install "nlpaug[audio]"
    • Lambada: pip install "nlpaug[lambada]"
    pip install "nlpaug[transformers]"
    pip install "nlpaug[nltk]"
    pip install "nlpaug[word-embs]"
    pip install "nlpaug[audio]"
    pip install "nlpaug[lambada]"
  7. Use Character Augmenters

    master

    Character-level augmentation simulates errors like OCR mistakes or keyboard typos.

    • OcrAug: Simulates optical character recognition errors (e.g., substituting 'o' with '0').
    • KeyboardAug: Simulates typos based on keyboard distance.
    • RandomCharAug: Performs random character operations using the action parameter.
  8. Use Word Augmenters

    master

    Word-level augmentation substitutes or inserts words using various techniques:

    • SpellingAug: Substitutes words with spelling mistakes from a dictionary.
    • WordEmbsAug: Uses word embeddings (word2vec, glove, or fasttext) to find similar words.
    • TfIdfAug: Uses TF-IDF similarity to insert or substitute words.
    • ContextualWordEmbsAug: Uses language models (BERT, DistilBERT, RoBERTA, or XLNet) to predict target words.
    • SynonymAug: Uses WordNet or PPDB to find synonyms.
    • AntonymAug: Substitutes words with their antonyms.
    • RandomWordAug: Performs random word operations (swap, delete, or crop).
    • SplitAug: Splits a word into two tokens.
    • BackTranslationAug: Translates text to another language and back to create variations.
    • ReservedAug: Protects specific tokens from being augmented.
  9. Train a classifier for LAMBADA augmentation

    master

    Before using LAMBADA augmentation, you must train a classifier on your input data. The input data should be a CSV file with text and label columns. Use the train_cls.py script to train the model.

    python ../scripts/lambada/train_cls.py  \
        --train_data_path ../test/res/text/classification.csv \
        --val_data_path ../test/res/text/classification.csv \
        --output_dir ../model/lambada/cls \
        --device cuda \
        --num_epoch 2
  10. Load and visualize audio spectrograms

    master

    Use AudioLoader.load_mel_spectrogram to load an audio file as a Mel spectrogram and AudioVisualizer.spectrogram to display it. This is useful for preparing data for spectrogram-based augmentation.

    from nlpaug.util.audio.loader import AudioLoader
    from nlpaug.util.audio.visualizer import AudioVisualizer
    
    path = 'Yamaha-V50-Rock-Beat-120bpm.wav'
    
    data = AudioLoader.load_mel_spectrogram(path, n_mels=128)
    AudioVisualizer.spectrogram('Original', data)