DOCSEARCHbyAlgolia
Submit Docs

torchtext Documentation

repository·main·Indexed 25 days ago

https://github.com/pytorch/text

A PyTorch library for processing text data, providing datasets, pre-trained models, vocabularies, and text transformation tools for NLP tasks. It includes modules for data loading via TorchData DataPipes, experimental text transforms (normalization, tokenization, and mapping), and metrics like bleu_score. The library also provides a C++ component, libtorchtext, and supports exporting tokenizers like GPT2BPETokenizer via TorchScript for high-performance C++ applications.

Tokens
7.7K
Snippets
17
Records
78
Agent score
86%

What's inside torchtext

  1. Overview of torchtext modules

    main

    The torchtext library is organized into several functional modules:

    • torchtext.datasets: Raw text iterators for common NLP datasets.
    • torchtext.data: Basic NLP building blocks.
    • torchtext.transforms: Basic text-processing transformations.
    • torchtext.models: Pre-trained models.
    • torchtext.vocab: Vocab and Vectors related classes and factory functions.

  2. Use experimental text transforms in torchtext

    main

    The torchtext.experimental.transforms module provides a suite of experimental tools for text preprocessing, including normalization, tokenization, and vocabulary mapping.

    Key components include:

    • Normalization: BasicEnglishNormalize for standardizing text.
    • Tokenization: RegexTokenizer for regex-based splitting, and SentencePieceTokenizer for SentencePiece-based tokenization.
    • Sequential Processing: TextSequentialTransforms to chain multiple transformations together.
    • Mapping: VocabTransform and VectorTransform for converting text to numerical representations.

  4. Important notice regarding TorchText development status

    main

    Warning: Development has stopped

    TorchText development is stopped. The 0.18 release (April 2024) is the last stable release of the library. Users should plan accordingly for future maintenance and compatibility.

  6. Use torchtext.datasets for text data loading

    main

    The torchtext.datasets module provides access to various datasets as datapipes from the torchdata project. You can import specific datasets and iterate over them to retrieve labels and text lines.

    Note that these datasets are built on torchdata datapipes, which are currently in Beta. When using them with a DataLoader, follow these best practices:

    • Shuffling: Do not call dp.shuffle(). Instead, pass shuffle=True to the DataLoader (e.g., DataLoader(dp, shuffle=True)).
    • Multi-processing: When using num_workers > 0, use worker_init_fn from torch.utils.data.backward_compatibility to prevent data duplication across workers.
    • Batch Consistency: Use drop_last=True in your DataLoader to ensure consistent batch sizes, which helps prevent issues with batch-norm and small end-of-epoch batches.
    from torchtext.datasets import IMDB

train_iter = IMDB(split='train')

def tokenize(label, line):
    return line.split()

tokens = []
for label, line in train_iter:
    tokens += tokenize(label, line)

  7. Implement a new functional dataset API

    main

    Torchtext datasets use a functional API based on TorchData DataPipes. To implement a new dataset:

    1. Create a new file in the datasets directory.
    2. Define a function where the first argument is root (the cache directory).
    3. If the dataset has splits (e.g., train, test), include a split keyword argument.
    4. Apply the following decorators to your function:
      • @_create_dataset_directory(dataset_name=...): Creates the appropriate directory in root for caching.
      • @_wrap_split_argument((...)): Allows users to pass the split argument as either a str or a tuple of strings.
    5. Register the new dataset by adding it to the datasets/__init__.py file to make it importable.

  9. Install TorchArrow with torch integration

    main

    To use natively integrated text operators like bpe_tokenize (for tokenization) and lookup_indices (for vocabulary look-up) within a TorchArrow DataFrame, you must install TorchArrow from source with the USE_TORCH=1 flag enabled. By default, TorchArrow does not depend on the torch library.

    USE_TORCH=1 python setup.py install

  10. Build Libtorchtext and example applications

    main

    The example applications in this directory require libtorch and libtorchtext. libtorch is included with a working PyTorch installation. libtorchtext is the C++ component library of torchtext (excluding Python components) and is built alongside the applications during the build process.

    To build libtorchtext and the examples, use the provided build.sh script.

    chmod +x build.sh # give script execute permission
./build.sh

  11. Dataset implementation workflow with DataPipes

    main

    When building datasets, use TorchData DataPipes to compose a pipeline. A typical workflow follows these stages:

    1. Download from source: Use HTTPReader or GDriveReader to fetch data from hosts like Google Drive or AWS S3.
    2. Caching: Use OnDiskCacheHolder or EndofDiskCacheHolder to ensure data is cached on disk and not re-downloaded every epoch. It is recommended to include hash checking for data integrity.
    3. Unarchiving: Use archive DataPipes to decompress files (zip, tar, etc.). Cache the decompressed files to avoid repeated extraction.
    4. Reading files: Use text file reading utilities (for CSV, JSON, etc.) stacked on IO Stream pipes like FileOpener.
    5. Data organization: Return samples as tuples (e.g., (label, text) for classification or (source, target) for translation).
    6. Shuffling and Sharding: Ensure the implementation supports data shuffling and sharding across ranks for distributed training.

  12. Configure DataLoader for torchtext datapipes

    main

    To ensure correct behavior when using torchtext datasets with a PyTorch DataLoader, use the following configuration patterns:

    For multi-processing (num_workers > 0): Use worker_init_fn to ensure data is not duplicated across workers.

    For consistent batch sizes: Use drop_last=True to avoid very small batches at the end of an epoch.

    Recommended DataLoader setup:

    from torch.utils.data.backward_compatibility import worker_init_fn
# Assuming dp is your torchtext datapipe
DataLoader(dp, num_workers=4, worker_init_fn=worker_init_fn, drop_last=True)