DOCSEARCHbyAlgolia
Submit Docs

spacy-transformers

repository·master·Indexed 23 days ago

https://github.com/explosion/spacy-transformers

A library providing spaCy components and architectures to integrate Hugging Face transformer models, such as BERT, RoBERTa, and XLNet, into spaCy pipelines. It enables the use of transformer embeddings and features for training spaCy components like ner or textcat via multi-task learning.

Tokens
649
Snippets
0
Records
4
Agent score
31%

What's inside spacy-transformers

  1. Understand the transformer component limitations

    master

    The transformer component in spacy-transformers is designed to provide transformer-based features to spaCy pipeline components (like ner or textcat) via multi-task learning.

    Key Limitation: The transformer component does not support task-specific heads (such as token or text classification) directly. It provides access to the transformer embeddings/features, but not the specific classification heads of a Hugging Face model.

    Workarounds:

    • To train spaCy components: Use a transformer model as a feature source to train components like ner or textcat using spaCy's multi-task learning capabilities.
    • To use existing Hugging Face predictions: If you need to use the actual predictions from an existing Hugging Face text or token classification model, use the spacy-huggingface-pipelines wrapper instead.

  2. Install spacy-transformers

    master

    Install the package via pip. This will automatically install dependencies including PyTorch and spaCy.

    Requirements:

    • Python 3.6+
    • PyTorch v1.5+
    • spaCy v3.0+

    Important: Install this package before you install any spaCy models.

    To install with GPU support, append your CUDA version in brackets (e.g., cuda100 for CUDA 10.0). You can find your version using nvcc --version.

  3. Repackage en_core_web_trf for spacy-transformers

    master

    The repackage_model.py script is a utility used to repackage the existing en_core_web_trf model to ensure compatibility with spacy-transformers (v1.4+) and spacy (v3.8+).

    It performs the following operations:

    1. Loads the installed en_core_web_trf model.
    2. Converts serialized transformer weights from torch pickle to the safetensors format.
    3. Updates meta.json with new versioning and compatibility requirements (e.g., setting spacy_version and adding spacy-transformers to requirements).
    4. Generates a new package structure including __init__.py, setup.cfg, pyproject.toml, and MANIFEST.in to allow the model to be installed as a standard Python package.

    After running the script, the repackaged model is placed in a dist/ directory.

  4. Build and load a repackaged spaCy model

    master

    Once the repackage_model.py script has finished running, you can build the model into a distributable wheel and load it using standard spaCy commands.

    To build a wheel: Navigate to the output directory (e.g., dist/en_core_web_hftrf-3.8.1) and use the build module:

    cd dist/en_core_web_hftrf-3.8.1 && python -m build --wheel

    To load the model: After installation, load the model using spacy.load() with the new package name:

    import spacy
nlp = spacy.load('en_core_web_hftrf')