spaCy NLP Library

spaCy is an industrial-strength library for advanced Natural Language Processing (NLP) in Python and Cython. It is designed for production use and supports tokenization and training for over 70 languages.

Key features include:

• Pretrained pipelines: Ready-to-use models for various NLP tasks.
• Neural network models: State-of-the-art models for tagging, parsing, named entity recognition (NER), and text classification.
• Transformer support: Multi-task learning with pretrained transformers like BERT.
• Production-ready training system: Tools for training, model packaging, deployment, and workflow management.

spaCy is a free, open-source Python library designed for advanced Natural Language Processing (NLP) in production environments. It is built to help developers build applications that process and understand large volumes of text, such as information extraction, natural language understanding, or pre-processing for deep learning.

Key distinctions:

• Not a platform/API: It is a library you use to build applications, not a SaaS or web service.
• Not a chatbot engine: It provides text processing capabilities that can power chatbots, but is not a chatbot itself.
• Not research software: Unlike NLTK or CoreNLP, spaCy is designed to be integrated and opinionated, offering specific, high-performance algorithms rather than a menu of many equivalent choices to simplify the developer experience.

The spaCy documentation provides various resources depending on your needs:

• Getting Started: Use [spaCy 101] for foundational knowledge.
• Learning & Guides: Consult [Usage Guides] for feature implementation, [Project Templates] for end-to-end workflows, or the [Online Course] for interactive learning.
• Technical Reference: Use the [API Reference] for detailed method signatures and the [Changelog] for version history.
• Advanced Features: Explore [GPU Processing] for CUDA support, [Models] for pre-trained pipelines, and [Large Language Models] for LLM integration.
• Ecosystem: Visit [Universe] for plugins and extensions, or use the [spaCy VS Code Extension] for working with configuration files.
• Community & Updates: Follow the [Blog], [Videos], or [Live Stream] for news and tutorials.

The Language class is the central object in spaCy, representing a text-processing pipeline. It is typically loaded once per process (often named nlp) and contains:

• A shared vocabulary (Vocab).
• Language data (linguistic features).
• Optional binary weights (from trained models).
• A processing pipeline consisting of components (like a tagger or parser) that are called sequentially on a Doc object.

You can also add custom components to this pipeline that take a Doc, modify it, and return it.

The EntityRuler is a pipeline component for rule-based named entity recognition (NER). It allows you to add spans to Doc.ents using token-based rules or exact phrase matches.

You can use it in two ways:

1. Combined with a statistical EntityRecognizer: To boost accuracy by combining machine learning with specific rules.
2. Standalone: To implement a purely rule-based entity recognition system.

When the EntityRuler makes predictions, they are accessible via Doc.ents. The labels are also stored in the underlying tokens in Token.ent_type and Token.ent_iob fields. Note that each token can only have one label.

import spacy

nlp = spacy.blank("en")
ruler = nlp.add_pipe("entity_ruler")
ruler.add_patterns([{"label": "ORG", "pattern": "Apple"}])

doc = nlp("Apple is a company.")
for ent in doc.ents:
    print(ent.text, ent.label_)

The Vocab object is a storage class that provides a lookup table for accessing Lexeme objects and the StringStore. It owns the underlying C-data that is shared between Doc objects in a language model.

Important Note: A Vocab instance is not static; it increases in size as new tokens are processed in texts. Some models may start with an empty vocabulary at initialization.

The SpanRuler is a pipeline component used for rule-based span and named entity recognition (NER). It allows you to add spans to Doc.spans[spans_key] (as a SpanGroup) and/or to Doc.ents using token-based rules or exact phrase matches.

When matches are assigned to Doc.ents, the annotations are stored in the Token.ent_type and Token.ent_iob fields.

The Sentencizer is a rule-based pipeline component used for sentence boundary detection. Unlike the DependencyParser, which uses a statistical model, the Sentencizer uses a simpler strategy based on punctuation. This makes it a lightweight alternative when you don't need the full power of a dependency parser but still need to segment text into sentences.

When applied to a Doc, it sets the following attributes:

• Token.is_sent_start: A boolean indicating if a token starts a new sentence.
• Doc.sents: An iterator over the detected sentences (Spans).

from spacy.lang.en import English

nlp = English()
nlp.add_pipe("sentencizer")
doc = nlp("This is a sentence. This is another sentence.")
assert len(list(doc.sents)) == 2

The AttributeRuler is a pipeline component used for rule-based token attribute assignment. It uses Matcher patterns to identify specific tokens and then sets their attributes (like POS, TAG, LEMMA, or MORPH).

Common use cases include:

• Handling exceptions: Overriding incorrect attributes for specific tokens.
• Attribute mapping: Mapping fine-grained tags to coarse-grained tags (e.g., mapping specific POS tags to a broader category).
• Morphological mapping: Mapping text and fine-grained tags to coarse-grained morphological features.

It is a non-trainable component (api_trainable: false).