spacyfishing

You can install spacyfishing via pip for standard usage, or clone the repository for development purposes.

# Normal installation
pip install spacyfishing

# Development installation
git clone https://github.com/Lucaterre/spacyfishing.git
virtualenv --python=/usr/bin/python3.8 venv
source venv/bin/activate
pip install -r requirements_dev.txt

Before using spacyfishing, you must install a pre-trained spaCy language model for the Named Entity Recognition (NER) task. You can use standard models like en_core_web_sm or your own custom NER models.

python -m spacy download en_core_web_sm

By default, spacyfishing uses the Science-Miner demo server. To avoid query limitations or to use a local/specific instance of the entity-fishing API (e.g., one running a more recent Wikidata dump), specify the api_ef_base parameter in the pipeline configuration.

To install your own instance of the underlying entity-fishing service, refer to the entity-fishing build documentation.

nlp.add_pipe("entityfishing", config={"api_ef_base": "<api-endpoint>"})

By default, the disambiguation model uses English. To process text in another language, specify the appropriate language code using the language parameter in the entityfishing component configuration.

Supported languages are documented in the NERD documentation.

import spacy

text_fr = "La bataille d'El-Alamein en Égypte oppose la 8e armée britannique..."
nlp_model_fr = spacy.load("fr_core_news_sm")

# Specify the language code in the config
nlp_model_fr.add_pipe("entityfishing", config={"language": "fr"})

doc_fr = nlp_model_fr(text_fr)

for ent in doc_fr.ents:
    # Access Wikidata URL via ent._.url_wikidata
    print((ent.text, ent.label_, ent._.kb_qid, ent._.url_wikidata))

When using spacyfishing, you can access enriched metadata for entities using custom spaCy extensions on the Span object (the ent in doc.ents).

The following attributes are available on an entity via the ._ underscore accessor:

• ent._.kb_qid: The Wikidata QID.
• ent._.normal_term: The normalized term.
• ent._.description: The description retrieved from Wikidata.
• ent._.src_description: The source description.
• ent._.other_ids: Other associated identifiers.

Example of iterating through entities to print this metadata:

for ent in doc_en.ents:
    print((ent.text, ent.label_, ent._.kb_qid, ent._.normal_term, ent._.description, ent._.src_description, ent._.other_ids))

To use spacyfishing, load a spaCy model and add the entityfishing component to the pipeline using nlp.add_pipe("entityfishing"). Once processed, you can access Wikidata-related metadata on each entity via custom spaCy extensions on the Span object.

import spacy

text_en = "Victor Hugo and Honoré de Balzac are French writers who lived in Paris."

nlp_model_en = spacy.load("en_core_web_sm")

nlp_model_en.add_pipe("entityfishing")

doc_en = nlp_model_en(text_en)

for ent in doc_en.ents:
        print((ent.text, ent.label_, ent._.kb_qid, ent._.url_wikidata, ent._.nerd_score))

For better performance when processing multiple texts, use the nlp.pipe method. You can control the number of documents processed at once by setting the batch_size parameter.

import spacy

texts_en = [
  "Victor Hugo and Honoré de Balzac are French writers who lived in Paris.",
  "Momofuko Ando is Taiwanese Japanese Business Magnate that invented instant ramen."
]

nlp_model_en = spacy.load("en_core_web_sm")

nlp_model_en.add_pipe("entityfishing")

# set number of documents to be processed at once via batch_size
docs_en = nlp_model_en.pipe(texts_en, batch_size=128)

for doc_en in docs_en:
    for ent in doc_en.ents:
        print((ent.text, ent.label_, ent._.kb_qid, ent._.url_wikidata, ent._.nerd_score))

You can check the status of the entity-fishing API requests (e.g., status codes and success flags for both the disambiguation text and terms services) using the ._.metadata attribute on the spaCy Doc object.

# Access metadata regarding the API service status
metadata = doc._.metadata

The full, raw JSON response from the entity-fishing API (including disambiguation text and terms services) is available on the spaCy Doc object through the ._.annotations attribute.

# Access the raw API response dictionary
annotations = doc._.annotations