DOCSEARCHbyAlgolia
Submit Docs

WikidataIntegrator

repository·main·Indexed 42 days ago

https://github.com/sulab/wikidataintegrator

A Python library for reading and writing to Wikidata and Wikibase, optimized for building bots that synchronize authoritative external resources. It features the WDItemEngine for data-driven and QID-driven workflows, SPARQL query execution, ShEx conformance testing, and support for fast-run mode to accelerate updates. The library includes tools for bot authentication via WDLogin, item merging, and specialized helpers for creating publication and database release records.

Tokens
8.6K
Snippets
37
Records
46
Agent score
53%

What's inside WikidataIntegrator

  1. Use the wdi_rdf extension for RDF output

    main
    The wdi_rdf extension replicates the RDF output of a Wikidata item, building on Wikidata/triplify-json. Unlike the default JSON output, the RDF output is richer in content, specifically providing SI unit conversions for Wikidata statements of the datatype unit. This allows for direct comparison of values across different units. It also captures merges and additional metadata.

  2. Extract subsets of RDF representations

    main
    To manage the redundancy in the Wikidata data model, the wdi_rdf extension allows you to extract subsets of the RDF representation of a Wikidata item. You can choose to extract either the full RDF graph or specific subsections, such as "truthy" statements (where context and provenance, like qualifiers and references, have been removed).

  3. How WDItemEngine works

    main

    The wdi_core.WDItemEngine is the central class for interacting with Wikidata. It provides two primary workflows:

    1. Data-driven (Preferred): You provide data (e.g., a unique central identifier), and WDItemEngine searches for, loads, or modifies an existing item or creates a new one. This method performs automatic consistency checks using SPARQL queries to prevent duplicates and ensure data integrity.
    2. QID-driven: You provide a specific Wikidata QID to modify. This should be used with extreme caution as it does not perform automatic consistency checks.

    Key Features:

    • Loads items by QID or by data-based identifiers.
    • Automatic conflict detection (e.g., multiple items with the same unique identifier).
    • Automatic verification that the loaded item matches the provided data.
    • Access to the full item as a JSON document.
    • Optimized HTTP request management for performance.
    • Support for two modes: normal (one item at a time) and fastrun (pre-loads data locally to speed up updates by up to 9x).

  4. Use Fast Run mode for efficient data operations

    main

    Fast run mode optimizes operations by filtering the Wikidata corpus based on specific property/value combinations.

    Requirements:

    • The data provided to the WDItemEngine constructor must contain at least one unique value/ID (e.g., an NCBI entrez gene ID) that exists on only one Wikidata item.
    • Set fast_run=True in the WDItemEngine constructor.
    • Provide a fast_run_base_filter dictionary. This dictionary uses properties as keys and item QIDs (or empty strings for literals) as values.

    Behavior:

    • Fast run mode checks for equality of property/value pairs, qualifiers (excluding qualifier attributes), labels, aliases, and descriptions.
    • Note: It ignores references by default. To include references, set fast_run_use_refs=True.
    from wikidataintegrator import wdi_core, wdi_login
    
login_instance = wdi_login.WDLogin(user='<bot user name>', pwd='<bot password>')
    
# Filter for human genes (P703 = Q15978631) using NCBI ID (P351)
fast_run_base_filter = {'P351': '', 'P703': 'Q15978631'}
fast_run = True
    
raw_data = {
    '50943': 'ENST00000376197',
    '1029': 'ENST00000498124'
}
    
for entrez_id, ensembl in raw_data.items():
    entrez_gene_id = wdi_core.WDString(value=entrez_id, prop_nr='P351')
    ensembl_transcript_id = wdi_core.WDString(value=ensembl, prop_nr='P704')
    
    data = [entrez_gene_id, ensembl_transcript_id]
    
    wd_item = wdi_core.WDItemEngine(data=data, fast_run=fast_run, fast_run_base_filter=fast_run_base_filter)
    wd_item.write(login_instance)

  5. Check conformance to EntitySchemas using Shape Expressions (ShEx)

    main

    Wikidata uses Shape Expressions (ShEx) via the EntitySchema extension to describe its underlying schema. You can leverage RDF entry points (either directly from Wikidata or via the Query Service) to check Wikidata items for consistency against these schemas.

    To perform these checks, you can use:

    1. The EntitySchema interface of Wikidata.
    2. Third-party tools like wikishape.
    3. The check_shex_conformance function available in wdi_core.

  6. Build a minimal Wikidata bot

    main

    A minimal bot requires three components:

    1. A login object (wdi_login.WDLogin).
    2. A data type object (e.g., wdi_core.WDString) containing the value and property number (prop_nr).
    3. A WDItemEngine object which processes the data and performs the write operation.

    Data objects should be passed to WDItemEngine as a list.

    from wikidataintegrator import wdi_core, wdi_login
        
# login object
login_instance = wdi_login.WDLogin(user='<bot user name>', pwd='<bot password>')
         
# data type object, e.g. for a NCBI gene entrez ID
entrez_gene_id = wdi_core.WDString(value='<some_entrez_id>', prop_nr='P351')
    
# data goes into a list
data = [entrez_gene_id]
    
# Search for and then edit/create new item
wd_item = wdi_core.WDItemEngine(data=data)
wd_item.write(login_instance)

  7. Perform pre-ingestion conformance checks

    main
    While EntitySchema conformance checks are typically done post-data ingestion (after data is converted to RDF by Wikibase internal scripts), the wdi_rdf extension is designed to support pre-ingestion checks. This allows bot operations to verify that data conforms to the required schema before it is added to Wikidata.

  8. Install WikidataIntegrator from source

    main

    You can also install the library by cloning the repository and running the setup script with administrator rights or within a virtual environment.

    git clone https://github.com/SuLab/WikidataIntegrator.git
cd WikidataIntegrator
python3 setup.py install

  9. Logging and error handling with try_write()

    main

    The wdi_core.WDItemEngine.log() method provides access to Python's built-in logging.

    For a more robust pattern, use wdi_helpers.try_write(). This helper performs both the write() operation and logging simultaneously. It captures exceptions and logs the action in a standard format, which is compatible with the Bot Log Parser for generating HTML reports.

  10. Access Wikidata RDF via URL

    main

    You can obtain the RDF of a Wikidata item directly through a web browser by modifying the concept URI. To do this, change wiki to entity in the URL and append .ttl to the end.

    Example: To get the RDF for item Q17853226:

    • Web address: https://www.wikidata.org/wiki/Q17853226
    • RDF address: https://www.wikidata.org/entity/Q17853226.ttl
    https://www.wikidata.org/entity/Q17853226.ttl

  12. Login to Wikidata using WDLogin

    main

    To perform write operations on Wikidata (such as updating descriptions), you must authenticate using a bot account. You can use wdi_login.WDLogin by providing a username and password. It is recommended to retrieve these credentials from environment variables (WDUSER and WDPASS) for security.

    import os
from wikidataintegrator import wdi_login

if "WDUSER" in os.environ and "WDPASS" in os.environ:
    WDUSER = os.environ['WDUSER']
    WDPASS = os.environ['WDPASS']
else:
    raise ValueError("WDUSER and WDPASS must be specified as environment variables")

login = wdi_login.WDLogin(WDUSER, WDPASS)