DOCSEARCHbyAlgolia
Submit Docs

ChemDataExtractor v2

repository·master·Indexed 41 days ago

https://github.com/cambridgemolecularengineering/chemdataextractor2

A Python-based toolkit for automating the extraction of chemical information, properties, and spectra from scientific literature in PDF, HTML, and XML formats. It features a chemistry-aware NLP pipeline, Named Entity Recognition (NER), table parsing, and a command-line interface (cde) for data extraction, model training, and configuration management.

Tokens
40.3K
Snippets
136
Records
184
Agent score
50%

What's inside ChemDataExtractor

  1. Overview of ChemDataExtractor features

    master

    ChemDataExtractor is a toolkit designed for extracting chemical information from scientific literature. Its core capabilities include:

    • Document Reading: Support for HTML, XML, and PDF formats.
    • NLP Pipeline: A chemistry-aware natural language processing pipeline.
    • Named Entity Recognition (NER): Identification of chemical entities within text.
    • Rule-based Parsing: Grammars for extracting specific properties and spectra.
    • Table Parsing: Extraction of data from tabulated formats.
    • Data Resolution: Document processing to resolve interdependencies between extracted data points.

  2. Overview of ChemDataExtractor API modules

    master

    ChemDataExtractor provides a wide range of modules for chemical data extraction, natural language processing, and document parsing. The API is organized into several functional areas:

    • Core: config, data, errors, and utils.
    • Bibliographic Extraction: biblio (including bibtex, person, and xmp support).
    • CLI Tools: A suite of command-line interfaces including cem, chemdner, cluster, config, data, dict, evaluate, pos, and tokenize.
    • Document Representation: doc module for handling document, element, figure, meta, table, and text objects.
    • Evaluation: eval module for performance assessment.
    • Modeling: model module for handling units, dimensions, quantities, and base models.
    • NLP: nlp module for abbrev, allennlpwrapper, cem, corpus, lexicon, pos, tag, and tokenize.
    • Parsing: parse module for various formats including mp, nmr, tg, and uvvis.
    • Readers: reader module for various source formats like acs, cssp, markup, nlm, pdf, plaintext, rsc, uspto, elsevier, and springer.
    • Relation Extraction: relex module for cluster, entity, pattern, phrase, relationship, and snowball.
    • Scraping: scrape module for base, clean, csstranslator, entity, fields, scraper, selector, and publisher-specific scrapers (nlm, rsc, springer, elsevier).
    • Text Processing: text module for chem, latex, normalize, and processors.

  3. Overview of ChemDataExtractor v2 features

    master

    ChemDataExtractor v2 is a toolkit designed for extracting chemical information from scientific literature. Its core capabilities include:

    • Document Reading: Support for HTML, XML, and PDF formats.
    • NLP Pipeline: A chemistry-aware natural language processing pipeline.
    • Named Entity Recognition (NER): Identification of chemical entities.
    • Property & Spectra Extraction: Rule-based parsing grammars for extracting specific chemical properties and spectra.
    • Table Parsing: Extraction of data from tabulated formats.
    • Data Resolution: Document processing to resolve interdependencies between extracted data points.

  7. Overview of ChemDataExtractor2 package roles

    master

    ChemDataExtractor2 is organized into several packages, each serving a specific role in the NLP pipeline:

    • biblio: Tools for parsing bibliographic information (e.g., BibTeX, author names).
    • cli: Command line interface tools.
    • doc: Logic for reading and splitting documents into constituent elements (Title, Paragraphs, etc.).
    • model: Tools for model representation.
    • nlp: NLP stages including POS tagging, Word clustering, CNER (Chemical Named Entity Recognition), and Abbreviation detection.
    • parse: Chemical property parsers.
    • reader: Document readers.
    • relex: Components related to the Snowball algorithm (relationship extraction).
    • scrape: Scrapers for various data sources.
    • text: General text processing utilities.

  8. Overview of the ChemDataExtractor pipeline

    master

    ChemDataExtractor is a natural language processing (NLP) pipeline designed to extract chemical property information from scientific literature. The extraction process follows a structured pipeline:

    1. Document Decomposition: Breaking a document into constituent elements (Title, Paragraphs, Sentences, Tables, Figures, etc.).
    2. Tokenization: Isolating individual tokens from the text.
    3. Part-of-speech (POS) Tagging: Identifying the semantic role of each token.
    4. Chemical Named Entity Detection: Using machine learning to identify chemical entities.
    5. Parsing: Applying nested rules to text and tables to identify chemical relationships.
    6. Dependency Resolution: Resolving interdependencies between different elements.
    7. Output Generation: Producing a set of mutually consistent chemical records.

  9. Use publisher-specific scrapers in chemdataextractor.scrape.pub

    master

    The chemdataextractor.scrape.pub module contains scrapers tailored for specific scientific publishers. Use these modules to target data from the following sources:

    • NLM: chemdataextractor.scrape.pub.nlm
    • RSC (Royal Society of Chemistry): chemdataextractor.scrape.pub.rsc
    • Springer: chemdataextractor.scrape.pub.springer
    • Elsevier: chemdataextractor.scrape.pub.elsevier

  10. Using specialized unit models for physical quantities

    master

    For representing physical measurements, ChemDataExtractor provides a specialized unit modeling system. Instead of generic values, you can use dimension-specific models to ensure semantic correctness.

    Available specialized modules include:

    • chemdataextractor.model.units.unit: Basic unit definitions.
    • chemdataextractor.model.units.dimension: Dimensional analysis components.
    • chemdataextractor.model.units.quantity_model: Models for representing quantities.
    • chemdataextractor.model.units.length: Models for length measurements.
    • chemdataextractor.model.units.mass: Models for mass measurements.
    • chemdataextractor.model.units.time: Models for time measurements.
    • chemdataextractor.model.units.temperature: Models for temperature measurements.

  12. Use specialized document readers for different file formats and publishers

    master

    ChemDataExtractor provides a variety of specialized reader modules designed to parse different document formats and publisher-specific structures. Depending on your source material, you should import and use the corresponding reader module from chemdataextractor.reader.

    Available reader types include:

    • General Formats:

      • chemdataextractor.reader.base: The base reader implementation.
      • chemdataextractor.reader.pdf: For parsing PDF documents.
      • chemdataextractor.reader.plaintext: For parsing plain text files.
      • chemdataextractor.reader.markup: For parsing markup-based documents.

    • Publisher-Specific Readers:

      • chemdataextractor.reader.acs: American Chemical Society (ACS) documents.
      • chemdataextractor.reader.rsc: Royal Society of Chemistry (RSC) documents.
      • chemdataextractor.reader.elsevier: Elsevier documents.
      • chemdataextractor.reader.springer: Springer documents.
      • chemdataextractor.reader.nlm: National Library of Medicine (NLM) formats.
      • chemdataextractor.reader.uspto: United States Patent and Trademark Office (USPTO) documents.

    • Other Specialized Readers:

      • chemdataextractor.reader.cssp: CSSP-specific reader.