DOCSEARCHbyAlgolia
Submit Docs

Docling Core

repository·main·Indexed 42 days ago

https://github.com/docling-project/docling-core

A Python library (v2.97.0) that defines core data types and transformation logic for the Docling ecosystem. It provides the DoclingDocument Pydantic model and APIs for serialization, chunking, and profiling. Includes the dclq CLI for querying structured DocLang documents (.dclg, .dclx, .xml) and the docling-serialize CLI for converting documents to formats like HTML, Markdown, and JSON.

Tokens
13.2K
Snippets
40
Records
59
Agent score
55%

What's inside Docling Core

  1. Overview of dclq

    main

    dclq is a deterministic, read-only CLI tool designed for querying DocLang documents. It allows developers to inspect, outline, and search semantic units within a document using XPath and regular expressions.

    Unlike standard text search tools, dclq operates on the semantic structure of the document (headings, lists, tables, fields, etc.) rather than just flattened text. It provides reusable XPath addresses for every result, enabling precise retrieval of document sections.

    Core Workflow:

    1. Inspect or outline a document.
    2. Grep or list semantic units.
    3. Receive bounded results with XPath addresses.
    4. Retrieve a specific section or exact source selection using show or select.

  2. Overview of Docling Core features

    main

    Docling Core provides the foundational data models and transformation APIs used in the Docling ecosystem. Key features include:

    • DoclingDocument: A Pydantic-based data model for document representation, supporting building, basic operations, and exporting to formats like Markdown and HTML.
    • Serialization API: An extensible API for converting documents into various formats, providing built-in implementations used by DoclingDocument helpers.
    • Chunking API: Provides built-in chunking capabilities with an extensible design to support custom use cases for generative AI applications.
    • Profiling API: Enables extraction of statistics from DoclingDocument objects (individual or collections), including metrics on structure (pages, tables, pictures, text) and statistical distributions (deciles, histograms).

  3. Extend serialization with the Serialization API

    main
    The Serialization API is designed to be easily extensible. While it provides feature-rich built-in implementations, developers can implement their own serialization logic to meet specific requirements. The helper methods on DoclingDocument are built upon these core serialization implementations.

  4. Analyze document structure with the Profiling API

    main
    The Profiling API allows you to extract comprehensive statistics from DoclingDocument objects. This applies to both single documents and large collections. You can retrieve metrics on document structure (e.g., number of pages, tables, pictures, and text items) and generate statistical distributions like deciles and histograms, as well as visualizations for analyzing collections at scale.

  5. Understand dclq XPath addressing and identity

    main

    Generated XPath addresses in dclq are absolute and namespace-qualified with the d prefix. They are based on element siblings and are positional among same-name siblings.

    Example XPath Formats:

    • /d:doclang/d:heading[1]
    • /d:doclang/d:list[2]/d:ldiv[3]
    • /d:doclang/d:table[3]/d:fcel[7]

    Document Identity: Machine-readable results include a SHA-256 hash of the raw document.xml bytes. This ensures that a positional XPath cannot be silently reused against a different version of the same document. The identity remains consistent for identical XML whether it is inside or outside a .dclx archive.

  6. Use DoclingDocument for document modeling and export

    main
    The DoclingDocument is the central Pydantic model in Docling Core. It allows for advanced data model control, customizability, and interoperability. It provides an API for constructing documents and performing basic operations, such as exporting the document to Markdown or HTML.

  7. Customize document splitting with the Chunking API

    main
    The Chunking API provides built-in capabilities for splitting documents into chunks, which is essential for generative AI workflows. The API is designed for easy extension, allowing developers to implement custom chunking strategies tailored to specific use cases.

  8. Install and use dclq for document querying

    main

    Overview

    dclq is an experimental CLI tool designed to query structured DocLang documents (.dclg, .dclx, .xml). It provides a grep-like workflow for semantic document units (headings, paragraphs, tables, etc.) and returns reusable XPath addresses for every result.

    Installation

    Install dclq from PyPI:

    pip install dclq

    Quickstart Workflow

    1. Convert documents to DocLang format using docling:
    python -m pip install docling
docling report.pdf handbook.docx --to dclx --output converted
    1. Search for patterns using grep:
    dclq grep -i 'termination|cancellation' converted/report.dclx
    1. Enumerate units using list (without a pattern):
    dclq list converted/report.dclx --type table_cell --page 3
    pip install dclq

# Convert
python -m pip install docling
docling report.pdf handbook.docx --to dclx --output converted

# Search
dclq grep -i 'termination|cancellation' converted/report.dclx

  9. Use dclq in shell pipelines and automation

    main

    dclq is designed for pipeline-friendly execution with support for json and jsonl formats and standard exit codes.

    Common Pipeline Patterns

    List matching files:

    dclq grep -i 'human annotation' documents/*.dclx -l

    Count matches:

    dclq grep -i 'inference|runtime' documents/*.dclx -c

    Emit structured data for agents/pipelines:

    dclq grep -i 'accuracy|performance' documents/*.dclx \
  --page 1-4 \
  --type text,table_cell \
  --limit 10 \
  --format jsonl

    Conditional execution (Quiet mode):

    if dclq grep -q -F 'CONFIDENTIAL' document.dclx; then
  echo "classified"
fi

    Exit Codes

    CodeMeaning
    0At least one result
    1No results
    2Input, query, or usage error
    # List matching files
dclq grep -i 'human annotation' documents/*.dclx -l

# Count matches
dclq grep -i 'inference|runtime' documents/*.dclx -c

# Structured output for pipelines
dclq grep -i 'accuracy|performance' documents/*.dclx \
  --page 1-4 \
  --type text,table_cell \
  --limit 10 \
  --format jsonl

# Quiet mode for shell scripts
if dclq grep -q -F 'CONFIDENTIAL' document.dclx; then
  echo "classified"
fi

  10. Inspect and navigate DocLang documents with dclq

    main

    Inspect document metadata

    View a structural inventory of a document (pages, semantic unit counts, etc.):

    dclq inspect paper.dclx

    View heading hierarchy

    Print the document outline with associated XPaths:

    dclq outline paper.dclg

    Retrieve content by XPath

    Use show to get the content of a specific structural element:

    dclq show paper.dclg '/heading[13]' --section --max-chars 220 -n

    Evaluate XPath expressions

    Use select to run arbitrary XPath queries against the source XML. This is useful for counting elements or extracting specific string values:

    # Count page breaks
dclq select paper.dclg 'count(//page_break) + 1' --format json

# Extract caption text
dclq select paper.dclg 'normalize-space(string(//table[1]/caption))'
    dclq inspect paper.dclx
dclq outline paper.dclg
dclq show paper.dclg '/heading[13]' --section --max-chars 220 -n
dclq select paper.dclg 'count(//page_break) + 1' --format json