DOCSEARCHbyAlgolia
Submit Docs

docext Documentation

repository·main·Indexed 24 days ago

https://github.com/nanonets/docext

An on-premises document intelligence toolkit powered by vision-language models (VLMs). docext enables developers to convert PDFs and images to structured markdown, perform OCR-free information extraction using custom fields or pre-built templates (e.g., Invoices, Passports), and benchmark model performance via the Intelligent Document Processing (IDP) Leaderboard across tasks like KIE, VQA, and table extraction.

Tokens
6.3K
Snippets
16
Records
35
Agent score
84%

What's inside docext

  1. Overview of Docext capabilities

    main

    docext is an on-premises document intelligence toolkit powered by vision-language models (VLMs). It is designed for running document processing tasks entirely on your own infrastructure (Linux, MacOS).

    The toolkit provides three core capabilities:

    1. PDF & Image to Markdown Conversion: Transforms documents into structured markdown. It supports intelligent content recognition including LaTeX equations, signatures, watermarks, tables, and semantic tagging.
    2. Document Information Extraction: Provides OCR-free extraction of structured information (fields, tables, etc.) from documents like invoices and passports, including confidence scoring.
    3. Intelligent Document Processing Leaderboard: A benchmarking platform to evaluate vision-language model performance across various tasks like OCR, KIE, and table extraction.

  2. Features of PDF to Markdown conversion

    main

    The docext conversion engine provides several intelligent document processing features:

    • LaTeX Equation Recognition: Converts inline and block LaTeX equations into markdown.
    • Intelligent Image Description: Generates detailed descriptions for images within <img></img> tags.
    • Signature Detection: Detects signatures and watermarks, extracting text within <signature></signature> and <watermark></watermark> tags.
    • Page Number Detection: Extracts page numbers within <page_number></page_number> tags.
    • Form Elements: Converts checkboxes and radio buttons into Unicode symbols (☐, ☑, ☒).
    • Table Detection: Converts complex tables into HTML tables.

  3. Intelligent Document Processing Leaderboard benchmarks

    main

    The leaderboard evaluates model performance across seven key document intelligence challenges:

    • Key Information Extraction (KIE): Extracting structured fields from unstructured text.
    • Visual Question Answering (VQA): Understanding document content via questions.
    • Optical Character Recognition (OCR): Accuracy in recognizing printed and handwritten text.
    • Document Classification: Categorizing document types.
    • Long Document Processing: Reasoning over lengthy, context-rich documents.
    • Table Extraction: Extracting data from complex tabular formats.
    • Confidence Score Calibration: Evaluating the reliability of model predictions.

  4. Features of PDF and Image to Markdown conversion

    main

    The PDF and Image to Markdown feature converts documents into structured markdown with the following capabilities:

    • LaTeX Equation Recognition: Converts inline and block LaTeX equations into markdown.
    • Intelligent Image Description: Generates detailed descriptions for images within <img></img> tags.
    • Signature Detection: Detects signatures and watermarks, extracting text within <signature></signature> tags.
    • Watermark Detection: Detects watermarks, extracting text within <watermark></watermark> tags.
    • Page Number Detection: Detects page numbers, extracting them within <page_number></page_number> tags.
    • Form Elements: Converts checkboxes and radio buttons into Unicode symbols (☐, ☑, ☒).
    • Table Detection: Converts complex tables into HTML tables.

  5. Features of Docext information extraction

    main

    The docext extraction module allows for structured data retrieval from documents with the following features:

    • Flexible extraction: Define custom fields or use pre-built templates.
    • Pre-built templates: Includes ready-to-use templates for common document types like Invoices and Passports (with the ability to add/delete fields/columns).
    • Table extraction: Extracts structured tabular data.
    • Confidence scoring: Provides confidence levels for extracted information.
    • Multi-page support: Processes documents with multiple pages.
    • On-premises deployment: Runs on Linux and MacOS.
    • REST API: Provides programmatic access for application integration.

  6. Run docext via Docker

    main

    To run docext in a Docker container, ensure you have a GPU available. CPU mode is not supported.

    Using local models (vLLM)

    Provide a Hugging Face token via HUGGING_FACE_HUB_TOKEN if not using the default model.

    docker run --rm \
  --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --network host \
  --shm-size=20.24gb \
  --gpus all \
  nanonetsopensource/docext:v0.1.10 --model_name "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ"

    Using vendor-hosted models

    docker run --rm \
  --env "OPENROUTER_API_KEY=<secret>" \
  --network host \
  nanonetsopensource/docext:v0.1.10 --model_name "openrouter/meta-llama/llama-4-maverick:free"
    docker run --rm \
  --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --network host \
  --shm-size=20.24gb \
  --gpus all \
  nanonetsopensource/docext:v0.1.10 --model_name "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ"

  7. Install docext for development

    main

    To set up a local development environment for docext, use uv to create a virtual environment with Python 3.11, activate it, and install the package in editable mode. It is also recommended to install and set up pre-commit to manage git hooks.

    # create the virtual environment
uv venv --python=3.11

# activate the virtual environment
source .venv/bin/activate

# install the dependencies
pip install -e .

# pre-commit
uv pip install pre-commit
pre-commit install

  8. Install docext via uv

    main

    To install docext, it is recommended to use uv to manage a Python 3.11 virtual environment. You can install directly from PyPI or from the source repository.

    Install from PyPI

    # Install uv if not installed
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a virtual environment with python 3.11
uv venv --python=3.11
source .venv/bin/activate

# Install from PyPI
uv pip install docext

    Install from source

    # Install uv if not installed
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a virtual environment with python 3.11
uv venv --python=3.11
source .venv/bin/activate

# Install from source
git clone https://github.com/nanonets/docext.git
cd docext
uv pip install -e .
    curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python=3.11
source .venv/bin/activate
uv pip install docext

  9. View Benchmark results

    main

    Once the benchmark execution is complete, results are generated in the working directory:

    • Accuracy metrics: Saved to accuracy.csv.
    • Cost metrics: Saved to cost.csv.
    • Cached model outputs: Stored in the directory specified by cache_dir in your configuration (default is docext_benchmark_cache).

  10. Access the docext REST API

    main

    You can programmatically extract information from documents using the Gradio Client.

    1. Start the API server

    Run the app with a higher concurrency limit to handle multiple requests:

    python -m docext.app.app --concurrency_limit 10

    2. Use the Python Client

    Use the gradio_client to call the /extract_information endpoint. You must provide file_inputs (a list of image dictionaries), model_name, and fields_and_tables (a dictionary defining the schema).

    Note: For PDF files, use PDFConverter from docext.core.file_converters.pdf_converter to convert pages to images before sending them to the API.

    import pandas as pd
import concurrent.futures
from gradio_client import Client, handle_file
from docext.core.file_converters.pdf_converter import PDFConverter

# Example usage for image input
CLIENT_URL = "http://localhost:7860"
fields_and_tables = {"headers": ["name", "type", "description"], "data": [["invoice_number", "field", "Invoice number"]], "metadata": None}
file_inputs = [{"image": handle_file("assets/invoice_test.jpeg")}]

client = Client(CLIENT_URL, auth=("admin", "admin"))
result = client.predict(
    file_inputs=file_inputs,
    model_name="hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ",
    fields_and_tables=fields_and_tables,
    api_name="/extract_information"
)

  11. Start the docext Web Interface

    main

    The docext package includes a Gradio-based web interface for document processing. By default, it is available at http://localhost:7860 with credentials admin/admin.

    Default configuration

    python -m docext.app.app

    Custom configuration

    You can specify a model and image size using flags:

    python -m docext.app.app --model_name "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ" --max_img_size 1024

    Available flags:

    • --model_name: The name of the model to use.
    • --max_img_size: Maximum image size.
    • --server_port: Change the port (default is 7860).
    • --concurrency_limit: Increase the number of parallel requests (default is 1).
    python -m docext.app.app --model_name "hosted_vllm/Qwen/Qwen2.5-VL-7B-Instruct-AWQ" --max_img_size 1024