OnnxTR (Onnx Text Recognition)

When detect_layout=True and detect_tables=True are set in ocr_predictor, the resulting Document object contains structured information about the page layout and tables.

• Layout Regions: Accessible via page.layout. Each region contains a type (e.g., Text, Title, Table, Picture), confidence, and geometry.
• Tables: Accessible via page.tables. Words inside tables are regrouped into cells. Each table provides num_rows, num_cols, and a list of cells containing row_start, col_start, and value.

Note: Enabling detect_tables automatically enables layout detection.

from onnxtr.io import DocumentFile
from onnxtr.models import ocr_predictor

doc = DocumentFile.from_pdf("path/to/your/doc.pdf")

model = ocr_predictor(detect_layout=True, detect_tables=True)
result = model(doc)

page = result.pages[0]

# Layout regions
for region in page.layout:
    print(region.type, region.confidence, region.geometry)

# Tables
for table in page.tables:
    print(f"{table.num_rows}x{table.num_cols}")
    for cell in table.cells:
        print(cell.row_start, cell.col_start, cell.value)

Install OnnxTR using pip with specific extras depending on your hardware and requirements.

Hardware Recommendations:

• NVIDIA GPU: Use gpu variants.
• Intel CPU/GPU: Use openvino variants.
• Apple Silicon Mac: Use cpu variants (CoreML is auto-detected).
• Other: Use cpu variants.

Note: For GPU support, CUDA and cuDNN must be installed on your system before installing the package. For OpenVINO, the default device is CPU with FP32 precision; configuration can be adjusted via ONNX Runtime settings.

# standard cpu support
pip install "onnxtr[cpu]"
pip install "onnxtr[cpu-headless]"

# with gpu support
pip install "onnxtr[gpu]"
pip install "onnxtr[gpu-headless]"

# OpenVINO cpu | gpu support for Intel CPUs | GPUs
pip install "onnxtr[openvino]"
pip install "onnxtr[openvino-headless]"

# with HTML support
pip install "onnxtr[html]"

# with support for visualization
pip install "onnxtr[viz]"

# with support for all dependencies
pip install "onnxtr[html, gpu, viz]"

You can pass custom ONNX Runtime SessionOptions and providers to the model engines using EngineConfig. This allows for fine-grained control over hardware acceleration (e.g., specifying CUDA device IDs) and memory management.

To dynamically manage memory, you can provide a run_options_provider callback to EngineConfig that modifies RunOptions during inference.

from onnxruntime import SessionOptions

from onnxtr.models import ocr_predictor, EngineConfig

general_options = (
    SessionOptions()
)  # For configuartion options see: https://onnxruntime.ai/docs/api/python/api_summary.html#sessionoptions
general_options.enable_cpu_mem_arena = False

# NOTE: The following would force to run only on the GPU if no GPU is available it will raise an error
# List of strings e.g. ["CUDAExecutionProvider", "CPUExecutionProvider"] or a list of tuples with the provider and its options e.g.
# [("CUDAExecutionProvider", {"device_id": 0}), ("CUDAExecutionProvider", {"arena_extend_strategy": "kSameAsRequested"})]
providers = [
    ("CUDAExecutionProvider", {"device_id": 0, "cudnn_conv_algo_search": "DEFAULT"})
]  # For available providers see: https://onnxruntime.ai/docs/execution-providers/

engine_config = EngineConfig(session_options=general_options, providers=providers)
# We use the default predictor with the custom engine configuration
# NOTE: You can define different engine configurations for detection, recognition and classification depending on your needs
predictor = ocr_predictor(det_engine_cfg=engine_config, reco_engine_cfg=engine_config, clf_engine_cfg=engine_config)

The Document object returned by the model can be exported into several formats using the export_as() method or specialized methods.

Supported Formats via export_as(format, ...):

• markdown (or md): Can specify direction (e.g., "auto", "ltr", "rtl", "ttb-rtl", "ttb-ltr").
• asciidoc (or adoc)
• html
• xml (hOCR format)
• text (or txt)
• json (or dict)

Other Export Methods:

• result.export(): Returns a nested dictionary.
• result.render(): Returns human-readable text.
• result.export_as_xml(): Returns XML (hOCR format) as a list of (bytes, ElementTree) tuples per page.

result.export_as("markdown")  # or "md"
result.export_as("asciidoc")  # or "adoc"
result.export_as("html")
result.export_as("xml")  # hOCR, one (bytes, ElementTree) tuple per page
result.export_as("text")  # or "text"
result.export_as("json")  # or "dict"

# Reading order can be resolved on export
result.export_as("markdown", direction="auto")  # auto | ltr | rtl | ttb-rtl | ttb-ltr

You can upload your own trained OnnxTR models to the HuggingFace Hub using push_to_hf_hub. You must first authenticate using login_to_hub(). When pushing, specify the task (e.g., detection, recognition, or classification) and the arch (the model architecture name).

from onnxtr.models import parseq, push_to_hf_hub, login_to_hub
from onnxtr.utils.vocabs import VOCABS

# Login to the hub
login_to_hub()

# Recognition model
model = parseq("~/onnxtr-parseq-multilingual-v1.onnx", vocab=VOCABS["multilingual"])
push_to_hf_hub(
    model,
    model_name="onnxtr-parseq-multilingual-v1",
    task="recognition",  # The task for which the model is intended [detection, recognition, classification]
    arch="parseq",  # The name of the model architecture
    override=False,  # Set to `True` if you want to override an existing model / repository
)

# Detection model
model = linknet_resnet18("~/onnxtr-linknet-resnet18.onnx")
push_to_hf_hub(model, model_name="onnxtr-linknet-resnet18", task="detection", arch="linknet_resnet18", override=True)