DOCSEARCHbyAlgolia
Submit Docs

OmniDocBench Documentation

repository·main·Indexed 22 days ago

https://github.com/opendatalab/omnidocbench

A high-quality benchmark and evaluation toolkit (omnidocbench-eval v1.6.0) for document parsing models. It evaluates layout detection, table recognition, formula recognition, and OCR across diverse document types. Key features include Character Detection Matching (CDM) for formula recognition, TEDS for tables, and integration with EvalScope. The toolkit supports end-to-end evaluation comparing Markdown output against ground truth JSON, with recommended deployment via Docker.

Tokens
28.7K
Snippets
45
Records
107
Agent score
83%

What's inside OmniDocBench

  1. Overview of OmniDocBench Benchmark

    main

    OmniDocBench is a diverse document parsing evaluation dataset designed for real-world scenarios. It covers 1,651 PDF pages across 10 document types (e.g., academic papers, financial reports, newspapers, textbooks, handwritten notes), 5 layout types, and 5 languages.

    Key Features:

    • Rich Annotations: Includes 28 block-level elements (paragraphs, titles, tables, etc.) and 4 span-level elements (text lines, inline formulas, subscripts, etc.) with precise localization and recognition results (Text, LaTeX for formulas, and both LaTeX/HTML for tables).
    • Reading Order: Provides annotated reading orders for document components.
    • Attribute Labels: Includes page attributes (5 types), text attributes (3 types), and table attributes (6 types).
    • High Quality: Data is verified through a combination of manual selection, intelligent annotation, and expert/LLM quality checks.
    • Evaluation Suite: Provides end-to-end and single-module evaluation code to ensure fairness and accuracy.

  2. Overview of OmniDocBench

    main

    OmniDocBench is a comprehensive benchmark designed for evaluating document parsing capabilities in real-world scenarios. It covers diverse document types (academic papers, financial reports, newspapers, textbooks, handwritten notes), layout types, and languages.

    Key Evaluation Dimensions

    • End-to-end evaluation: Assessing the complete parsing pipeline.
    • Layout detection: Identifying document elements like paragraphs, headings, and tables.
    • Table recognition: Extracting structured data from tables.
    • Formula recognition: Identifying and converting mathematical formulas.
    • Text OCR: Optical Character Recognition for text elements.

    Supported Metrics

    • Normalized Edit Distance
    • BLEU
    • METEOR
    • TEDS (for tables)
    • COCODet (mAP, mAR, etc., for detection tasks)

  3. What is Character Detection Matching (CDM)?

    main
    Character Detection Matching (CDM) is an image-level evaluation metric designed for formula recognition. Unlike traditional text-based metrics like BLEU or Edit Distance, which are sensitive to LaTeX representation diversity, CDM renders both the predicted LaTeX and the ground-truth LaTeX into images. It then uses visual feature extraction and localization to perform precise, spatially-aware character-level matching. This ensures more objective and equitable evaluation by focusing on the visual representation of the formula rather than just the string content.

  4. Understand OmniDocBench end-to-end evaluation methods

    main

    OmniDocBench provides two primary methods for end-to-end evaluation of document parsing models:

    1. end2end (Recommended): Uses the OmniDocBench JSON files as the Ground Truth. This method is preferred because it preserves sample categories and attribute information, allowing for the ignoring of specific categories and providing attribute-specific results. Configuration can be found in configs/end2end.yaml.

    2. md2md: Uses the OmniDocBench markdown format as the Ground Truth. This is a markdown-to-markdown comparison method.

    Evaluation covers four key dimensions:

    • Text paragraphs
    • Inline formulas
    • Tables
    • Reading order

  5. Evaluation Dimensions and Metrics in OmniDocBench

    main

    OmniDocBench supports evaluation across several dimensions using specific metrics:

    Evaluation Dimensions:

    • End-to-End Evaluation: Supports end2end and md2md methods.
    • Layout Detection
    • Table Recognition
    • Formula Recognition
    • Text OCR

    Supported Metrics:

    • Normalized Edit Distance
    • BLEU
    • METEOR
    • TEDS (for tables)
    • COCODet (including mAP, mAR, etc., for detection tasks)

  6. Understand the End-to-End (E2E) evaluation metric

    main

    End-to-end evaluation measures the accuracy of a model's PDF page parsing. The model's output for an entire PDF page (in Markdown format) is treated as the Prediction.

    The Overall score is calculated using the following formula:

    $$\text{Overall} = \frac{(1-\textit{Text Edit Distance}) \times 100 + \textit{Table TEDS} +\textit{Formula CDM}}{3}$$

    This metric combines text accuracy (Edit Distance), table structure accuracy (TEDS), and formula rendering accuracy (CDM).

  7. Configure worker counts for OmniDocBench

    main

    Worker counts control parallelism for matching and metric calculation. Setting these too high on small nodes can cause Out-Of-Memory (OOM) errors, especially with CDM (formula rendering).

    Recommended Worker Rules:

    • Standard (4 CPU / 8 GB RAM): Use 2 for match_workers, cdm_workers, and teds_workers. If unstable, use 1.
    • Large Nodes: Use 4 or more depending on available resources.
    • Avoid Defaults: Do not use the default value of 13 on small nodes.

    Worker Keys in Config:

    • dataset.match_workers: Page matching.
    • metrics.display_formula.cdm_workers: CDM rendering/comparison (memory-heavy).
    • metrics.table.teds_workers: Table TEDS calculation.

  8. Understand the OmniDocBench dataset format

    main

    The OmniDocBench dataset is provided in JSON format. Each entry in the top-level array represents a page and contains layout_dets (a list of block-level elements), page_info (metadata about the page), and an extra field for relationships between annotations.

    Key fields within layout_dets include:

    • category_type: The type of layout element (e.g., text_block).
    • poly: A list of coordinates [x1, y1, x2, y2, ...] defining the bounding polygon.
    • text, latex, html: Optional fields containing OCR results, LaTeX for formulas/tables, or HTML for tables.
    • line_with_spans: A list of span-level annotations contained within that block.
    • merge_list: Used for elements with merge relationships (e.g., connecting split paragraphs).

    The extra field contains a relation list that defines connections between annotations using source_anno_id and target_anno_id, with relationship types like parent_son or truncated (indicating a paragraph split due to layout).

    [{
    "layout_dets": [
        {
            "category_type": "text_block",
            "poly": [136.0, 781.0, 340.0, 781.0, 340.0, 806.0, 136.0, 806.0],
            "ignore": false,
            "order": 0,
            "anno_id": 0,
            "text": "xxx",
            "latex": "$xxx$",
            "html": "xxx",
            "attribute": {"xxx": "xxx"},
            "line_with_spans": [
                {
                    "category_type": "text_span",
                    "poly": [...],
                    "ignore": false,
                    "text": "xxx",
                    "latex": "$xxx$"
                }
            ],
            "merge_list": [
                {
                    "category_type": "text_block",
                    "poly": [...],
                    "line_with_spans": [...]
                }
            ]
        }
    ],
    "page_info": {
        "page_no": 0,
        "height": 1684,
        "width": 1200,
        "image_path": "xx/xx/",
        "page_attribute": {"xxx": "xxx"}
    },
    "extra": {
        "relation": [
            {
                "source_anno_id": 1,
                "target_anno_id": 2,
                "relation": "parent_son"
            },
            {
                "source_anno_id": 5,
                "target_anno_id": 6,
                "relation_type": "truncated"
            }
        ]
    }
}, ...]

  9. Understand the End-to-End evaluation metric

    main

    End-to-end evaluation measures how accurately a model parses PDF page content by comparing its Markdown output against ground truth.

    The Overall metric is calculated using the following formula:

    $$\text{Overall} = \frac{(1-\textit{Text Edit Distance}) \times 100 + \textit{Table TEDS} +\textit{Formula CDM}}{3}$$

    This combines text accuracy (via Edit Distance), table structure accuracy (via TEDS), and formula accuracy (via CDM).

  10. Understand the end-to-end evaluation methods

    main

    OmniDocBench provides two primary approaches for end-to-end evaluation of document parsing:

    1. end2end: This method uses OmniDocBench's JSON files as the Ground Truth. This is the recommended approach because it preserves category and attribute information, allowing for special category ignore operations and attribute-level result output.
    2. md2md: This method uses OmniDocBench's markdown format as the Ground Truth (detailed in the markdown-to-markdown evaluation section).

    The end2end evaluation assesses four dimensions:

    • Text paragraphs
    • Display formulas
    • Tables
    • Reading order

  11. Understand the OmniDocBench dataset structure

    main

    OmniDocBench is a benchmark dataset containing 1651 PDF pages across 10 document types, 5 layout types, and 5 language types. It features hierarchical annotations:

    • Block-level annotations: 28 categories including text paragraphs, titles, tables, figures, etc.
    • Span-level annotations: 4 categories including text lines, inline formulas, and superscripts/subscripts.
    • Content types: Text recognition (OCR), LaTeX for formulas and tables, and HTML for tables.
    • Reading Order: Annotations include the reading sequence for document components.
    • Attributes: Includes page-level attributes (e.g., language, layout) and block-level attributes (e.g., table structure, text rotation).

  12. Layout Detection Benchmark

    main
    OmniDocBench provides bounding box information for all document components on each PDF page, allowing for component-level layout detection evaluation. Results are typically reported as mAP (mean Average Precision) across different document types such as Books, Slides, Research Reports, Textbooks, Exam Papers, Magazines, Academic Literature, Notes, and Newspapers.