yomitoku

TableSemanticParser is an end-to-end pipeline that detects tables in document images and converts them into structured data. It performs table region detection, cell segmentation, OCR, cell role estimation (header vs. cell), and semantic structure estimation.

Key Features:

• Cell Roles: Identifies header vs cell.
• Key-Value Pairs: Extracts 1:1 relationships (e.g., Label $\rightarrow$ Value).
• Grid Structures: Extracts matrix-like tabular records (rows and columns).
• Searchable IDs: Tables are identified as t0, t1, etc., and cells use position-based IDs like r{row}c{col} (e.g., r0c0).

Note: This module only supports forms that have separator lines (ruled lines). Tables without separator lines are not supported.

YomiToku Extractor extracts structured data from images or PDFs based on a YAML schema. You can choose between two extraction modes:

1. Rule-based (yomitoku_extract): Uses Key-Value search, grid matching, and regular expressions. It does not require an LLM. This is fast and ideal for fixed-form documents (e.g., applications, reports, slips) where positions and text patterns are predictable.

2. LLM-based (yomitoku_extract_with_llm): Utilizes an LLM server (like vLLM). This is more flexible and ideal for non-fixed-form documents (e.g., business cards, receipts, invoices) where layouts and value patterns vary, as it can understand context.

To extract tabular data, set structure: table and define the columns using the columns key. Column matching (via cell_id, bbox, or description) is performed against the grid's header cells.

- name: order_items
  description: Order Details
  structure: table
  columns:
    - name: product
      description: Product Name
      type: string
    - name: quantity
      description: Quantity
      type: number
      normalize: numeric
    - name: price
      description: Amount
      type: number
      normalize: numeric

The TableSemanticParser does more than just detect cells; it estimates the semantic structure of a table using two distinct concepts:

1. kv_items (Key-Value): Used for data where a header (key) and a value have a 1:1 relationship. This is ideal for treating the table as a dictionary (e.g., Name $\rightarrow$ John Doe).
2. grids (Grid Data): Used for data that follows a row/column matrix structure. This is ideal for tables where multiple records exist as rows, and each row shares the same set of attributes/columns (e.g., a list of dates, times, and amounts).

Both kv_items and grids contain cell IDs, which can be used to look up detailed cell information (text, role, coordinates) in the cells object.

You can specify how to locate a target cell using four methods. If multiple methods are provided, they are evaluated in the following priority order:

1. cell_id: Direct specification using the ID assigned by TableSemanticParser (e.g., c12).
2. bbox: Bounding box specification [x1, y1, x2, y2]. Matches if overlap is 50% or more.
3. description: Text search (default). Performs a partial match against KV keys or table column headers.
4. regex: Regular expression pattern. Used for scalar fields to extract a match from cell or paragraph text when the key is uncertain but the format is known.

Note on Fallback Behavior: If cell_id is provided with description, cell_id takes precedence. If cell_id is not found, it falls back to description. regex is treated as the final resort if other methods fail.

Each field and table column supports four methods for identifying target cells. When multiple methods are provided, they are evaluated in the following priority order:

1. cell_id: Direct specification of a cell ID (e.g., c12) assigned by the TableSemanticParser.
2. bbox: Bounding box position specification [x1, y1, x2, y2]. Matches by 50% or more overlap.
3. description: Text search. Performs a partial match against KV item key text or grid header text.
4. regex: Regular expression pattern. Used for scalar fields to extract the first matching string from cell/paragraph text.

Note: If description and cell_id are both specified, cell_id takes priority. If cell_id is not found, it falls back to description. regex is evaluated as a last resort.

The parser provides two types of semantic structured outputs beyond raw cells:

1. kv_items (Key-Value): Represents data where a header (label) and a value correspond 1:1. This is ideal for dictionary-like data (e.g., Name $\rightarrow$ John Doe).
2. grids (Grid / Tabular Records): Represents data that follows a row/column matrix structure. This is ideal for repeating records with multiple attributes (e.g., a list of dates, times, and descriptions).

Both kv_items and grids store cell IDs, which can be used to look up detailed CellSchema information.

KV fields are used for forms. You can set structure: kv to explicitly signal to LLMs that the field is a single scalar value.

Merging Multiple Values

If a single key is associated with multiple value cells (e.g., an address spanning multiple rows), set merge_values: true. This collects all values, sorts them by position, and joins them with the specified separator.

Sorting Logic:

• If vertical spread (Y) $\ge$ horizontal spread (X): Sort top-to-bottom.
• Otherwise: Sort left-to-right.

- name: address
  structure: kv
  description: Address
  type: string
  merge_values: true
  separator: ""