DOCSEARCHbyAlgolia
Submit Docs

invoice2data

repository·master·Indexed 24 days ago

https://github.com/invoice-x/invoice2data

A command-line tool and Python library for automating data extraction from PDF invoices. It utilizes a three-step pipeline consisting of pluggable text extraction backends (such as pdfium, pdftotext, and various OCR engines), regex-based YAML/JSON templates, and structured output generation in CSV, JSON, or XML. The library includes optional AI support for fallback extraction and template generation via providers like OpenAI, Gemini, and Ollama.

Tokens
20.8K
Snippets
60
Records
126
Agent score
80%

What's inside invoice2data

  1. Configure input backends and template pinning

    master

    The default input reader cascade is [pdfium, pdftotext]. pypdfium2 is the default for speed, falling back to pdftotext if it fails to match or misses required fields.

    Template Pinning: If a template is sensitive to layout (e.g., relies on specific column alignment), you can pin the backend by adding input_module: pdftotext as a top-level key in the template YAML. This ensures the template always uses pdftotext even in auto mode.

    Manual Override: Use the --input-reader CLI flag or input_module= to force a specific backend. Note that forcing a backend ignores any input_module pin defined within a template.

  2. Understand template matching and data extraction

    master

    Templates (defined in YAML or JSON) are the core of the extraction logic. They work in two phases:

    1. Template Selection

    The system matches the extracted text against the keywords defined in the template. A match identifies which layout to use.

    2. Field Extraction

    Once a template is selected, it uses several mechanisms to pull data:

    • Regex: Regular expressions to locate specific patterns.
    • Static: Fixed values for fields that never change.
    • Plugins: Specialized tools for complex data:
      • lines: For line-item extraction.
      • tables: For general table structures.
      • camelot: For advanced table extraction.

    After extraction, data is normalized to a canonical field schema and validated for accuracy (e.g., verifying tax totals).

  3. Use AI for fallback extraction and template generation

    master

    The AI subsystem is an opt-in feature that requires the ai extra to be installed. It is provider-pluggable, supporting both cloud LLMs and local instances like Ollama.

    Key AI capabilities include:

    • LLM fallback extraction: Using AI to extract data when standard template-based extraction fails.
    • AI template generation: Using AI to help create new templates.
    • Provider interface: A pluggable interface to connect different AI providers.

    Refer to the ai documentation for specific configuration details.

  4. Configure template keywords and exclusion rules

    master

    Templates use regex patterns to identify which invoices they should process.

    • keywords (Required): A list of regex patterns. All patterns in this list must match the invoice for the template to be applied. Use specific identifiers like VAT numbers, emails, or unique company names to avoid collisions with other templates.
    • exclude_keywords (Optional): A list of regex patterns. If any of these patterns match the invoice, the template will be skipped. This is useful for distinguishing between actual invoices and other documents like receipts or payment confirmations.

  5. Use the optional AI fallback

    master

    If the template matching process fails (either no template matches the keywords or a matched template is missing required fields), invoice2data can optionally use an LLM to extract the canonical fields.

    Key details:

    • This is an opt-in feature.
    • It is text-only (it processes the extracted text, not the raw image/PDF directly).
    • It targets the same canonical schema used by the template system.

  6. How input modules and backends work

    master

    invoice2data uses backends to read input files. A backend can be resolved by its name or by passing a module object.

    If no specific backend is forced, the library follows an ordered cascade of attempts (as described in the internal documentation) and falls back to OCR if text extraction fails. Backends implement a common interface and use is_available() to self-exclude if their optional dependencies are not installed.

    Supported input modules include:

    • PDF Backends: pdfium (default), pdftotext, pdfplumber, pdfminer, pdfoxide, hotpdf.
    • Text: text.
    • OCR Backends: tesseract, ocrmypdf, docTR (deep-learning), PaddleOCR (deep-learning), Google Vision (cloud-based).

  7. Supported configuration shorthands in 1.0

    master

    The following shorthand patterns are supported in 1.0 but are under review for potential changes in future major releases:

    • Field-name shorthand: Using field: 'regex' as a terse alternative to field: {parser: regex, regex: 'regex'}.
    • Auto-typing by name: Fields whose names start with amount or start/end with date are automatically coerced to float or date without requiring an explicit type: key.
    • static_ prefix: Using a prefix like static_vat: FR123... to define a constant value. This is equivalent to using parser: static.
    • sum_amount... prefix: Using a prefix followed by a list of regexes to sum the matches. This is equivalent to using parser: regex with group: sum.

  8. Extract tax lines and resolve tax rates

    master

    To process VAT/tax summaries, use the tax_lines key. This allows you to map tax codes found on individual line items to specific rates found in a summary table.

    Required Fields for Tax Lines:

    • price_subtotal: The amount excluding tax.
    • line_tax_percent: The percentage of tax.
    • line_tax_amount: The amount of tax.

    Automatic Resolution: If you capture a field named line_tax_code on your standard lines and a matching line_tax_code + line_tax_percent in your tax_lines, invoice2data will automatically join them. The rate from the summary is applied to the line, and line_tax_amount is computed from price_subtotal.

      lines:
    parser: lines
    start: 'Article'
    end: 'Subtotal'
    line: '(?P<line_tax_code>\d)\s+(?P<description>.+?)\s+(?P<price_subtotal>[\d.,]+)'
    types:
      price_subtotal: float
  tax_lines:
    parser: lines
    start: 'BTW'
    end: '\Z'
    line: '(?P<line_tax_code>\d)\s+(?P<line_tax_percent>[\d.,]+)%'
    types:
      line_tax_percent: float

  9. Normalize Units of Measure (UoM) to UNECE Rec 20

    master

    invoice2data automatically maps common printed unit of measure (UoM) literals to their corresponding UNECE Recommendation 20 codes. This is used to ensure compatibility with systems like OCA's account_invoice_import_invoice2data.

    Mapping Logic:

    • If a template captures unece_code directly, it takes precedence.
    • Otherwise, the mapping fills unece_code based on the uom literal.
    • Unknown literals are preserved in the uom field.

    Common Mappings:

    Literal (case-insensitive)UNECE Code
    l, ltr, liter, litre, ℓLTR
    mlMLT
    kg, kilogramKGM
    g, gr, gramGRM
    m, meter, metreMTR
    cm, mm, kmCMT, MMT, KMT
    pcs, pc, piece, ea, unit, stuk, stk, xH87
    setSET
    h, hour, uurHUR
    min, minuteMIN
    d, day, dagDAY
    month, maand, mndMON
    year, jaarANN

  10. How invoice2data processes invoices

    master

    invoice2data follows a multi-stage pipeline to convert PDF or image files into structured data (CSV, JSON, or XML). The workflow is as follows:

    1. Text Extraction: The system uses an input module to extract text from the file. It uses a cascade approach by default (starting with pdfium, then pdftotext, and finally OCR via ocrmypdf).
    2. Template Matching: The extracted text is compared against keywords defined in YAML or JSON templates. Once a template is matched, the system uses regexes and plugins to locate specific fields.
    3. Data Extraction: Values are pulled using regex, static values, or plugins (like lines, tables, or camelot).
    4. Normalization & Validation: Extracted fields are normalized to a canonical schema and validated (e.g., checking tax totals or typo-aware field names).
    5. AI Fallback (Optional): If no template matches or required fields are missing, an optional LLM can be used to extract data from the text.
    6. Output: The final structured data is exported to CSV, JSON, or XML, or used to rename the source file.

  11. How invoice2data works

    master

    invoice2data follows a three-step pipeline to automate data extraction:

    1. Text Extraction: It extracts text from PDF files using a pluggable, cascading backend system. Supported backends include pdfium (default), pdftotext, text, pdfminer, pdfplumber, or OCR engines like tesseract, ocrmypdf, docTR, paddleocr, and gvision.
    2. Template Matching: It searches for regex patterns in the extracted text using a YAML or JSON-based template system. It also supports an optional AI fallback.
    3. Output Generation: It saves the structured results as CSV, JSON, or XML, and can optionally rename PDF files to match the extracted content.

  12. Understand the input-backend cascade and fallback mechanism

    master

    The invoice2data extraction process uses a cascade of text-extraction backends to balance speed and accuracy. By default, the system attempts to use a fast backend first and falls back to pdftotext (Poppler) if the fast backend fails to match a template, misses a required field, or returns empty lines or tables data.

    Backend Comparison

    backendaccuracyspeed (raw to_text)
    pdftotext85.9 %20.6 ms/file
    pdfium42.9 %4.2 ms/file
    pdfoxide34.7 %5.4 ms/file
    pdfminer24.7 %77.9 ms/file
    pdfplumber7.1 %120.9 ms/file
    hotpdf4.1 %368.1 ms/file

    pdftotext is considered the accuracy anchor because its -layout mode is what most templates expect. pdfium (pypdfium2) is the recommended fast-path backend due to its high speed and superior accuracy among layout-less backends.