DOCSEARCHbyAlgolia
Submit Docs

nlm-ingestor

repository·main·Indexed 23 days ago

https://github.com/nlmatics/nlm-ingestor

A service layer for the llmsherpa API providing RAG-friendly parsers for PDF, HTML, text, DOCX, and PPTX. It utilizes a modified Apache Tika server to extract structured data such as sections, tables, and lists, and supports OCR via Tesseract for scanned PDFs. The package includes a visual ingestor to transform raw Tika HTML output into structured formats and integrates with LayoutPDFReader for document parsing.

Tokens
2.6K
Snippets
8
Records
11
Agent score
79%

What's inside nlm-ingestor

  1. Supported file formats and parsing capabilities

    main

    The nlm-ingestor provides RAG-friendly parsers for several formats:

    PDF

    Uses a rule-based parser leveraging text coordinates (bounding boxes), graphics, and font data from a modified version of Apache Tika. Key features include:

    • Extraction of sections, subsections, and their levels.
    • Paragraph reconstruction (combining lines).
    • Links between sections and paragraphs.
    • Table extraction with section context.
    • List and nested list detection.
    • Joining content across page boundaries.
    • Removal of repeating headers, footers, and watermarks.
    • OCR support via apply_ocr.

    HTML

    A layout-aware parser that creates blocks designed to improve RAG performance through higher quality chunking.

    Text

    A parser that identifies lists, tables, and headers purely from text content without relying on visual or font metadata.

    DOCX, PPTX, and other Apache Tika formats

    These files are processed by converting them to HTML via Tika and then passing the output through the HTML parser.

  2. Install and run nlm-ingestor locally

    main

    To run the ingestor service directly on your machine, follow these steps:

    1. Install Java: Ensure you have the latest version of Java installed from the Oracle website.
    2. Start the Tika Server: Run the modified Tika JAR file included in the repository's jars/ directory:
      java -jar <path_to_nlm_ingestor>/jars/tika-server-standard-nlm-modified-2.9.2_v2.jar
    3. Install the package: Use pip to install the ingestor:
      pip install nlm-ingestor
    4. Run the Ingestion Daemon: Start the service using the following module command:
      python -m nlm_ingestor.ingestion_daemon
    java -jar <path_to_nlm_ingestor>/jars/tika-server-standard-nlm-modified-2.9.2_v2.jar

!pip install nlm-ingestor

python -m nlm_ingestor.ingestion_daemon

  3. Run nlm-ingestor using Docker

    main

    You can run the ingestor using a pre-built Docker image from the GitHub Container Registry.

    1. Pull the image:
      docker pull ghcr.io/nlmatics/nlm-ingestor:latest
    2. Run the container: Map port 5001 inside the container to a port of your choice on your host machine (e.g., 5010).
      docker run -p 5010:5001 ghcr.io/nlmatics/nlm-ingestor:latest-<version>
    docker pull ghcr.io/nlmatics/nlm-ingestor:latest

docker run -p 5010:5001 ghcr.io/nlmatics/nlm-ingestor:latest-<version>

  5. Parse documents using the NLM-modified Tika server

    main

    You can use the tika.parser library to extract content from files (like PDFs) using the NLM-modified Tika server. You can control whether OCR (Optical Character Recognition) is performed via the X-Tika-OCRskipOcr header.

    Note on OCR: OCR is slow. By default, it is recommended to set needs_ocr = False. If you need to parse scanned files, set needs_ocr = True and provide the necessary Tesseract-related headers.

    import os
from tika import parser

doc_loc = '/path/to/your/document.pdf'
needs_ocr = False
timeout = 3000

if not needs_ocr:
    headers = {
        "X-Tika-OCRskipOcr": "true",
    }
    parsed = parser.from_file(doc_loc, xmlContent=True, requestOptions={'headers': headers, 'timeout': timeout})
else:
    print("ocr")
    headers = {
        "X-Tika-OCRskipOcr": "false",
        "X-Tika-OCRoutputType": "hocr",
        "X-Tika-OCRocrEngineMode": "3",
        "X-Tika-PDFExtractInlineImages":"false",
        "X-Tika-Timeout-Millis": str(100*timeout),
        "X-Tika-OCRtimeoutSeconds": str(timeout),
    }
    parsed = parser.from_file(doc_loc, xmlContent=True, requestOptions={'headers': headers, 'timeout': timeout})

html_str = parsed["content"]

  6. Use the Visual Ingestor to process Tika HTML output

    main

    The visual_ingestor takes the raw HTML content returned by the NLM-modified Tika server and transforms it into a structured format (similar to LLMSherpa).

    Workflow:

    1. Parse the Tika output using BeautifulSoup to find all div elements with the class page.
    2. Initialize a visual_ingestor.Doc object using these pages.
    3. Access the processed HTML via parsed_doc.html_str.

    Warning: You can pass a subset of pages to visual_ingestor.Doc(pages[start:end], []), but this will result in incorrect document statistics and potentially inconsistent behavior compared to parsing the full document.

  7. Set up the NLM-modified Tika server

    main

    The nlm-ingestor requires a specific modified version of the Apache Tika server to function correctly. Using the default pytika server will not provide the necessary modifications.

    To set up the server:

    1. Install the latest version of Java.
    2. Run the NLM-modified Tika JAR file:
      java -jar /jars/tika-server-standard-nlm-modified-2.4.1_v4.jar
    3. Set the TIKA_SERVER_ENDPOINT environment variable in your Python environment to point to your running server (e.g., http://localhost:9998).
    os.environ["TIKA_SERVER_ENDPOINT"] = "http://localhost:9998"

  8. Use the llmsherpa API to parse documents

    main

    Once the ingestor server is running, you can use the llmsherpa API library to retrieve chunks for LLM projects. The base URL for the parsing endpoint is:

    http://localhost:5010/api/parseDocument?renderFormat=all

    Query Parameters

    • renderFormat=all: Required to get the full parsed output.
    • applyOcr=yes: Enables OCR (using Tesseract) for scanned pages in PDFs.
    • useNewIndentParser=yes: Uses a different algorithm to assign header levels based on indentation.

    Note: For production environments, it is recommended to run this server behind a secure gateway like Nginx or a cloud gateway.

    http://localhost:5010/api/parseDocument?renderFormat=all

  9. Use LayoutPDFReader to parse documents

    main

    The LayoutPDFReader class is used to parse documents (PDFs, URLs, or local file paths) by communicating with an llmsherpa API endpoint.

    API URL Configuration

    You must provide a llmsherpa_api_url. This can be the hosted service or a local instance. To enable OCR, append &applyOcr=yes to the URL.

    Supported Input Formats

    The read_pdf method accepts:

    • Remote URLs (e.g., https://arxiv.org/pdf/1910.13461.pdf)
    • Local file paths (e.g., /Users/path/to/file.pdf)
    • Web pages or other text-based URLs
    from llmsherpa.readers import LayoutPDFReader

# Example using a local API with OCR enabled
llmsherpa_api_url = "http://localhost:5001/api/parseDocument?renderFormat=all&useNewIndentParser=true&applyOcr=yes"
pdf_url = "https://en.wikipedia.org/wiki/Language_model"

pdf_reader = LayoutPDFReader(llmsherpa_api_url)
doc = pdf_reader.read_pdf(pdf_url)

  10. Extract content from parsed documents

    main

    Once a document is parsed into a doc object, you can access its structure and content using several methods:

    • doc.to_html(): Converts the entire document structure to HTML.
    • doc.sections(): Returns a list of document sections.
    • doc.sections()[i].to_text(): Extracts plain text from a specific section.
    • doc.sections()[i].to_html(include_children=True, recurse=True): Converts a specific section to HTML, including its nested children.
    • doc.sections()[i].block_json: Returns the JSON representation of a section's blocks.
    • doc.sections()[i].bbox: Returns the bounding box for a section.

  11. Configure Visual Ingestor debug flags

    main

    The visual_ingestor and its sub-modules provide several debug flags to inspect the parsing process at different stages (blocks, indents, tables, etc.). These are useful for troubleshooting the transformation from raw HTML to structured document.

    Available debug flags include:

    • block_renderer.HTML_DEBUG
    • visual_ingestor.LINE_DEBUG
    • indent_parser.LEVEL_DEBUG
    • indent_parser.NO_INDENT
    • visual_ingestor.MIXED_FONT_DEBUG
    • table_parser.TABLE_DEBUG
    • table_parser.TABLE_COL_DEBUG
    • table_parser.TABLE_HG_DEBUG
    • table_parser.TABLE_BOUNDS_DEBUG
    • visual_ingestor.HF_DEBUG
    • order_fixer.REORDER_DEBUG
    • visual_ingestor.MERGE_DEBUG
    • table_parser.TABLE_2_COL_DEBUG
    from nlm_ingestor.ingestor.visual_ingestor import block_renderer, visual_ingestor, indent_parser, table_parser, order_fixer

block_renderer.HTML_DEBUG = True
visual_ingestor.LINE_DEBUG = False
indent_parser.LEVEL_DEBUG = False
indent_parser.NO_INDENT = False
visual_ingestor.MIXED_FONT_DEBUG = False
table_parser.TABLE_DEBUG = False
table_parser.TABLE_COL_DEBUG = False
table_parser.TABLE_HG_DEBUG = False
table_parser.TABLE_BOUNDS_DEBUG = False
visual_ingestor.HF_DEBUG = False
order_fixer.REORDER_DEBUG = False
visual_ingestor.MERGE_DEBUG = False
table_parser.TABLE_2_COL_DEBUG = False