DOCSEARCHbyAlgolia
Submit Docs

Unstructured

repository·main·Indexed 12 days ago

https://github.com/Unstructured-IO/unstructured

An open-source library for ingesting and pre-processing unstructured data—including PDFs, HTML, images, and over 60 other file types—into structured formats to optimize data pipelines for LLMs and RAG applications. It features a `partition` function for automatic file type detection and supports Model Context Protocol (MCP) for agentic document processing.

Tokens
13K
Snippets
43
Records
59
Agent score
97%

What's inside Unstructured

  1. Overview of the Unstructured library

    main
    The unstructured library provides open-source components for ingesting and pre-processing unstructured data, including images and text documents like PDFs, HTML, Word docs, and over 60 other file types. It is designed to streamline and optimize data processing workflows for Large Language Models (LLMs) by transforming unstructured data into structured outputs through modular functions and connectors.

  2. Identify sample documents for parser testing

    main

    The example-docs directory provides several file types to test the library's various parsers:

    • example-10k.html: A 10-K SEC filing in HTML format (useful for testing HTML parsers).
    • layout-parser-paper.pdf: A PDF document (useful for testing PDF and layout parsers).
    • factbook.xml / factbook.xsl: XML and XSL files (useful for testing XML/stylesheet processing).

  3. Overview of the LayoutParser architecture

    main

    LayoutParser is a unified toolkit designed for Deep Learning (DL)-based Document Image Analysis (DIA). It provides a structured approach to document processing through several integrated components:

    • Core Library: Provides off-the-shelf tools for layout detection, OCR, visualization, and storage.
    • Layout Data Structure: A specialized data structure designed to represent document elements.
    • Model Zoo: A repository of pre-trained neural network models for various DIA tasks.
    • Customization Tools: Functions for efficient layout annotation and model training to improve accuracy on specific target datasets.
    • Community Platform (DIA Model Hub): A hub for sharing pre-trained models and full document digitization pipelines to promote reusability and reproducibility.

  4. Initialize LayoutParser models using semantic syntax

    main

    LayoutParser models are initialized using a specific semantic syntax string that identifies the dataset and the model architecture. This allows users to easily select pre-trained weights optimized for specific document types (e.g., scientific documents, newspapers, or historical documents).

    Syntax: lp://<dataset-name>/<model-architecture-name>

    Example: lp://PubLayNet/faster_rcnn_R_50_FPN_3x/config

  5. How LayoutParser data structures work

    main

    LayoutParser uses a hierarchical data model to represent document analysis results, providing three levels of abstraction:

    1. Coordinate: The foundation for spatial information. It supports Interval (1D), Rectangle (2D), and Quadrilateral (for skewed or distorted regions with 8 degrees of freedom).
    2. TextBlock: Combines Coordinate information with extra features like block text, element types, and reading orders.
    3. Layout: A collection of TextBlock objects (or nested Layout objects) that allows for batch processing and hierarchical structures.

    All these classes support a common set of transformation and operation APIs.

  6. Understand the TELNET Synch mechanism

    main

    The TELNET Synch mechanism is used to clear the data path and regain control from a "runaway" process. It works by combining a TCP Urgent notification with the TELNET command DATA MARK (DM).

    When a Synch is active, the data stream is immediately scanned for "interesting" signals (like IP, AO, or AYT), and intervening data is discarded. The DM command acts as the synchronizing mark, indicating that the recipient can return to normal data processing.

    To implement a Synch, send the DM as the only character in a TCP urgent mode send operation. If a protocol needs to ensure a specific command (like IP) gets through a blocked connection, it should follow this sequence:

    1. Send the TELNET IP character.
    2. Send the TELNET SYNC sequence (the DM character in TCP urgent mode).
    3. Send the target command string.
    4. Send the protocol's analog of the TELNET DM, if applicable.
    # Example Synch Sequence for a custom protocol
1. Send TELNET IP
2. Send DM (via TCP Urgent)
3. Send 'STOP' (custom command)
4. Send protocol-specific DM

  7. Explore Unstructured Pipelines for Production

    main

    For production-grade data processing, Unstructured offers Unstructured Pipelines. This enterprise offering provides:

    • Improved processing performance.
    • Advanced features like chunking, embedding, and image/table enrichment.
    • A low-code UI or API for managing workflows.

  9. Build a historical document digitization pipeline with LayoutParser

    main

    LayoutParser can be used to build complex digitization pipelines for documents with challenging layouts (e.g., vertical text, scan noise, or archaic fonts). A typical pipeline involves:

    1. Layout Detection: Using object detection models to identify document structures like columns, tokens, or specific regions (e.g., number regions).
    2. Document Reorganization: Using detected bounding boxes to rearrange text (e.g., creating dense text blocks from sparse tokens) to improve OCR performance.
    3. Customized OCR: Applying specialized OCR engines to specific regions identified by the layout model (e.g., using a CNN-RNN model for unique glyphs/fonts).
    4. Coordinate Transformation: Using LayoutParser's flexible coordinate system to map OCR results back to their original positions on the page.

  10. Run performance benchmarks

    main

    The benchmarking script tracks the time taken to partition a fixed set of test documents. Results can be stored in S3 along with architecture, instance type, and git hash metadata.

    Environment Variables

    • DOCKER_TEST: Set to true to run the benchmark inside a Docker container (default: false).
    • NUM_ITERATIONS: The number of iterations to run (e.g., 100) (default: 3).
    • INSTANCE_TYPE: The type of benchmark instance (e.g., "c5.xlarge") (default: unspecified).
    • PUBLISH_RESULTS: Set to true to publish results to an S3 bucket (default: false).

    Usage

    Execute the benchmark script from the repository root.

    ./scripts/performance/benchmark.sh

  11. Profile library performance (Time and Memory)

    main

    The profiling script allows you to inspect how time and memory are consumed during partitioning operations. It supports time profiling via cProfile and memory profiling via memray.

    Environment Variables

    • DOCKER_TEST: Set to true to run profiling inside a Docker container (default: false).

    Execution

    On Linux:

    ./scripts/performance/profile.sh

    On macOS (py-spy requires sudo):

    sudo -E ./scripts/performance/profile.sh

    Profiling Modes

    When running the script, you will be prompted to choose a mode:

    • 'run' mode: Profile custom files or select from existing test files. Note that test documents are synced from an S3 bucket to a local directory before execution.
    • 'view' mode: View previously generated profiling results.

    Visualization

    You can choose from several visualization formats, including flamegraphs, tables, trees, summaries, and statistics.

    # Linux
./scripts/performance/profile.sh

# macOS
sudo -E ./scripts/performance/profile.sh

  12. Install the Unstructured Python library

    main

    Install the library via pip. Depending on your requirements, you can install a minimal version or include extras for specific document types.

    • All document types: pip install "unstructured[all-docs]"
    • Minimal (Text, HTML, XML, JSON, Email): pip install unstructured
    • Specific extras: pip install "unstructured[docx,pptx]"

    System Dependencies

    Depending on the document types you parse, you may need to install these system packages:

    • libmagic-dev: Filetype detection
    • poppler-utils: Images and PDFs
    • tesseract-ocr: Images and PDFs (install tesseract-lang for extra languages)
    • libreoffice: MS Office docs
    • pandoc: Automatically handled via pypandoc-binary (no system install required)
    pip install "unstructured[all-docs]"