DocLLM: Layout-Aware Generative Language Model

DocLLM is a layout-aware generative language model designed for multimodal document understanding. Unlike traditional multimodal LLMs that rely on expensive image encoders, DocLLM incorporates spatial layout information by using bounding box information directly.

Key technical features include:

• Spatial Layout Integration: It avoids image encoders and instead uses bounding box information to capture the spatial structure of documents.
• Disentangled Attention: The model captures cross-alignment between text and spatial modalities by decomposing the transformer attention mechanism into a set of disentangled matrices.
• Infilling Pre-training: A pre-training objective that learns to infill text segments, allowing the model to handle irregular layouts and heterogeneous content common in enterprise documents (e.g., forms, invoices, receipts, and contracts).

DocLLM is designed to perform reasoning over visual documents across four core document intelligence tasks:

1. VQA (Visual Question Answering): Answering questions based on document content (e.g., DocVQA, WTQ, VisualMRC, DUDE, BizDocs).
2. NLI (Natural Language Inference): Reasoning about relationships between textual statements (e.g., TabFact).
3. KIE (Key Information Extraction): Extracting specific entities or data points from documents (e.g., KLC, CORD, FUNSD, DeepForm, PWC, SROIE, VRDU, BizDocs).
4. CLS (Classification): Categorizing documents into specific types (e.g., RVL-CDIP, BizDocs).