pdfminer.six Documentation

repository·master·Indexed 27 days ago

https://github.com/pdfminer/pdfminer.six

A community-maintained Python tool for extracting and analyzing information from PDF documents. It focuses on text data, layout analysis, and font/color information. The library provides a high-level API for text extraction, a CLI tool (pdf2txt.py) for text and image extraction, and utilities for parsing AcroForm interactive fields, Table of Contents outlines, and internal PDF structures via dumppdf.py.

Tokens
6.7K
Snippets
22
Records
33
Agent score
91%

What's inside pdfminer.six

  1. Understand the purpose of cmaprsrc resources

    master
    The cmaprsrc directory contains Adobe CMap resources. These resources are required to correctly decode text data written in CJK (Chinese, Japanese, Korean) languages within PDF documents. Without these CMaps, text extraction for these specific languages may fail or produce incorrect characters.
  2. Understand the Layout Analysis Algorithm

    master

    Because PDF files only store characters and their coordinates rather than semantic structures like paragraphs or sentences, pdfminer.six uses a heuristic-based layout analysis algorithm to reconstruct these structures.

    The algorithm operates in three hierarchical stages:

    1. Grouping characters into words and lines: Uses character bounding boxes to identify proximity.
    2. Grouping lines into boxes: Groups lines that are horizontally overlapping and vertically close.
    3. Grouping textboxes hierarchically: Repeatedly merges the closest textboxes based on the area between them.

    The precision of this reconstruction is controlled via parameters in the LAParams class.

  3. Extract AcroForm interactive form fields from a PDF

    master

    You can extract interactive AcroForm field names and values from a PDF by parsing the document catalog and resolving the AcroForm entry. Note that only AcroForm interactive forms are supported; XFA forms are not supported.

    To perform extraction, follow these steps:

    1. Initialize PDFParser and PDFDocument.
    2. Resolve the document catalog using resolve1(doc.catalog).
    3. Check for the presence of the 'AcroForm' key in the resolved catalog.
    4. Access the 'Fields' list within the resolved AcroForm object.
    5. Iterate through the fields, resolving each field object with resolve1().
    6. Retrieve the field name (key 'T') and field value (key 'V').
    7. Use decode_text() to handle field names and a custom decoding logic to handle PSLiteral, PSKeyword, and bytes values.
    from pdfminer.pdfparser import PDFParser
    from pdfminer.pdfdocument import PDFDocument
    from pdfminer.pdftypes import resolve1
    from pdfminer.psparser import PSLiteral, PSKeyword
    from pdfminer.utils import decode_text
    
    
    data = {}
    
    
    def decode_value(value):
        # decode PSLiteral, PSKeyword
        if isinstance(value, (PSLiteral, PSKeyword)):
            value = value.name
    
        # decode bytes
        if isinstance(value, bytes):
            value = decode_text(value)
    
        return value
    
    
    with open(file_path, 'rb') as fp:
        parser = PDFParser(fp)
    
        doc = PDFDocument(parser)
        res = resolve1(doc.catalog)
    
        if 'AcroForm' not in res:
            raise ValueError("No AcroForm Found")
    
        fields = resolve1(doc.catalog['AcroForm'])['Fields']  # may need further resolving
    
        for f in fields:
            field = resolve1(f)
            name, values = field.get('T'), field.get('V')
    
            # decode name
            name = decode_text(name)
    
            # resolve indirect obj
            values = resolve1(values)
    
            # decode value(s)
            if isinstance(values, list):
                values = [decode_value(v) for v in values]
            else:
                values = decode_value(values)
    
            data.update({name: values})
    
            print(name, values)
  4. Resolve ToC entry destinations to page numbers

    master

    Since get_outlines() returns references (Dest, A, or SE) rather than page numbers, you must implement a resolver to map these references to actual page numbers.

    A resolver works by:

    1. Mapping all page objects in the document to their respective page numbers using PDFPage.create_pages(document).
    2. Recursively resolving PDFObjRef types, DICTIONARY types (looking for the D key), LIST types (finding the first PDFObjRef), or NAMED_REF types (using document.get_dest(ref)).
    3. Checking if the resolved object is a page by verifying if its Type is LITERAL_PAGE.
    from enum import Enum, auto
    from pathlib import Path
    from typing import Any, Optional
    from pdfminer.pdfdocument import PDFDocument
    from pdfminer.pdfpage import PDFPage, LITERAL_PAGE
    from pdfminer.pdfparser import PDFParser
    from pdfminer.pdftypes import PDFObjRef
    
    class PDFRefType(Enum):
        PDF_OBJ_REF = auto()
        DICTIONARY = auto()
        LIST = auto()
        NAMED_REF = auto()
        UNK = auto()
    
    class RefPageNumberResolver:
        def __init__(self, document: PDFDocument):
            self.document = document
            # obj_id -> page_number
            self.objid_to_pagenum: dict[int, int] = {
                page.pageid: page_num
                for page_num, page in enumerate(PDFPage.create_pages(document), 1)
            }
    
        @classmethod
        def get_ref_type(cls, ref: Any) -> PDFRefType:
            if isinstance(ref, PDFObjRef):
                return PDFRefType.PDF_OBJ_REF
            elif isinstance(ref, dict) and "D" in ref:
                return PDFRefType.DICTIONARY
            elif isinstance(ref, list) and any(isinstance(e, PDFObjRef) for e in ref):
                return PDFRefType.LIST
            elif isinstance(ref, bytes):
                return PDFRefType.NAMED_REF
            else:
                return PDFRefType.UNK
    
        @classmethod
        def is_ref_page(cls, ref: Any) -> bool:
            return isinstance(ref, dict) and "Type" in ref and ref["Type"] is LITERAL_PAGE
    
        def resolve(self, ref: Any) -> Optional[int]:
            ref_type = self.get_ref_type(ref)
    
            if ref_type is PDFRefType.PDF_OBJ_REF and self.is_ref_page(ref.resolve()):
                return self.objid_to_pagenum.get(ref.objid)
            elif ref_type is PDFRefType.PDF_OBJ_REF:
                return self.resolve(ref.resolve())
    
            if ref_type is PDFRefType.DICTIONARY:
                return self.resolve(ref["D"])
    
            if ref_type is PDFRefType.LIST:
                return self.resolve(next(filter(lambda e: isinstance(e, PDFObjRef), ref)))
    
            if ref_type is PDFRefType.NAMED_REF:
                return self.resolve(self.document.get_dest(ref))
    
            return None
  5. Generate CMap JSON files for CJK support

    master

    To decode text data in CJK (Chinese, Japanese, Korean) languages, pdfminer.six requires CMap resources. These are stored as *.json.gz files in the pdfminer/cmap directory. If these files are missing, you can generate them using the tools/conv_cmap.py script and the Adobe CMap resources.

    ### On Linux/macOS (using make):
    ```bash
    make cmap

    On Windows:

    mkdir pdfminer\cmap
    python tools\conv_cmap.py -c B5=cp950 -c UniCNS-UTF8=utf-8 pdfminer\cmap Adobe-CNS1 cmaprsrc\cid2code_Adobe_CNS1.txt
    python tools\conv_cmap.py -c GBK-EUC=cp936 -c UniGB-UTF8=utf-8 pdfminer\cmap Adobe-GB1 cmaprsrc\cid2code_Adobe_GB1.txt
    python tools\conv_cmap.py -c RKSJ=cp932 -c EUC=euc-jp -c UniJIS-UTF8=utf-8 pdfminer\cmap Adobe-Japan1 cmaprsrc\cid2code_Adobe_Japan1.txt
    python tools\conv_cmap.py -c KSC-EUC=euc-kr -c KSC-Johab=johab -c KSCms-UHC=cp949 -c UniKS-UTF8=utf-8 pdfminer\cmap Adobe-Korea1 cmaprsrc\cid2code_Adobe_Korea1.txt
  6. Build documentation locally

    master

    To build the pdfminer.six documentation on your local machine, follow these steps:

    1. Create and activate a Python virtual environment (recommended).
    2. Install the documentation dependencies using pip install '.[docs]'.
    3. Run the build command using make.

    The documentation is generated using Sphinx and reStructuredText.

  7. Extract images from a PDF using pdf2txt.py

    master

    You can extract all images from a PDF file and save them to a directory using the pdf2txt.py command-line tool. This is useful for batch-extracting visual assets from documents.

    Ensure pdfminer.six is installed before running the command.

    $ pdf2txt.py example.pdf --output-dir cats-and-dogs
  8. Extract font names and sizes from PDF characters

    master

    To extract specific character properties like font names, font sizes, and colors, use pdfminer.high_level.extract_pages to get the layout hierarchy.

    Note that font information is typically stored at the LTChar level, as font properties can change for individual characters within a line or box. Higher-level layout elements like LTPage, LTTextBoxHorizontal, and LTTextLineHorizontal do not contain font information.

    Key attributes available on LTChar objects:

    • fontname: The name of the font.
    • size: The font size.
    • graphicstate.scolor: The stroking color (if specified in the PDF).