pdf2json

repository·master·Indexed 24 days ago

https://github.com/modesty/pdf2json

A Node.js module and command-line utility that converts binary PDF files into structured JSON and plain text. Powered by a port of PDF.JS to Node.js, it supports text extraction, interactive form element parsing (AcroForms and XFA), and stream-based processing. It provides detailed page-level data including text blocks, lines, fills, and metadata, utilizing color and style dictionaries to optimize output size.

Tokens
10.3K
Snippets
17
Records
54
Agent score
78%

What's inside pdf2json

  1. Understand the parsed PDF output format

    master

    The parsed data from pdf2json is structured into four main sub-objects that describe the PDF document:

    • Transcoder: Contains the pdf2json version number.
    • Meta: Contains full document metadata (replaces the deprecated Agency and Id fields). This includes properties like PDFFormatVersion, Author, Creator, Producer, and an xmp metadata object.
    • Pages: An array of Page objects, each describing the elements (lines, fills, texts) and dimensions of a specific page.
    • Width: The PDF page width in page units.

    Note: The Agency and Id fields are deprecated since v2.0.0 and have been replaced by the Meta object.

  2. Parse interactive form elements (AcroForms and XFA)

    master

    The parser supports interactive form elements like text inputs, radio buttons, checkboxes, link buttons, and drop-down lists. These elements are grouped within the corresponding Page object:

    • Checkboxes and Radio Buttons: Located in the Boxsets array. A checkbox is an object in Boxsets where the boxes array contains exactly one element. A radio button group is an object in Boxsets where the boxes array contains multiple elements.
    • Other Elements: Text inputs, drop-down lists, and link buttons are located in the Fields array.

    Field Types Reference

    Element TypeLocation in JSONKey Details
    Text InputFields arrayContains V (field value) and TU (alternative text for accessibility).
    Drop-down ListFields arrayUses PL object: PL.D contains labels, PL.V contains values.
    Link ButtonFields arrayThe URL is stored in FL.form.Id.
    SignatureFields arrayIf signed, contains Sig object with Name, M (ISO 8601 timestamp), Location, Reason, and ContactInfo.
    // Example of a checkbox in Boxsets
    Boxsets: [{
     boxes: [{ x: 47, y: 40, w: 3, h: 1, style: 48, TI: 39, AM: 4, id: { Id: "F8888" }, T: { Name: "box" } }],
     id: { Id: "A446" }
    }]
    
    // Example of a text input in Fields
    {
     style: 48,
     T: { Name: "alpha", TypeInfo: { } },
     id: { Id: "p1_t40", EN: 0 },
     TU: "alternative text",
     TI: 0,
     x: 6.19,
     y: 5.15,
     w: 30.94,
     h: 0.85,
     V: "field value"
    }
  3. Understand the Node.js implementation of pdf.js in pdf2json

    master

    The pdf2json module is a Node.js port of pdf.js. Because pdf.js was originally designed for browser environments with HTML5 support, pdf2json implements several shims and modifications to allow it to run in Node.js:

    • File Loading: Replaces XMLHttpRequest with Node's fs (File System) module to load PDF files.
    • XML Parsing: Replaces the browser's DOMParser with the xmldom module.
    • Threading: Uses a "fake worker" approach where parsing occurs in the same thread as the main process, rather than a background Web Worker.
    • Canvas API: Uses a custom PDFCanvas implementation. Instead of drawing graphics to an HTML5 canvas, PDFCanvas writes 2D context API calls to a JavaScript object in JSON format.
    • DOM: All DOM manipulation code from the original pdf.js is commented out to prevent errors in the non-browser environment.
    • Fonts: Instead of downloading fonts, it parses font information into a CSS font format used in the output JSON's texts array.
  4. Access exact text styles via the TS field

    master

    To avoid misalignment caused by the style dictionary, pdf2json provides an exact text style array in the TS field. If the text style does not match any entry in the style dictionary, the S field is set to -1 and the TS field contains the precise style data.

    The TS array follows this format:

    1. TS[0]: Font Face ID (integer)
    2. TS[1]: Font Size (px)
    3. TS[2]: Font weight (1 if bold, 0 otherwise)
    4. TS[3]: Font style (1 if italic, 0 otherwise)

    If a color is not in the style dictionary, clr is set to -1 and the original hex color is provided in the oc field.

    {
     x: 7.11,
     y: 2.47,
     w: 1.6,
     clr: 0,
     A: "left",
     R: [
      {
       T: "Modesty%20PDF%20Parser%20NodeJS",
       S: -1,
       TS: [0, 15, 1, 0]
      }
     ]
    }
  5. Detect text input field formatter types

    master

    When a widget field type is Tx and the additional-actions dictionary AA is set, the parser detects specific formatting types. These types are populated in the field's T object. Supported types include:

    • Standard Types: number, ssn, date (including custom yyyy), zip, phone, and percent.
    • Arbitrary Masks: If the format is mask, the mask string is provided in the MV field (e.g., 9999 for a 4-digit PIN).

    Special Single-Character Masks

    If an arbitrary mask has only one character, it represents:

    • a: Alphabet only
    • n: Numeric only (no locale formatting)
    • d: Numeric only (with locale formatting, one decimal point allowed)
    • -: Negative number only
    • +: Positive number only
    // Example of a 'number' field
    {
     style: 48,
     T: { Name: "number", TypeInfo: { } },
     id: { Id: "FAGI", EN: 0 },
     TI: 0,
     x: 68.35,
     y: 22.43,
     w: 21.77,
     h: 1.08
    }
  6. Install pdf2json on Ubuntu

    master

    To install pdf2json on Ubuntu, ensure Node.js is installed. If your system uses nodejs instead of node, you may need to create a symbolic link to ensure compatibility with npm and other tools.

    1. Verify your Node.js version.
    2. Create a symbolic link from node to nodejs if necessary.
    3. Verify the node command is working.
    4. Install pdf2json globally using npm.
  7. Access Color and Style dictionaries

    master

    To reduce payload size, pdf2json uses dictionaries for colors and styles. Elements in the output (like Texts or Fills) reference these dictionaries using integer indices. To render the output correctly, you must use the same dictionary definitions.

    Since version 3.1.0, you can access these dictionaries in two ways:

    1. Importing constants directly from the package.
    2. Using static getters on the PDFParser class.

    Note: If you are using a version older than 3.1.0, you may need to import from ./lib/pdfconst.js instead of pdf2json.

    // Option 1: Import constants (since v3.1.0)
    import { kColors, kFontFaces, kFontStyles } from "pdf2json";
    
    // Option 2: Use static getters on PDFParser
    console.dir(PDFParser.colorDict);
    console.dir(PDFParser.fontFaceDict);
    console.dir(PDFParser.fontStyleDict);
  8. Run pdf2json as a command line utility

    master

    You can use pdf2json as a CLI tool to transcode local PDF files or entire directories into JSON format. This is useful for pre-processing static PDFs to improve scalability by serving JSON directly to clients instead of parsing on a web server.

    Using via Node.js

    If you haven't installed the package globally, run it using node:

    node pdf2json.js -f [input directory or pdf file]

    Using as a global command

    To run pdf2json directly without specifying the path to the script, install it globally:

    npm install pdf2json -g

    Then use:

    pdf2json -f [input directory or pdf file]

    Key Behaviors

    • Files: If -f points to a .pdf file, it creates a .json file with the same name in the same directory.
    • Directories: If -f points to a directory, it scans and processes all .pdf files within it (skipping dotfiles).
    • Output Directory: Use -o [output directory] to specify where results should be saved. The directory is created automatically if it doesn't exist.
    node pdf2json.js -f [input directory or pdf file] -o [output directory]
  9. Migrate to pdf2json v3.x (ES Modules)

    master

    Starting from version 3.0.0, pdf2json has converted from CommonJS to ES Modules. In version 3.1.0, the project outputs both ES Module and CommonJS bundles.

    Breaking Change: PDFParser is no longer the default export. You must use a named import when importing the parser into your project.

  10. Migrate to ES Modules (v3.0.0+)

    master

    Starting from v3.0.0, pdf2json was converted from CommonJS to ES Modules. To use the library in your project, you must ensure your environment supports ES Modules. For example, in tsconfig.json, set:

    {
      "compilerOptions": {
        "module": "ESNext"
      }
    }
  11. Install pdf2json

    master

    You can install pdf2json as a local dependency via npm or install it globally to use it as a command-line utility.

    To install locally:

    npm i pdf2json

    To install globally:

    npm i pdf2json -g

    To update to the latest version:

    npm update pdf2json -g