pdf2md

repository·master·Indexed 19 days ago

https://github.com/opengovsg/pdf2md

A JavaScript library and CLI tool for parsing PDF files and converting them into Markdown format using pdf.js. It provides a Node.js API to convert PDF buffers to Markdown strings and a CLI for batch converting directories with support for recursive processing. The library includes a BlockType enumeration for handling headlines, lists, and paragraphs, and a WordType class for formatting links and footnotes.

Tokens
2.4K
Snippets
10
Records
13
Agent score
67%

What's inside pdf2md

  1. Use pdf2md as a JavaScript library

    master

    You can integrate pdf2md into your Node.js application to convert PDF buffers into Markdown text. The library accepts a PDF buffer and returns a Promise that resolves with the converted Markdown string.

    Note: The example usage requires a callbacks argument in the function signature, though the implementation details of these callbacks are not specified in the README.

    const path = require('path');
    const fs = require('fs')
    const pdf2md = require('@opendocsg/pdf2md')
    
    const pdfBuffer = fs.readFileSync(filePath)
    pdf2md(pdfBuffer, callbacks)
      .then(text => {
        let outputFile = allOutputPaths[i] + '.md'
        console.log(`Writing to ${outputFile}...`)
        fs.writeFileSync(path.resolve(outputFile), text)
        console.log('Done.')
      })
      .catch(err => {
        console.error(err)
      })
  2. Understand the structure of a ParseResult

    master

    When using the pdf2md library, a transformation or PDF parse operation returns a ParseResult object. This object contains the processed content and metadata used for subsequent transformations or debugging.

    Key properties of a ParseResult instance:

    • pages: An array of Page objects representing the parsed content of the PDF.
    • globals: An object containing properties accessible to all subsequent transformations (primarily used in debug mode).
    • messages: An array of messages intended for display during transformations (primarily used in debug mode).
  3. Understand WordType markdown element enumerations

    master

    The WordType class defines how different types of markdown elements (like links and footnotes) are formatted into text. It is an enumeration used by the library to determine the string representation of specific word elements during the PDF to Markdown conversion process.

    Available types include:

    • LINK: Formats a string as a standard markdown link: [text](url).
    • FOOTNOTE_LINK: Formats a string as a footnote reference: ^text.
    • FOOTNOTE: Formats a string as a footnote content: (^text).
    // Note: This is a conceptual representation of the WordType enum values
    // used internally by the library to format markdown output.
    
    WordType.LINK // returns '[text](text)'
    WordType.FOOTNOTE_LINK // returns '^text'
    WordType.FOOTNOTE // returns '(^text)'
  4. Fix 'JavaScript heap out of memory' error in CLI

    master

    When performing recursive conversions on a large number of files, you may encounter the error Allocation failed - JavaScript heap out of memory. To resolve this, bypass npx and run the CLI script directly using node with the --max-old-space-size flag to increase the available memory.

    $ node lib/pdf2md-cli.js --max-old-space-size=4096 --inputFolderPath=[your input folder path] --outputFolderPath=[your output folder path] --recursive
  5. Reference the pdf2md CLI options

    master

    When using the CLI tool, you can provide the following options:

    • --inputFolderPath: The path to the existing folder containing the PDF files to be converted.
    • --outputFolderPath: The path to the existing folder where Markdown files will be saved.
    • --recursive: If specified, the tool will convert all PDFs found in subfolders within the input folder. Omit this flag to only process the top-level directory.
  6. Convert PDFs using the CLI tool

    master

    The pdf2md CLI tool allows you to batch convert PDF files from an input directory to an output directory. You can run it via npx or directly via node for advanced memory management.

    $ npx @opendocsg/pdf2md --inputFolderPath=[your input folder path] --outputFolderPath=[your output folder path] --recursive
  7. Convert a PDF buffer to Markdown using the default export

    master

    The main entrypoint of the library is an asynchronous function that accepts a PDF source and an optional set of callbacks. It returns a Promise that resolves to the converted Markdown text. Page breaks in the resulting Markdown are marked with the <!-- PAGE_BREAK --> delimiter.

    Parameters

    • pdfBuffer: A string, TypedArray, DocumentInitParameters, or PDFDataRangeTransport. This is passed directly to pdfjs.getDocument().
    • callbacks (optional): An object containing lifecycle hooks for the parsing process:
      • metadataParsed: Invoked when document metadata is parsed.
      • pageParsed: Invoked when a page is parsed.
      • fontParsed: Invoked when a font is parsed.
      • documentParsed: Invoked when the document parsing is complete.
    const pdf2md = require('pdf2md');
    
    async function convert() {
      const pdfBuffer = /* your PDF buffer or path */;
      const markdown = await pdf2md(pdfBuffer, {
        pageParsed: (page) => console.log('Parsed page:', page),
        documentParsed: () => console.log('Conversion complete!')
      });
      console.log(markdown);
    }
    
    convert();
  8. Check if a type is a headline with isHeadline()

    master

    The isHeadline(type) function returns true if the provided BlockType is a headline (i.e., its name starts with 'H' and is two characters long, such as H1, H2, etc.).

    const { isHeadline } = require('./lib/models/markdown/BlockType');
    
    const result = isHeadline(BlockType.H1); // true
    const result2 = isHeadline(BlockType.PARAGRAPH); // false
  9. Convert a block to Markdown text with blockToText()

    master

    The blockToText function converts a LineItemBlock into its Markdown string representation based on its assigned BlockType. If the block has no type assigned, it defaults to rendering the items as plain text (without inline formatting).

    const { blockToText } = require('./lib/models/markdown/BlockType');
    
    // Assuming 'block' is a valid LineItemBlock object
    const markdownString = blockToText(block);
  10. Get a headline type by level with headlineByLevel()

    master

    The headlineByLevel(level) function returns the corresponding BlockType for a given heading level (1 through 6).

    • If level is between 1 and 6, it returns the matching H1-H6 member.
    • If level is outside this range, it logs a warning and defaults to BlockType.H6.
    const { headlineByLevel } = require('./lib/models/markdown/BlockType');
    
    const h1 = headlineByLevel(1); // Returns BlockType.H1
    const h9 = headlineByLevel(9); // Returns BlockType.H6 and logs a warning
  11. Reference the BlockType enumeration

    master

    The BlockType enum defines the different types of Markdown blocks that the library can generate from PDF content. Each type includes metadata (like headlineLevel) and a toText method for rendering the block as a Markdown string.

    Available BlockType values:

    KeyProperties / Behavior
    H1 to H6Headlines with corresponding headlineLevel (1-6).
    TOCTable of Contents; uses mergeToBlock: true.
    FOOTNOTESFootnotes; uses mergeToBlock: true and mergeFollowingNonTypedItems: true.
    CODECode blocks; wraps content in triple backticks (```).
    LISTLists; uses mergeToBlock: false and mergeFollowingNonTypedItemsWithSmallDistance: true.
    PARAGRAPHStandard text paragraphs.
    const BlockType = require('./lib/models/markdown/BlockType');
    
    // Accessing enum members
    console.log(BlockType.H1);
    console.log(BlockType.CODE);