DOCSEARCHbyAlgolia
Submit Docs

pdf-inspector

repository·main·Indexed 23 days ago

https://github.com/firecrawl/pdf-inspector

A high-performance Rust library for fast PDF classification and text extraction. It detects if PDFs are text-based or scanned and converts text-based PDFs into structured Markdown without OCR. Available as a Rust library, Python package, Node.js NAPI bindings (@firecrawl/pdf-inspector), and a WebAssembly version for browsers (@firecrawl/pdf-inspector-wasm). Includes CLI tools for PDF-to-Markdown conversion and type detection.

Tokens
24.3K
Snippets
61
Records
122
Agent score
81%

What's inside pdf-inspector

  1. How PDF classification works in pdf-inspector

    main

    Classification works by parsing the xref table and page tree without loading the full document. It samples content streams looking for text operators (Tj/TJ) and image operators (Do).

    This allows for extremely fast detection (~10-50ms) and provides a pages_needing_ocr list, which allows callers to route only specific pages to an OCR service instead of the whole document.

    Scan Strategies

    StrategyBehaviorBest for
    EarlyExit (default)Scan all pages, stop on first non-text pagePipelines routing TextBased PDFs to fast extraction
    FullScan all pages, no early exitAccurate Mixed vs Scanned classification
    Sample(n)Sample n evenly distributed pagesVery large PDFs where speed matters more than precision
    Pages(vec)Only scan specific 1-indexed page numbersWhen the caller knows which pages to check

  2. Benchmark pdf-inspector against OpenDataLoader

    main

    You can run a paired harness to compare a candidate pdf2md binary against a baseline binary using the OpenDataLoader corpus. This process evaluates both outputs and reports aggregate and per-document deltas to ensure consistency.

    To run the benchmark, build your candidate in release mode and use the scripts/bench_opendataloader.py script.

    Key Flags:

    • --bench-dir: Path to the OpenDataLoader benchmark directory.
    • --baseline: Path to the released or worktree build of the baseline pdf2md binary.
    • --candidate: Path to your current pdf2md build.
    • --max-document-regression: Maximum allowed regression per document (e.g., 0.02).
    • --json-output: Path where the benchmark results will be saved.
    • --reference-evaluation: Path to an existing evaluation.json to report deltas against a specific evaluation.
    • --require-reference-lead: If set, a negative reference delta will cause the run to fail.
    • --min-overall-delta: Sets a required aggregate gain for the candidate.

    By default, the candidate is not allowed to regress the baseline overall score or introduce missing predictions.

    cargo build --release
python3 scripts/bench_opendataloader.py \
  --bench-dir ../opendataloader-bench \
  --baseline ../pdf-inspector-main/target/release/pdf2md \
  --candidate target/release/pdf2md \
  --max-document-regression 0.02 \
  --json-output /tmp/pdf-inspector-benchmark.json

  5. Build @firecrawl/pdf-inspector-wasm from source

    main

    To build the WASM package from source, use wasm-pack. Ensure you have the correct version of wasm-pack installed.

    cargo install wasm-pack --version 0.15.0 --locked
wasm-pack build wasm --target web --scope firecrawl --release

  6. Use @firecrawl/pdf-inspector-wasm for PDF processing

    main

    To use the library, first call init() to initialize the WASM module. You can then process a PDF by passing a Uint8Array to the processing functions.

    Note: Extraction is synchronous after initialization. For large documents, it is recommended to run these calls within a Web Worker to prevent blocking the main UI thread.

    import init, { processPdf } from "@firecrawl/pdf-inspector-wasm";

await init();

const response = await fetch("/annual-report.pdf");
const pdf = new Uint8Array(await response.arrayBuffer());
const result = processPdf(pdf);

console.log(result.pdfType);
console.log(result.markdown);

  7. Run the backend evidence probe

    main

    The backend evidence probe is used to compare positioned pdf2md items with MuPDF structured text on the same pages. It helps identify deterministic extraction or layout evidence that could justify native implementations. It does not merge MuPDF output into Markdown or add runtime dependencies.

    Prerequisites:

    • Install MuPDF's mutool.
    • Build the pdf2md binary.

    Usage: Run scripts/probe_backend_evidence.py followed by the target PDF and necessary flags.

    Key Flags:

    • --pdf2md: Path to the pdf2md binary.
    • --json-output: Path to save the evidence report.
    • --min-token-gain: Configurable threshold for token gain.
    • --min-alternate-only-ratio: Configurable threshold for alternate-only ratio.
    • --min-anchor-gain: Configurable threshold for anchor gain.

    The resulting JSON report flags pages where MuPDF shows material net token gain, repeated alignment anchors absent from local evidence, or additional image blocks.

    python3 scripts/probe_backend_evidence.py document.pdf \
  --pdf2md target/release/pdf2md \
  --json-output /tmp/backend-evidence.json

  10. Install pdf-inspector in Rust

    main

    To use pdf-inspector in your Rust project, add it via Cargo:

    cargo add pdf-inspector

    If you want to use the latest unreleased changes directly from the repository, use the git dependency in your Cargo.toml:

    [dependencies]
pdf-inspector = { git = "https://github.com/firecrawl/pdf-inspector" }

    You can also install the included CLI binaries (pdf2md and detect-pdf) using:

    cargo install pdf-inspector
    cargo add pdf-inspector

  11. Publish the Browser WebAssembly package to npm

    main

    The browser package is published as @firecrawl/pdf-inspector-wasm. The version is managed in wasm/Cargo.toml.

    For the initial release (bootstrap), you must manually publish the package before configuring GitHub Actions trusted publishing. Subsequent releases are automated via .github/workflows/publish-wasm.yml when the version in wasm/Cargo.toml is bumped and merged to main.

    Initial Manual Release Steps:

    1. Build the web target using wasm-pack: wasm-pack build wasm --target web --scope firecrawl --out-dir pkg --release
    2. Verify the package contents: npm pack --dry-run ./wasm/pkg
    3. Publish to npm: npm publish ./wasm/pkg --access public
    4. Configure the GitHub Actions trusted publisher in the npm package settings using:
      • Organization: firecrawl
      • Repository: pdf-inspector
      • Workflow: publish-wasm.yml
      • Allowed action: npm publish
    wasm-pack build wasm --target web --scope firecrawl --out-dir pkg --release
npm pack --dry-run ./wasm/pkg
npm publish ./wasm/pkg --access public