Xberg Documentation

The Benchmark Harness is a Rust-based CLI tool used for comparative benchmarking of document extraction. It evaluates 13 Xberg language bindings and 7 reference frameworks (Docling, MinerU, PyMuPDF4LLM, Unstructured, MarkItDown, LiteParse, and Tika) across performance metrics (latency, throughput, memory) and quality metrics (TF1, SF1) against ground truth data.

It supports two primary workflows:

1. CI benchmarking: Automated cross-framework comparisons triggered via GitHub Actions.
2. Local quality assessment: Developer-facing pipeline comparisons against ground truth for regression detection and quality triage.

Xberg is a document intelligence core that provides extraction capabilities for 98 file formats and 306 programming languages.

Key Features:

• Extraction Types: Text, tables, images, metadata, entities, keywords, code intelligence, and transcripts (where transcription is enabled).
• Format Support: PDF, Office, images, HTML/XML, email, archives, notebooks, citations, scientific formats, plain text, and audio/video formats.
• OCR Options: Supports Tesseract, PaddleOCR, Candle, and VLM OCR through liter-llm, with plugin hooks for custom backends.
• Unified Engine: Uses the same Rust implementation as all other language bindings (Python, Node.js, Go, etc.).

Xberg provides Optical Character Recognition (OCR) to extract text from images and scanned PDFs.

Automatic Behavior:

• Images: Always trigger OCR.
• Scanned PDFs: Trigger OCR per-page.
• Hybrid PDFs: Only OCR pages that lack an existing text layer.

Manual Control:

• Use force_ocr=True to force OCR on all pages regardless of whether a text layer exists.

The xberg-paddle-ocr crate provides high-performance text detection and recognition using PaddlePaddle's models via ONNX Runtime.

Key Capabilities:

• Text Detection: Uses DBNet (Differentiable Binarization).
• Text Recognition: Uses CRNN (Convolutional Recurrent Neural Network).
• Angle Detection: Detects rotated text using PPOCRv2 mobile angle classification models.
• Multi-language Support: Supports multiple languages via various PaddleOCR models.
• Automatic Model Management: Models (including PP-OCRv5 server detection, PP-OCRv5 recognition models for 11 script families, and angle classification) are automatically downloaded and cached on first use.

xberg-ffi provides a stable C ABI for native integration and cross-language interop. It uses the same Rust core as all other language bindings (Python, Node.js, Go, etc.).

Key Capabilities:

• Document Intelligence: Extraction of text, tables, images, metadata, entities, keywords, code intelligence, and transcripts.
• Format Coverage: Supports 98 file formats including PDF, Office, images, HTML/XML, email, archives, notebooks, citations, scientific formats, and audio/video (where transcription is enabled).
• OCR Backends: Supports Tesseract, PaddleOCR, Candle, and VLM OCR via liter-llm.
• Stable ABI: Provides a shared library surface for custom hosts and secondary bindings.

Xberg's core functionality revolves around the extract and extract_batch methods.

• extract: Performs extraction on a single input.
• extract_batch: Performs extraction on multiple inputs.
• ExtractInput: The configuration object used to define what to extract. Setting kind = "uri" allows you to provide a local file path, a file:// URI, or an HTTP(S) URL.
• ExtractionResult: The envelope returned by extraction methods, containing a results list of the extracted data.

Before using these APIs, ensure you have installed the appropriate language binding for your environment.

Xberg can be integrated into your workflow using several different modes depending on your use case:

• Library: Embed extraction directly into your application by importing the package in your preferred language (Python, TypeScript, Rust, Go, etc.).
• CLI: Use for one-off extractions, scripting, or CI pipelines via the xberg command.
• REST API: Use for multi-service architectures by running an API server.
• MCP Server: Integrate with AI agents (like Claude Desktop or Continue.dev) using xberg mcp via stdio transport.
• Docker: Use ghcr.io/xberg-io/xberg:latest for reproducible deployments with all dependencies bundled.

Xberg is built in Rust to provide high-performance extraction through several key architectural optimizations:

• Batch processing: Uses a work-stealing scheduler to achieve 6-10x speedup over sequential extraction.
• Caching: Uses a SQLite-backed cache with automatic invalidation, achieving 85%+ hit rates for repeated files.
• Streaming: Processes large files in 4KB chunks to maintain constant memory usage regardless of file size.
• Lazy initialization: Subsystems like Tokio and plugins are only initialized on first use.
• SIMD acceleration: Whitespace detection and character classification are optimized via SIMD, running 15-37x faster than scalar operations.
• Zero-copy: Uses borrowed references for string slicing to avoid heap allocations.

Xberg provides the following core capabilities:

• Text Extraction: Extract text with position and formatting.
• Metadata Extraction: Retrieve properties like author and creation date.
• Table Extraction: Parse tables while preserving structure.
• Image Extraction: Extract embedded images and render previews.
• OCR Support: Integrate multiple backends for scanned content.
• Code Intelligence: Extract symbols, imports, and docstrings from 306 languages.
• Output Formats: Supports Plain text, Markdown, Djot, HTML, JSON tree structure, or Structured JSON with OCR metadata.