repository·main·Indexed 41 days ago
https://github.com/bzsanti/oxidizepdfA comprehensive, pure-Rust PDF toolkit for standard workflows (generation, parsing, manipulation) and advanced AI/RAG applications. It features structure-aware, metadata-rich chunking via the `rag_chunks()` API, providing heading context, page numbers, and element types. The project includes official bindings for Python (v0.5.x), .NET (OxidizePdf.NET v0.9.x), LlamaIndex (v0.1.x), and an MCP server.
The oxidize-pdf test corpus is organized into 7 tiers (T0–T6) designed for different testing objectives ranging from regression prevention to adversarial safety.
| Tier | PDFs | Trigger | Purpose |
|---|---|---|---|
| T0 | 749 | Every commit | Regression prevention |
| T1 | ~2,000 | Every commit | Spec conformance |
| T2 | 2,000 | Nightly | Generator diversity |
| T3 | 750 | Nightly | Error recovery |
| T4 | 500 | Weekly | AI/RAG accuracy |
| T5 | ~900 | Weekly | Quality benchmarking |
| T6 | 200 | Weekly | Adversarial safety |
This plan outlines a major refactor of the font subsetting system to fix CJK character rendering bugs and significantly reduce font subset output size (by 3-14x).
Key Architectural Changes:
cff_subsetter.rs into a modular cff/ directory structure.Technical Specifications:
OxidizePDF is distributed in three editions, separating open-source core functionality from commercial enhancements.
Includes the core PDF engine and basic manipulation tools:
Includes all Community features plus:
Includes all PRO features plus:
To support Type0 text extraction for non-identity CID encoding, you must implement the EncodingCMap structure. This structure maps character codes (which can be variable-width) to CIDs. It is distinct from a standard CMap (ToUnicode), which maps to Unicode hex strings.
Key components of EncodingCMap include:
codespace_ranges: Defines the byte widths for character codes.single_cid: A map for exact cidchar matches.cid_ranges: A list of ranges for cidrange matches.usecmap_parent: The name of the parent CMap if specified via the usecmap keyword.#[derive(Debug, Clone, Default)]
pub(crate) struct EncodingCMap {
pub codespace_ranges: Vec<CodeRange>,
pub single_cid: std::collections::HashMap<Vec<u8>, u16>,
pub cid_ranges: Vec<CidRange>,
pub notdef_ranges: Vec<CidRange>,
pub ordering: Option<String>,
pub usecmap_parent: Option<String>,
pub wmode: u8,
}Rich table structures (merged cells and multi-level headers) are only produced when a hard signal is present in the PDF:
› separator.The implementation of the row_id Y-up-jump heuristic (Issue #265) for resolving line interleaving can cause significant changes in the number of chunks produced during PDF extraction. Developers should not rely on a strict ±5% per-document chunk-count guardrail, as legitimate structural improvements can cause larger deltas:
Instead of monitoring chunk count variance, evaluate extraction quality based on the absence of 'alphabet soup' (garbled substrings) and the presence of coherent prose.
Starting with version 2.8.0, oxidize-pdf supports per-document custom font metrics. This replaces the previous process-wide CUSTOM_FONT_METRICS registry to prevent memory leaks, race conditions, and name leaks in long-running services or concurrent environments.
Document now owns a FontMetricsStore (Arc-backed, Clone + Send + Sync).FontMetricsStore attached to the Document.#[deprecated(since = "2.8.0")]).text::metrics::get_font_metrics function no longer auto-registers default metrics into the global registry on a read miss; it is now a pure read operation.When implementing or refactoring CFF (Compact Font Format) subsetting, use byte-level desubroutinizer tests to verify correctness. While outline verification (comparing glyph paths) is a more advanced method, desubroutinizer tests are sufficient for catching byte-level errors in charstring processing.
Additionally, use debug_assert! for structural validation after CFF subset generation. These assertions should perform lightweight checks, such as parsing the output header and verifying INDEX counts, to ensure the generated subset is structurally sound.
To prevent memory leaks in long-running services (resolving issue #230), oxidize-pdf is moving from a process-wide global registry (CUSTOM_FONT_METRICS) to a hierarchical, document-scoped model.
Key Concepts:
FontMetricsStore: An Arc-backed, Clone + Send + Sync type that lives on a specific Document. It bounds custom-font metrics to the lifetime of that document.FontMetricsStore attached to the Document (or Page). If the metrics are not found in the document scope, it falls back to the legacy global registry.Document::add_font_from_bytes. Page objects carry an Option<FontMetricsStore>, which is injected during page creation via Document::new_page_*() or Document::add_page.To extract text from Type0/CID fonts correctly, oxidize-pdf follows a multi-step decoding pipeline. If a PDF uses non-Identity encoding, the extractor must first map the content-stream bytes to a CID (Character Identifier) before mapping that CID to a Unicode character.
The pipeline follows this flow:
content-stream bytes $\rightarrow$ [Encoding CMap: code $\rightarrow$ CID] $\rightarrow$ [CID $\rightarrow$ Unicode] $\rightarrow$ text
If the code $\rightarrow$ CID step is skipped (which happens if the extractor incorrectly assumes code == CID), the resulting text will appear as 'glyph-code garbage'.
Tests in oxidize-pdf are designed to detect if their required corpus is present. If a corpus is missing, the corresponding tests will gracefully skip with an informational message rather than failing.
This behavior allows:
The oxidize-pdf library (available in Rust, Python, and .NET) is designed to ingest complex, real-world PDFs and partition them into structured, RAG-ready (Retrieval-Augmented Generation) chunks.
Instead of simple text extraction, the library uses a rag_chunks() method (utilizing a HybridChunkConfig by default) to produce chunks that include hierarchical context, element types (e.g., paragraph, heading), and token estimates. This makes the output immediately compatible with vector stores and LLM orchestration frameworks like LangChain or LlamaIndex.
chunks = doc.rag_chunks() // default HybridChunkConfig