HunyuanOCR-1.5 Documentation

The train_verl/ directory provides a complete reinforcement learning (RL) training suite for HunyuanOCR-1.5. It includes GRPO training scripts, a reward scoring system, data preparation tools, and Ray cluster startup scripts.

It uses a specialized fork of the verl framework (pspdada/verl-HYOCR) which implements FSDP2 actor + vLLM asynchronous rollout. The goal is to provide a minimal runnable scaffold for RL training rather than a full reproduction of technical report experiments.

HunyuanOCR-1.5 is a lightweight, end-to-end Vision Language Model (VLM) specialized for OCR tasks, including document parsing, text spotting, information extraction, and image-to-text translation.

Key upgrades from version 1.0 include:

• DFlash Speculative Decoding: A block-diffusion draft model that predicts multiple candidate tokens to reduce decoding latency in long-structured output scenarios (like tables or formulas) without changing the target model's output distribution.
• PC Deployment: Support for llama.cpp using GGUF weights, enabling deployment on CPUs, consumer GPUs, and laptops.
• Agentic Data Flow: An agent-driven data construction system used to strengthen long-tail capabilities like low-resource OCR and ancient script recognition.
• Enhanced Training: Support for up to 4K image resolution and 128K context windows.

CHAOS-Bench (Character-level Hallucination Assessment for OCR Seeing-is-believing) is a diagnostic benchmark designed to test the "seeing-is-believing" capability of Vision Language Models (VLMs).

It evaluates whether a model truly recognizes text from pixels or relies on linguistic priors to hallucinate "reasonable" words. The benchmark uses academic paper images where specific words have undergone character-level tampering (e.g., changing participant to qarticipant). A model that truly perceives pixels should output the tampered string, whereas a model relying on language priors will incorrectly "correct" it to the original word.

HunyuanOCR is a versatile OCR VLM (Vision Language Model) with the following core capabilities:

• Text Spotting: Outputs text content and line-level coordinates for documents, handwriting, street views, and more.
• Complex Document Processing: Digitizes multilingual documents, maintaining reading order, using LaTeX for formulas, and HTML for complex tables.
• Open-field Information Extraction: Parses specific fields (e.g., names, addresses, amounts) from cards and tickets into JSON.
• Video Subtitle Extraction: Automatically extracts subtitles (including bilingual) from video content.
• Image Text Translation: Performs end-to-end translation of images containing minor languages (e.g., German, Spanish, Japanese, etc.) into Chinese or English.

HunyuanOCR is a specialized OCR VLM that outperforms many general-purpose VLMs and traditional OCR methods across several benchmarks:

• Text Spotting: Achieves 70.92% overall performance, outperforming PaddleOCR, BaiduOCR, and Qwen3VL models.
• Document Parsing (OmniDocBench): Achieves 94.10% overall performance (1B model size), significantly outperforming Gemini-2.5-pro and Qwen3-VL-235B in text, formula, and table extraction.
• Information Extraction & VQA: High performance on Cards (92.29%), Receipts (92.53%), and Video Subtitles (92.87%).
• Translation: The 1B model achieves results comparable to the 235B Qwen3-VL model in photo translation tasks (e.g., Other2En, Other2Zh).

The train_verl/ directory is organized into four functional areas:

• Training & Installation: train_grpo.sh (main launcher) and install_cu13.sh (dependency installer).
• Data Preparation: data/prepare_data.py (JSONL to Parquet conversion).
• Reward System: reward/ contains the scoring logic, including ocr_scorer.py (dispatcher), task_scorers/ (specific rules for spotting, layout, parsing, IE, and chart deplot), and reward_ocr.py (the entry point for verl).
• Utilities: utils/ contains Ray cluster launchers (utils/ray/) and FSDP checkpoint merging scripts (utils/ckpt/).

To prevent visual features from landing at incorrect rotary positions, verl/workers/rollout/utils.py implements a deduplication step.

If a VLM processor (specifically HunYuanVLImageProcessor or Qwen2VLImageProcessor) expands <image> tags into multiple consecutive image_token_ids, the qwen2_5_vl_dedup_image_tokens function collapses them back into a single placeholder. This ensures the vLLM engine receives exactly one placeholder per image, allowing it to handle its own expansion correctly.

When using HunyuanOCR-1.5, the input_ids are expanded by the HF processor (e.g., <img_start><ph><img_end> becomes <img_start><ph>*N<img_end>). To prevent validate_inputs failures during subsequent processor calls, you must collapse these continuous placeholders back into a single instance.

This is implemented via the _collapse_expanded_mm_placeholders(text, processor) utility function in verl/experimental/agent_loop/agent_loop.py. It uses attributes like image_start_token to guard the operation and returns the text unchanged if the processor is not a VLM.

To maximize GPU utilization and remove padding waste, HunyuanOCR training requires packed JSONL files. Instead of one sample per line, each line contains multiple original samples concatenated into a single sequence up to a specified pack_length.

The pipeline consists of two phases:

1. Count phase: Tokenizes each sample in parallel and writes token counts to a temporary directory.
2. Pack phase: Uses a First-Fit Decreasing strategy to greedily pack samples into sequences up to the pack_length limit.

Note: The count phase only tokenizes text; images are not pre-loaded and are handled lazily during training.

To prevent visual feature misalignment in vLLM, you must ensure that only one placeholder exists per image/video. If the HF processor has already expanded <image> into $N$ tokens, vLLM will incorrectly apply position IDs to each individual token.

In verl/workers/rollout/utils.py, the qwen2_5_vl_dedup_image_tokens function is extended to support HunYuanVLImageProcessor. It enables token collapsing for any processor that exposes a non-empty image_token_id attribute.

Because FSDP flattens parameters, standard Hugging Face implementations for HunyuanVL fail or behave incorrectly after wrapping. verl/workers/engine/fsdp/transformer_impl.py applies _patch_vlm_get_image_features(module) to fix the following:

1. StopIteration in feature extraction: Replaces next(self.vit.parameters()).dtype (which fails on empty FSDP iterators) with engine_config.model_dtype.
2. 3D position_ids compatibility: Slices position_ids[:, -1, :] (the 4th plane of xdrope) before passing it to create_causal_mask, which only accepts 2D tensors. The full 3D tensor is still passed to decoder layers for apply_rotary_pos_emb_xdrope.
3. 3D layout fix: Removes the .transpose(0, 1) in FSDPEngineWithLMHead.forward to ensure position_ids maintain the (bs, 4, seq) shape required by apply_rotary_pos_emb_xdrope.