PaddleFormers

PaddleOCR-VL-1.5 is a lightweight Vision-Language Model (VLM) with 0.9B parameters designed for high-performance document parsing. It is optimized for robustness against physical interferences like scanning artifacts, tilting, curling, and uneven lighting.

Key Capabilities:

• Multi-task Support: Includes Seal Recognition and end-to-end Text Spotting (supporting irregular shapes and polygon detection).
• Document Handling: Supports cross-page table merging and cross-page paragraph title recognition.
• Multilingual Support: Handles Tibetan, Bengali, and various rare characters/ancient texts.

Region Recognition (RegionOCR) Task: While the model can parse entire documents, the RegionOCR task focuses on extracting text from specific Regions of Interest (ROI). This is useful for:

• Forms & Invoices: Extracting specific fields like 'Amount' or 'Date'.
• Layout Analysis: Targeting specific cells or paragraphs in dense layouts.
• Seal/Signature Extraction: Specialized extraction of official seals or signatures.

This directory provides best practice tutorials for fine-tuning the PaddleOCR-VL-1.5 model.

Model Profile

PaddleOCR-VL-1.5 is an upgraded version of PaddleOCR-VL with 0.9B parameters. It achieves SOTA performance in document parsing (OmniDocBench v1.5) and features significant improvements in:

• Robustness against interference
• Multi-task expansion (e.g., seal/text localization)
• Long document and multi-language support

Available Fine-tuning Tasks

Depending on your application, you can follow specific guides for:

1. RegionOCR (Region Recognition): For extracting text from specific Regions of Interest (ROI), such as fields in invoices/forms (amounts, serial numbers), or extracting seals and signatures.
2. Complex Table Recognition: For understanding and reconstructing complex table structures, including merged cells, nested headers, borderless tables, and multi-page tables.

PaddleFormers is a Transformers library built on the PaddlePaddle deep learning framework. It provides model interfaces and functionality equivalent to the Hugging Face Transformers project, specifically designed for the PaddlePaddle ecosystem.

Key capabilities include:

• Model Support: Training for both Large Language Models (LLM) and Vision-Language Models (VLM).
• Distributed Training: Full support for mainstream large model distributed training strategies, including Tensor Parallelism, Pipeline Parallelism, and Expert Parallelism.
• Performance Optimization: Built-in support for automatic mixed precision and high-performance training that can outperform Megatron-LM on key models like DeepSeek-V4 and GLM-4.5-Air.
• Post-training: Supports CPT, SFT, SFT-LoRA, DPO, and DPO-LoRA.
• Interoperability: Supports the Safetensors format, ensuring trained models are compatible with Hugging Face-supported tools like FastDeploy, vLLM, and SGLang.

A Processor is a multimodal preprocessing tool used to prepare inputs that combine multiple modalities, such as text, images, video, or audio. It provides a unified interface for transformations like text tokenization and image resizing/normalization.

Key features:

• Unified Interface: Wraps multiple specialized components (e.g., an ImageProcessor and a Tokenizer) into a single class via ProcessorMixin.
• Paddle Tensor Support: Can return processed inputs directly as Paddle Tensors by setting return_tensors="pd".
• Automatic Configuration: Using from_pretrained automatically loads all necessary configuration files (e.g., processor_config.json, tokenizer_config.json) to ensure preprocessing matches the model's training/inference requirements.

Commonly used with AutoProcessor to load the correct processor for a specific model architecture.

from paddleformers.transformers import AutoProcessor

# Load a processor for a specific model
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

# Process inputs and return Paddle Tensors
inputs = processor(text="Hello", images=my_image, return_tensors="pd")

When fine-tuning ultra-large models like DeepSeek-V3 (671B), use the following techniques to manage memory and stability:

• Optimizer Offload: Since optimizer states often exceed GPU memory, use Offload technology to swap states between CPU memory and GPU memory.
• Subbatch Processing: For long sequence inputs (e.g., 32K, 128K), activation peaks can cause OOM. Use the Subbatch method to perform segmented calculations, trading time for memory stability.
• MoE Load Balancing: To prevent OOM caused by expert imbalance in Mixture-of-Experts (MoE) models, ensure proper use of AuxLoss and follow these implementation details:
  • Gate Calculation Isolation: Ensure e_score_correction_bias is used only for gating weight calculation and is not passed to subsequent FFN modules.
  • AuxLoss Calculation: When using Sequence Parallelism (SP) or Subbatching, verify the actual seq_len value used in loss calculations to ensure correctness.
  • Hyperparameter Tuning: Adjust parameters like router_aux_loss_coef (from Hugging Face defaults) to suit your specific training scenario.

PaddleFormers allows you to define custom chat_template and mm_plugin (multimodal data processing plugins) to handle specific model requirements. This is useful when a model's template differs from standard versions.

To use a custom setup, set the following in your configuration:

• template: The name of the template to use.
• custom_register_path: The local file path containing the registration logic.
• template_backend: Set to custom to enable the use of the custom registration path.

Example registration logic in a Python file:

from paddleformers.datasets.template.template import *
from paddleformers.datasets.template.mm_plugin import *

# Define a custom plugin class inheriting from BasePlugin
@dataclass
class PaddleOCRVLV15Plugin(BasePlugin):
    # ... implementation ...

# Register the plugin
register_mm_plugin(
    name = "paddleocr_vl_v15",
    plugin_class = PaddleOCRVLV15Plugin,
)

# Register the template using the plugin
register_template(
    name="paddleocr_vl_v15",
    format_user=StringFormatter(slots=["User: {{content}}\nAssistant:\n"]),
    format_assistant=StringFormatter(slots=["{{content}}"]),
    format_system=StringFormatter(slots=["{{content}}\n"]),
    format_prefix=EmptyFormatter(slots=["<|begin_of_sentence|>"]),
    chat_sep="<|end_of_sentence|>",
    mm_plugin=get_mm_plugin(name="paddleocr_vl_v15", image_token="<|IMAGE_PLACEHOLDER|>"),
)

from paddleformers.datasets.template.template import *
from paddleformers.datasets.template.mm_plugin import *
from paddleformers.datasets.template.augment_utils import *

@dataclass
class PaddleOCRVLV15Plugin(BasePlugin):
    image_bos_token: str = "<|IMAGE_START|>"
    image_eos_token: str = "<|IMAGE_END|>"

    def __init__(self, image_token, video_token, audio_token, **kwargs):
        super().__init__(image_token, video_token, audio_token, **kwargs)
        self.image_augmentation = self.get_ocr_augmentations(
            rotation_p=0.0,
            jpeg_p=0.0,
            scale_p=0.0,
            padding_p=0.0,
            color_jitter_p=0.0,
        )

    # ... (other methods) ...

register_mm_plugin(
    name = "paddleocr_vl_v15",
    plugin_class = PaddleOCRVLV15Plugin,
)

register_template(
    name="paddleocr_vl_v15",
    format_user=StringFormatter(slots=["User: {{content}}\nAssistant:\n"]),
    format_assistant=StringFormatter(slots=["{{content}}"]),
    format_system=StringFormatter(slots=["{{content}}\n"]),
    format_prefix=EmptyFormatter(slots=["<|begin_of_sentence|>"]),
    chat_sep="<|end_of_sentence|>",
    mm_plugin=get_mm_plugin(name="paddleocr_vl_v15", image_token="<|IMAGE_PLACEHOLDER|>"),
)

Function Calling datasets for PaddleFormers must follow a specific format containing messages and tools fields.

Data Schema

• messages: A list of conversation turns. Each turn contains:
  • role: One of system, user, assistant, or tool. Note that system should only appear in the first turn.
  • content: The text content of the message.
  • tool_calls (optional): Used in assistant roles to specify structured tool calls. Each call includes a type (e.g., function) and the function definition (name and arguments).
• tools (optional): A list of available tools. Each tool is defined as an object with a type (e.g., function) and a function object containing the tool's name, description, and parameters.

Example JSONL Entry

{
  "messages": [
    {
      "role": "user",
      "content": "I need to generate an invoice for a customer named John Doe..."
    },
    {
      "role": "assistant",
      "content": "<think>\n</think>",
      "tool_calls": [
        {
          "type": "function",
          "function": {
            "name": "generate_invoice",
            "arguments": {"customer_name": "John Doe", "items": [...]}
          }
        }
      ]
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "generate_invoice",
        "description": "Generate an invoice for a customer",
        "parameters": { ... }
      }
    }
  ]
}

{"messages": [{"role": "user", "content": "I need to generate an invoice for a customer named John Doe. He bought 2 apples for $1 each and 3 oranges for $2 each."}, {"role": "assistant", "content": "<think>\n</think>", "tool_calls": [{"type": "function", "function": {"name": "generate_invoice", "arguments": {"customer_name": "John Doe", "items": [{"name": "apple", "quantity": 2, "price": 1}, {"name": "orange", "quantity": 3, "price": 2}]}}}]}], "tools": [{"type": "function", "function": {"name": "generate_invoice", "description": "Generate an invoice for a customer", "parameters": {"type": "object", "properties": {"customer_name": {"type": "string", "description": "Name of the customer"}, "items": {"type": "array", "items": {"type": "object", "properties": {"name": {"type": "string", "description": "Name of the item"}, "quantity": {"type": "integer", "description": "Quantity of the item"}, "price": {"type": "number", "description": "Price of the item"}}, "required": ["name", "quantity", "price"], "description": "List of items in the invoice"}}, "required": ["customer_name", "items"]}}}]}]}

For table recognition tasks in PaddleOCR-VL, data must be formatted using the OTSL (Optical Table Structure Language) format within the assistant's message content.

OTSL Control Tokens:

• <ecel>: End Cell. Marks the end of the current cell.
• <fcel>: First Cell. Marks the beginning of the first cell in a row.
• <xcel>: eXtended Cell. Marks the beginning of subsequent cells in a row.
• <lcel>: Last Cell. Marks the end of the current row.
• <ucel>: Union Cell. Represents merged cells (spanning multiple rows or columns).
• <nl>: New Line. Represents a line break.

Workflow Recommendation:

1. Use PPOCRLabel for annotation. This produces HTML format tables.
2. Use a script to convert HTML annotations into the OTSL format.
3. To verify model output, you can convert the recognized OTSL back to HTML using the convert_otsl_to_html utility to check if the structure matches expectations.

{
    "messages": [
        {"role": "user", "content": "<image>Table Recognition:"},
        {"role": "assistant", "content": "<fcel>分组<fcel>频数<fcel>频率<nl><fcel>[41,51)<fcel>2<fcel>\\( \\frac{2}{30} \\)<nl>"}
    ],
    "images": ["./assets/table_example.png"]
}