MonkeyOCR Documentation

MonkeyOCR is a document parsing system based on the Structure-Recognition-Relation (SRR) triplet paradigm. It is designed to simplify multi-tool pipelines while maintaining efficiency by avoiding the use of large multimodal models for full-page processing.

Key model variants include:

• MonkeyOCR-pro-1.2B: A leaner, faster version that outperforms the 3B version in accuracy, speed, and efficiency (specifically on Chinese documents).
• MonkeyOCR-pro-3B: A larger model that achieves high performance on English and Chinese documents, outperforming several closed-source and extra-large open-source VLMs on benchmarks like OmniDocBench.

MonkeyOCR has different licensing terms for code and model weights:

• Source Code: Released under the Apache License 2.0.
• Model Weights (v1): Provided for academic research and non-commercial evaluation only.

Commercial Use: Any commercial deployment, SaaS integration, redistribution, or production use requires a separate written commercial license. For inquiries, contact xbai@hust.edu.cn or ylliu@hust.edu.cn.

vLLM is an optional inference backend supporting CUDA 12.6 and 11.8. After installation, you must update the chat_config.backend field in your model_configs.yaml file to vllm.

pip install uv --upgrade
uv pip install vllm==0.9.1 --torch-backend=cu${CUDA_VERSION}
pip install -e .

In model_configs.yaml:

chat_config:
    backend: vllm

The transformers backend supports CUDA 12.4 and 12.1.

1. Create and activate a Python 3.10 environment.
2. Clone the repository and enter the directory.
3. Install MonkeyOCR in editable mode.
4. Install PyTorch matching your CUDA version.
5. Install flash-attn==2.7.4.post1 with --no-build-isolation.
6. Update model_configs.yaml to set chat_config.backend to transformers and optionally adjust batch_size.

conda create -n MonkeyOCR python=3.10
conda activate MonkeyOCR

git clone https://github.com/Yuliang-Liu/MonkeyOCR.git
cd MonkeyOCR

pip install -e .

export CUDA_VERSION=124
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu${CUDA_VERSION}

pip install flash-attn==2.7.4.post1 --no-build-isolation

# model_configs.yaml
chat_config:
  backend: transformers
  batch_size: 10

To run MonkeyOCR inside a container using WSL2 and Docker Desktop:

1. Ensure Docker Desktop is running.
2. Open your WSL terminal:

   wsl
   cd ~

3. Clone the repository and navigate to the directory:

   git clone https://github.com/Yuliang-Liu/MonkeyOCR
   cd MonkeyOCR

4. Follow the 'Docker Deployment' instructions in the main README.md to build and run the image.

Tip: You can use the VS Code Dev Containers extension to connect directly to the container for easier development and modification.

The transformers backend is an optional method supporting CUDA 12.6. It requires PyTorch and Flash Attention 2. After installation, update the chat_config in model_configs.yaml with the transformers backend and an appropriate batch_size.

pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu126
pip install -e .
pip install flash-attn==2.7.4.post1 --no-build-isolation

In model_configs.yaml:

chat_config:
  backend: transformers
  batch_size: 10  # Adjust based on your available GPU memory

To use the PP-DocLayout_plus-L model, you must install PaddlePaddle and PaddleX in addition to the project's base requirements.

1. Install PaddlePaddle

Choose the command based on your NVIDIA driver version to install the GPU-accelerated version. Ensure your PyTorch version is compatible with the installed PaddlePaddle version.

• For GPU driver version ≥450.80.02 (Linux) or ≥452.39 (Windows): Use the cu118 package.
• For GPU driver version ≥550.54.14 (Linux) or ≥550.54.14 (Windows): Use the cu126 package.

2. Install PaddleX

Install the base version of PaddleX using pip.

To run MonkeyOCR directly on Windows, follow the standard installation guide with CUDA support. After environment setup, you must download the model weights and install a specific Triton version for Windows compatibility.

1. Download Model Weights

You can use either Huggingface or ModelScope:

Using Huggingface:

pip install huggingface_hub
python tools/download_model.py

Using ModelScope:

pip install modelscope
python tools/download_model.py -t modelscope

2. Install Triton for Windows

Run the following command to ensure compatibility:

pip install -U "triton-windows<3.4"

To use PP-DocLayoutV2, you must install PaddlePaddle and PaddleX. Ensure your PyTorch version is compatible with the PaddlePaddle version you are installing. It is recommended to use a dedicated Conda environment.

conda create -n MonkeyOCR python=3.10
conda activate MonkeyOCR

git clone https://github.com/Yuliang-Liu/MonkeyOCR.git
cd MonkeyOCR

export CUDA_VERSION=126 # for CUDA 12.6
# export CUDA_VERSION=118 # for CUDA 11.8

pip install paddlepaddle-gpu==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu${CUDA_VERSION}/
pip install paddlex[base]==3.3.0
pip install langchain==0.3.26

Setup

1. Navigate to the docker directory.
2. Ensure nvidia-docker2 is installed. If not, run bash env.sh to set up the environment.

Build and Run

Standard Build:

docker compose build monkeyocr

IMPORTANT: GPU Compatibility Fix If using NVIDIA GPUs from the 20/30/40-series, V100, L20/L40, or similar, you must build the patched image to avoid triton.runtime.errors.OutOfResources: out of resource: shared memory errors:

docker compose build monkeyocr-fix

Running Services:

• Gradio Demo (Port 7860): docker compose up monkeyocr-demo
• FastAPI Service (Port 7861): docker compose up monkeyocr-api
• Interactive Dev Environment: docker compose run --rm monkeyocr-dev