DeepSeek-OCR

repository·main·Indexed 12 days ago

https://github.com/deepseek-ai/deepseek-ocr

A model specializing in high-fidelity optical character recognition and document parsing, such as converting documents to markdown. It supports multiple resolution modes (Tiny, Small, Base, Large, and Dynamic) and can be run via vLLM or Transformers APIs. The model includes specific prompting patterns for document-to-markdown conversion, figure parsing, and image description.

Tokens: 2K
Snippets: 5
Records: 6
Agent score: 48%

What's inside DeepSeek-OCR

Install DeepSeek-OCR

main

To install DeepSeek-OCR, ensure your environment uses cuda11.8 and torch2.6.0.

Clone the repository:

git clone https://github.com/deepseek-ai/DeepSeek-OCR.git

Create and activate a Conda environment:

conda create -n deepseek-ocr python=3.12.9 -y
conda activate deepseek-ocr

Install dependencies. Note that you may need to download the vllm-0.8.5 wheel manually as specified in the installation steps.

pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu118
pip install vllm-0.8.5+cu118-cp38-abi3-manylinux1_x86_64.whl
pip install -r requirements.txt
pip install flash-attn==2.7.3 --no-build-isolation

git clone https://github.com/deepseek-ai/DeepSeek-OCR.git
conda create -n deepseek-ocr python=3.12.9 -y
conda activate deepseek-ocr
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu118
pip install vllm-0.8.5+cu118-cp38-abi3-manylinux1_x86_64.whl
pip install -r requirements.txt
pip install flash-attn==2.7.3 --no-build-isolation

Run DeepSeek-OCR via vLLM Inference
main
You can run inference using the provided scripts in the DeepSeek-OCR-vllm directory.
Important: Before running, update INPUT_PATH and OUTPUT_PATH in DeepSeek-OCR-master/DeepSeek-OCR-vllm/config.py.
- Image (streaming output): python run_dpsk_ocr_image.py
- PDF (high concurrency): python run_dpsk_ocr_pdf.py
- Batch evaluation: python run_dpsk_ocr_eval_batch.py
If you are using an upstream version of vLLM (v0.11.1 or later), install it via nightly builds:
```
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
```
```
cd DeepSeek-OCR-master/DeepSeek-OCR-vllm
python run_dpsk_ocr_image.py
```
https://github.com/deepseek-ai/deepseek-ocr
DeepSeek-OCR Prompting Guide
main
Use specific prompt patterns to trigger different OCR behaviors:
- Document to Markdown: <image>\n<|grounding|>Convert the document to markdown.
- General OCR: <image>\n<|grounding|>OCR this image.
- No Layouts: <image>\nFree OCR.
- Figure Parsing: <image>\nParse the figure.
- General Description: <image>\nDescribe this image in detail.
- Reference/Location: <image>\nLocate <|ref|>xxxx<|/ref|> in the image.
```
# document
"<image>\n<|grounding|>Convert the document to markdown."

# other image
"<image>\n<|grounding|>OCR this image."

# without layouts
"<image>\nFree OCR."

# figures in document
"<image>\nParse the figure."

# general
"<image>\nDescribe this image in detail."

# rec
"<image>\nLocate <|ref|>xxxx<|/ref|> in the image."
```
https://github.com/deepseek-ai/deepseek-ocr

Use vLLM API for DeepSeek-OCR Inference

main

To use the model programmatically with vllm, initialize the LLM instance with NGramPerReqLogitsProcessor and pass your images as multi_modal_data within a list of dictionaries.

Use SamplingParams to configure generation, including extra_args for the ngram logit processor which controls ngram_size, window_size, and whitelist_token_ids.

from vllm import LLM, SamplingParams
from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor
from PIL import Image

# Create model instance
llm = LLM(
    model="deepseek-ai/DeepSeek-OCR",
    enable_prefix_caching=False,
    mm_processor_cache_gb=0,
    logits_processors=[NGramPerReqLogitsProcessor]
)

# Prepare batched input
image_1 = Image.open("path/to/your/image_1.png").convert("RGB")
image_2 = Image.open("path/to/your/image_2.png").convert("RGB")
prompt = "<image>\nFree OCR."

model_input = [
    {
        "prompt": prompt,
        "multi_modal_data": {"image": image_1}
    },
    {
        "prompt": prompt,
        "multi_modal_data": {"image": image_2}
    }
]

sampling_param = SamplingParams(
            temperature=0.0,
            max_tokens=8192,
            extra_args=dict(
                ngram_size=30,
                window_size=90,
                whitelist_token_ids={128821, 128822},  # e.g., <td>, </td>
            ),
            skip_special_tokens=False,
        )

model_outputs = llm.generate(model_input, sampling_param)

for output in model_outputs:
    print(output.outputs[0].text)

Use Transformers API for DeepSeek-OCR Inference

main

For standard Transformers-based inference, load the model using AutoModel and AutoTokenizer with trust_remote_code=True. It is recommended to use _attn_implementation='flash_attention_2' and torch.bfloat16 for efficiency.

Call model.infer() with the required parameters such as prompt, image_file, output_path, and resolution settings (base_size, image_size, crop_mode).

from transformers import AutoModel, AutoTokenizer
import torch
import os

os.environ["CUDA_VISIBLE_DEVICES"] = '0'
model_name = 'deepseek-ai/DeepSeek-OCR'

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, _attn_implementation='flash_attention_2', trust_remote_code=True, use_safetensors=True)
model = model.eval().cuda().to(torch.bfloat16)

prompt = "<image>\n<|grounding|>Convert the document to markdown. "
image_file = 'your_image.jpg'
output_path = 'your/output/dir'

res = model.infer(
    tokenizer, 
    prompt=prompt, 
    image_file=image_file, 
    output_path=output_path, 
    base_size=1024, 
    image_size=640, 
    crop_mode=True, 
    save_results=True, 
    test_compress=True
)

DeepSeek-OCR Supported Resolution Modes
main
The model supports several native resolution modes and a dynamic resolution mode:
Native Resolutions:
- Tiny: 512×512 (64 vision tokens)
- Small: 640×640 (100 vision tokens)
- Base: 1024×1024 (256 vision tokens)
- Large: 1280×1280 (400 vision tokens)
Dynamic Resolution:
- Gundam: $n imes 640 imes 640 + 1 imes 1024 imes 1024$
https://github.com/deepseek-ai/deepseek-ocr