DOCSEARCHbyAlgolia
Submit Docs

dots.ocr (dots.mocr)

repository·master·Indexed 27 days ago

https://github.com/rednote-hilab/dots.ocr

A high-performance multilingual OCR and vision-language model (VLM) based on a 1.7B LLM. It provides unified layout detection and content recognition for document parsing, scene text recognition, and web screen parsing. Specialized variants include dots.mocr-svg for converting charts, diagrams, and chemical formulas directly into SVG code. Supports deployment via vLLM (0.11.0+) and the Hugging Face transformers library.

Tokens
5.4K
Snippets
10
Records
31
Agent score
85%

What's inside dots.ocr

  1. Overview of dots.ocr capabilities

    master

    dots.ocr is a multilingual document parser that uses a single 1.7B-parameter vision-language model (VLM) to unify layout detection and content recognition.

    Key capabilities include:

    • Unified Architecture: Performs layout detection, content recognition, and reading order preservation by simply altering the input prompt, eliminating the need for complex multi-model pipelines.
    • High Performance: Achieves SOTA performance on OmniDocBench for text, tables, and reading order. Formula recognition is comparable to much larger models like Doubao-1.5 and gemini2.5-pro.
    • Multilingual Support: Robust parsing for low-resource languages.
    • Efficiency: The compact 1.7B LLM foundation allows for faster inference speeds compared to larger models.

  2. Overview of dots.ocr

    master

    dots.ocr is a universal OCR and vision model designed to recognize virtually any human script. It is capable of:

    • Multilingual Document Parsing: Achieving state-of-the-art performance for models of comparable size.
    • Structured Graphics Parsing: Converting charts and diagrams directly into SVG code.
    • General Vision Tasks: Parsing web screens and spotting scene text.

  3. Understand dots.ocr limitations

    master

    Be aware of the following current limitations when using dots.ocr:

    • Complex Elements: Extraction of high-complexity tables and formulas is not yet perfect.
    • Images: Pictures contained within documents are currently not parsed.
    • Throughput: The model is not yet optimized for high-throughput processing of large volumes of PDFs.
    • Special Characters: High density of characters like ... or _ can cause output loops.

  4. Structured Graphics Parsing with dots.mocr-svg

    master
    The dots.mocr-svg model is specialized for parsing visual languages (such as charts, graphics, chemical formulas, and logos) directly into SVG code. This allows for high-fidelity interpretation of dense human knowledge encapsulated in visual formats.

  5. Download Model Weights

    master

    Use the provided utility script to download the necessary model weights. You can download via the default method or specifically via ModelScope.

    Warning: Ensure your model save path does not contain periods (e.g., use DotsMOCR).

  6. Install dots.mocr via Conda and Pip

    master

    To set up a local environment for dots.mocr, create a Conda environment with Python 3.12, clone the repository, and install the package in editable mode.

    Note: When saving model weights, use a directory name without periods (e.g., DotsMOCR instead of dots.mocr) to avoid issues with current Transformers integration.

    conda create -n dots_mocr python=3.12
conda activate dots_mocr

git clone https://github.com/rednote-hilab/dots.mocr.git
cd dots.mocr

# Install the package
pip install -e .

  7. Deploy dots.mocr using vLLM

    master
    Dots OCR is officially integrated into vLLM (version 0.11.0+). It is highly recommended to use vLLM for deployment. You can launch a model server for dots.mocr or the SVG-specialized dots.mocr-svg using the vllm serve command.

  8. Evaluate with OmniDocBench

    master

    To evaluate the performance of DotsOCR, use the OmniDocBench framework.

    1. Clone and set up the OmniDocBench environment.
    2. Configure end2end.yaml to point to your ground truth data and your model's prediction output directory.
    3. Run pdf_validation.py with your config file.

    Configuration Note: In end2end.yaml, ensure dataset.prediction.data_path is set to the directory containing your model's output files.

    git clone https://github.com/opendatalab/OmniDocBench.git
cd OmniDocBench

# Setup environment
conda create -n omnidocbench python=3.10
conda activate omnidocbench
pip install -r requirements.txt

# Run evaluation
python pdf_validation.py --config ./end2end.yaml > evaluation_output.log

  9. Register and Launch the DotsOCR vLLM Server

    master

    To use the model via a vLLM server, follow these steps to download weights, register the model with vLLM, and launch the service.

    Important: When saving model weights, use a directory name without periods (e.g., DotsOCR instead of dots.ocr) to avoid integration issues with Transformers.

    1. Download the model weights using tools/download_model.py.
    2. Set hf_model_path to your weights directory.
    3. Add the model to the vllm entrypoint using sed.
    4. Launch the server using vllm serve.
    # model setup and register
python3 tools/download_model.py
export hf_model_path=./weights/DotsOCR
export PYTHONPATH=$(dirname "$hf_model_path"):$PYTHONPATH
sed -i '/^from vllm\.entrypoints\.cli\.main import main$/a\nfrom DotsOCR import modeling_dots_ocr_vllm' `which vllm` 

# launch vllm server
CUDA_VISIBLE_DEVICES=0 vllm serve ${hf_model_path} --tensor-parallel-size 1 --gpu-memory-utilization 0.95  --chat-template-content-format string --served-model-name model --trust-remote-code

  11. Set up the Model Environment

    master

    To set up the environment for reproducing OmniDocBench results, use the provided Docker image: rednotehilab/dots.ocr.

    Alternatively, you can set up the environment manually by cloning the repository and installing the package in editable mode.

    git clone https://github.com/rednote-hilab/dots.ocr.git
cd dots.ocr
pip install -e .

  12. Configure end2end.yaml for OmniDocBench Evaluation

    master

    The end2end.yaml file defines the metrics and dataset paths for the OmniDocBench evaluation.

    Supported metrics include:

    • text_block: Edit_dist
    • display_formula: Edit_dist, CDM
    • table: TEDS, Edit_dist
    • reading_order: Edit_dist
    end2end_eval:
  metrics:
    text_block:
      metric:
        - Edit_dist
    display_formula:
      metric:
        - Edit_dist
        - CDM
    table:
      metric:
        - TEDS
        - Edit_dist
    reading_order:
      metric:
        - Edit_dist
  dataset:
    dataset_name: end2end_dataset
    ground_truth:
      data_path: ./OmniDocBench.json  # change to omnidocbench official gt
    prediction:
      data_path: /path/to/dots.ocr/output_omni/  # change to your own output dir
    match_method: quick_match