Character Detection Matching (CDM) is an evaluation metric designed for formula recognition. Unlike traditional text-based metrics like BLEU or Edit Distance, which are sensitive to LaTeX representation variations and training data distribution, CDM uses a vision-based approach.

It works by:

1. Rendering both the predicted LaTeX and the ground truth (GT) LaTeX into images.
2. Using visual feature extraction and localization to perform precise character-level matching.
3. Incorporating spatial position information to ensure objective and accurate evaluation.

This method avoids the pitfalls where a model might get a high BLEU score despite incorrect recognition, or a low score despite correct recognition due to different LaTeX formatting.

The UniMER dataset consists of two main parts:

• UniMER-1M: A training set with over 1 million LaTeX-image pairs.
• UniMER-Test: A benchmark test set with 23,757 samples categorized into SPE (Simple Printed), CPE (Complex Printed), SCE (Screen Capture), and HWE (Handwritten).

Datasets can be downloaded from OpenDataLab (recommended for users in China) or Hugging Face. After downloading, extract them into the following directory structure:

• ./data/UniMER-1M
• ./data/UniMER-Test

To evaluate the model performance:

1. Configure Test Paths: Open the files in the configs/val folder and set the paths to your test datasets.
2. Execute Testing: Run the testing script using the provided shell script.

Note: The test.py script performs the evaluation. Ensure you have updated the test set paths in test.py to match your actual data locations.

# After configuring paths in configs/val
bash script/test.sh

UniMERNet can be executed in three different ways depending on your needs:

1. Streamlit GUI: Provides an interactive, user-friendly interface for real-time formula recognition and rendering. Run via the unimernet_gui command.
2. Command Line Demo: Predicts LaTeX code from images using a script. Run via python demo.py.
3. Jupyter Notebook: Recognizes and renders formulas within a notebook environment. Run via jupyter-lab ./demo.ipynb.

# 1. Streamlit GUI
unimernet_gui

# 2. Command line demo
python demo.py

# 3. Jupyter Notebook demo
jupyter-lab ./demo.ipynb

To use UniMERNet, you must download the model weights and configuration files. The files can be obtained from the Hugging Face repository. Ensure you have the following files in your local model directory:

• README.md
• config.json
• preprocessor_config.json
• pytorch_model.bin
• tokenizer.json
• tokenizer_config.json

https://huggingface.co/wanderkid/unimernet/tree/main

If local environment setup is difficult, you can use Docker to deploy CDM.

1. Build the image: Use the provided Dockerfile.
2. Run the container: Start an interactive bash session. Use the -v flag to map host directories to the container so that evaluation results are persisted on your host machine.

# Build the docker image
docker build -f DockerFile -t cdm:latest .

# Start a container
docker run -it cdm bash

# Example with volume mapping to save results
docker run -it -v /path/on/host:/path/in/container cdm bash

The UniMER dataset consists of the UniMER-1M training set and the UniMER-Test set. You can download them from OpenDataLab or HuggingFace.

After downloading, extract them to the following structure:

./data/UniMER-1M
./data/UniMER-Test

You can install UniMERNet using pip for general use or via editable install for development. It is recommended to use a clean Conda environment with Python 3.10.

Setup Environment

conda create -n unimernet python=3.10
conda activate unimernet

Installation Options

Option 1: Standard Installation (Recommended for most users)

pip install --upgrade unimernet
pip install "unimernet[full]"

Option 2: Developer Installation (Local editable mode)

pip install -e ."[full]"

To evaluate the UniMERNet model on a test set:

1. Configure Test Path: Open the files in the configs/val directory and set the path to your test dataset.
2. Execute Testing: Run the test script.

bash script/test.sh

Note: The test.py script uses the path specified in the configs/val folder. Ensure you update test.py with your actual path if necessary.

UniMERNet provides several ways to perform mathematical expression recognition (converting images to LaTeX):

1. Streamlit GUI: For an interactive, user-friendly experience with real-time rendering.

   unimernet_gui

   *Note: Requires `pip install "unimernet[full]"`.*

2. **Command Line Demo**: Predict LaTeX code from images via a script.
   ```bash
python demo.py

3. Jupyter Notebook: For interactive recognition and rendering in a notebook environment.

Run this in your terminal

jupyter-lab ./demo.ipynb

To set up the Mathematical Formula Detection (MFD) component, create a dedicated Conda environment with Python 3.10 and install the ultralytics package. You can download the model weights (weights.pt) using either ModelScope or HuggingFace.

conda create -n mfd python=3.10
conda activate mfd
pip install ultralytics

# download with modelscope
cd MFD/
wget -c https://www.modelscope.cn/models/wanderkid/PDF-Extract-Kit/resolve/master/models/MFD/weights.pt