YuzuMarker.FontDetection

If the generation process was interrupted, the cache might contain corrupted images or labels. Use the following script to identify and remove them. You should rerun the generation script after this to fill the gaps created by the removal.

python font_ds_detect_broken.py

Use font_ds_generate_script.py to generate the dataset. The script accepts two positional parameters:

1. Index: The index of the current task partition.
2. Partitions: The total number of partitions to split the work into.

Generated images are saved in dataset/font_img.

Example: Running 4 parallel partitions To speed up generation, run these commands in parallel across different processes or machines:

python font_ds_generate_script.py 1 4
python font_ds_generate_script.py 2 4
python font_ds_generate_script.py 3 4
python font_ds_generate_script.py 4 4

To deploy the demo locally using Python, you need either the full font dataset located at ./dataset/fonts or a cache file named font_demo_cache.bin (to be released).

1. Generate sample images: If you have the font dataset, run the generation script first:

   python generate_font_sample_image.py

2. Start the server: Run demo.py with your desired configuration using the CLI arguments provided below.

python demo.py -h

Before generating the dataset, you must organize your assets in the following directory structure:

1. Fonts: Download the CJK font pack and extract it to dataset/fonts.
2. Backgrounds: Place your background images in dataset/pixivimages.

Once the files are in place, run the preprocessing script to clean the filenames:

python dataset_filename_preprocess.py

For high-performance parallel generation on Linux clusters, you can compile a task scheduler using gcc. This approach uses three parameters defined via C macros:

• TOTAL_MISSION: Total number of partitions.
• MIN_MISSION: The starting partition index for this specific machine.
• MAX_MISSION: The ending partition index for this specific machine.

Compilation Example (for 64 partitions split across 4 machines):

# Machine 1 (handles partitions 1-16)
gcc -D MIN_MISSION=1 \
    -D MAX_MISSION=16 \
    -D TOTAL_MISSION=64 \
    batch_generate_script_linux.c \
    -o mission-1-16.out

Execution: Run the resulting object file on the respective machine:

./mission-1-16.out

Pretrained models for YuzuMarker.FontDetection are available on Hugging Face at: https://huggingface.co/gyrojeff/YuzuMarker.FontDetection/tree/main.

Important Compatibility Note: Models were trained using PyTorch 2.0 with torch.compile. To use these pretrained models, you must install PyTorch 2.0 and use torch.compile as demonstrated in demo.py.

If Docker is installed, you can deploy the project directly using a container. You can modify the last line of the Dockerfile to accommodate specific network or port requirements before building.

# Build the docker image
docker build -t yuzumarker.fontdetection .

# Run the docker image
docker run -it -p 7860:7860 yuzumarker.fontdetection

Use the following script to monitor the progress of your dataset generation task.

python font_ds_stat.py

To run generation on a Linux cluster, use the provided setup script. This script creates a virtual environment in the venv directory and installs necessary packages, including libraqm (required for PIL text rendering).

Prerequisites:

• python3 available in PATH
• python3-venv installed

./linux_venv_setup.sh

Start training using train.py. You can provide multiple dataset paths as command-line arguments, and the script will automatically merge them. Ensure your datasets are located under a dataset directory (or specify paths explicitly).

python train.py -s ./dataset/font_img -m resnet50 -n my_model