DOCSEARCHbyAlgolia
Submit Docs

Real-ESRGAN

repository·master·Indexed 12 days ago

https://github.com/xinntao/real-esrgan

A practical algorithm for general image and video restoration using pure synthetic data for real-world blind super-resolution. It provides PyTorch and NCNN implementations for various tasks, including general scene restoration (realesr-general-x4v3), anime image enhancement (RealESRGAN_x4plus_anime_6B), and anime video restoration (RealESRGAN AnimeVideo-v3). Features include face enhancement via GFPGAN, arbitrary scaling, and support for tiling and 16-bit images.

Tokens
11.6K
Snippets
32
Records
40
Agent score
90%

What's inside Real-ESRGAN

  1. Avoid using face_enhance for anime content

    master
    The face_enhance option is designed exclusively for real human faces. Do not use this option when processing anime images or animation videos. Avoiding this feature for non-photorealistic content helps save GPU memory.

  2. Use Real-ESRGAN discriminator models for fine-tuning

    master

    The Model Zoo includes discriminator models (_netD) intended for training or fine-tuning purposes rather than direct inference. Use the discriminator that corresponds to your target generator model:

    General Image Discriminators:

    • RealESRGAN_x4plus_netD (for RealESRGAN_x4plus)
    • RealESRGAN_x2plus_netD (for RealESRGAN_x2plus)

    Anime Image Discriminators:

    • RealESRGAN_x4plus_anime_6B_netD (for RealESRGAN_x4plus_anime_6B)

  3. Use Anime Video Models via PyTorch

    master

    You can perform video super-resolution using the realesr-animevideov3 model via the inference_realesrgan_video.py script. This model is optimized for anime and supports X1, X2, X3, and X4 scaling.

    Setup

    First, download the model weights into a weights directory:

    wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth -P weights

    Inference Modes

    • Single GPU, Single Process: Standard inference.
    • Single GPU, Multi-Process: Improves GPU utilization by running multiple processes per GPU. Ensure the total processes do not exceed available CUDA memory.
    • Multi-GPU, Multi-Process: Distributes workload across multiple GPUs using CUDA_VISIBLE_DEVICES.
    # Single GPU and single process inference
CUDA_VISIBLE_DEVICES=0 python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2

# Single GPU and multi process inference
CUDA_VISIBLE_DEVICES=0 python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2 --num_process_per_gpu 2

# Multi GPU and multi process inference
CUDA_VISIBLE_DEVICES=0,1,2,3 python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2 --num_process_per_gpu 2

  4. Run inference on real-world images

    master

    To upscale real-world photos, download the RealESRGAN_x4plus.pth model into a weights folder and run the inference script. You can optionally use the --face_enhance flag to improve facial details using GFPGAN.

    # Download the model
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P weights

# Run inference on the 'inputs' folder with face enhancement
python inference_realesrgan.py -n RealESRGAN_x4plus -i inputs --face_enhance

  5. Convert Real-ESRGAN models to NCNN format

    master

    To use Real-ESRGAN models with the NCNN framework, you must follow a multi-step conversion process: converting the PyTorch model to ONNX, then to NCNN raw format, and finally optimizing it.

    Step 1: PyTorch to ONNX

    Use the scripts/pytorch2onnx.py script to convert your model. Note that you may need to modify the script's code to match your specific model architecture.

    Step 2: ONNX to NCNN Raw

    Navigate to the NCNN build tools directory and use onnx2ncnn.exe to generate the .param and .bin files.

    Step 3: Optimize NCNN Model

    Use ncnnoptimize.exe to optimize the model. Using the 1 flag enables FP16 mode, which is recommended for performance.

    Step 4: Final Parameter Adjustment

    After optimization, you must manually edit the resulting .param file to ensure the blob names for data and output are correct for your specific implementation requirements.

    # 1. Convert to ONNX
python scripts/pytorch2onnx.py

# 2. Convert ONNX to NCNN raw
cd ncnn-master/ncnn/build/tools/onnx
onnx2ncnn.exe realesrgan-x4.onnx realesrgan-x4-raw.param realesrgan-x4-raw.bin

# 3. Optimize NCNN model (FP16 mode)
cd ../../build/tools
ncnnoptimize.exe realesrgan-x4-raw.param realesrgan-x4-raw.bin realesrgan-x4.param realesrgan-x4.bin 1

# 4. Modify blob names 'data' and 'output' in realesrgan-x4.param

  6. Fine-tune Real-ESRGAN with dynamic degradation

    master

    Fine-tuning with dynamic degradation only requires high-resolution (HR) images. The degradation model described in Real-ESRGAN generates low-quality (LQ) images during training.

    1. Prepare Dataset: Follow the standard dataset preparation steps (HR images + meta-info).
    2. Download Pre-trained Models: Download both RealESRGAN_x4plus.pth and RealESRGAN_x4plus_netD.pth to experiments/pretrained_models.
    3. Configure Options: Modify options/finetune_realesrgan_x4plus.yml. Set type: RealESRGANDataset, dataroot_gt, and meta_info.
    4. Run Training: Use the realesrgan/train.py command with the fine-tuning config.
    # Download models
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P experiments/pretrained_models
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.3/RealESRGAN_x4plus_netD.pth -P experiments/pretrained_models

# Fine-tune with 1 GPU
python realesrgan/train.py -opt options/finetune_realesrgan_x4plus.yml --auto_resume

  7. Finetune Real-ESRGAN with degraded images on the fly

    master

    If you only have high-resolution (HR) images, you can finetune Real-ESRGAN by generating low-quality (LQ) images on the fly using the Real-ESRGAN degradation process.

    Steps:

    1. Prepare your HR dataset as described in the dataset preparation guide.
    2. Download pre-trained models (RealESRGAN_x4plus.pth and RealESRGAN_x4plus_netD.pth) into experiments/pretrained_models.
    3. Modify options/finetune_realesrgan_x4plus.yml to point to your dataroot_gt and meta_info.
    4. Run training using the RealESRGANDataset type.
    # Download pre-trained models
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P experiments/pretrained_models
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.3/RealESRGAN_x4plus_netD.pth -P experiments/pretrained_models

# Finetune (Multi-GPU)
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/finetune_realesrgan_x4plus.yml --launcher pytorch --auto_resume

  8. Use the portable NCNN executable for inference

    master

    For users who do not want to set up a Python/PyTorch environment, you can use the portable NCNN-Vulkan executable files. These versions work on Intel, AMD, and Nvidia GPUs and do not require CUDA or PyTorch.

    Available downloads:

    Note: The NCNN implementation processes images in tiles and merges them, which may result in slight visual differences compared to the PyTorch inference script. It also does not support all features of the Python script (e.g., --outscale).

    ./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png -n model_name

  9. Use portable executable files (NCNN) for quick inference

    master

    For users who do not want to set up a CUDA or PyTorch environment, you can use portable NCNN executable files for Intel, AMD, or Nvidia GPUs. These binaries include all necessary models and dependencies.

    Available Models:

    1. realesrgan-x4plus (default)
    2. realesrnet-x4plus
    3. realesrgan-x4plus-anime (optimized for anime images)
    4. realesr-animevideov3 (for animation videos)

    Note: The NCNN version may introduce block inconsistency or slightly different results compared to the PyTorch implementation because it processes images using a tiling and stitching approach. It also lacks certain features like outscale available in the Python script.

    ./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png -n model_name

  10. Use the RealESRGAN_x4plus_anime_6B model for PyTorch inference

    master

    The RealESRGAN_x4plus_anime_6B model is optimized specifically for anime images and features a much smaller model size compared to standard models. To use it via PyTorch, you must first download the weights into a weights directory and then run the inference_realesrgan.py script, specifying the model name with the -n flag.

    # download model
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P weights

# inference
python inference_realesrgan.py -n RealESRGAN_x4plus_anime_6B -i inputs

  11. Train Real-ESRGAN

    master

    The second stage of training uses the trained Real-ESRNet model as the generator initialization. It employs a combination of L1 loss, perceptual loss, and GAN loss.

    Steps:

    1. Locate the trained Real-ESRNet weights (e.g., experiments/train_RealESRNetx4plus_.../model/net_g_1000000.pth).
    2. Update pretrain_network_g in options/train_realesrgan_x4plus.yml to point to these weights.
    3. Run training using the same multi-GPU or single-GPU commands used for Real-ESRNet, targeting the train_realesrgan_x4plus.yml option file.
    # Debug training (Single GPU)
python realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --debug

# Formal training (Multi-GPU)
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --auto_resume

  12. Use NCNN Executable for Anime Video Inference

    master

    For environments without a full PyTorch setup, you can use the NCNN portable executables. This requires a three-step workflow: extracting frames, running inference, and merging frames.

    Step 1: Extract frames with ffmpeg

    Create a tmp_frames directory and extract frames from your source video:

    ffmpeg -i onepiece_demo.mp4 -qscale:v 1 -qmin 1 -qmax 1 -vsync 0 tmp_frames/frame%08d.png

    Step 2: Run NCNN Inference

    Download the appropriate executable for your OS (Windows, Linux, or MacOS) and run:

    ./realesrgan-ncnn-vulkan.exe -i tmp_frames -o out_frames -n realesr-animevideov3 -s 2 -f jpg

    Note: Ensure the out_frames directory exists before running.

    Step 3: Merge frames back into video

    1. Determine the original video's FPS using ffmpeg -i <input_video>.
    2. Merge the enhanced frames:
    ffmpeg -r 23.98 -i out_frames/frame%08d.jpg -c:v libx264 -r 23.98 -pix_fmt yuv420p output.mp4
    1. To include audio from the original video:
    ffmpeg -r 23.98 -i out_frames/frame%08d.jpg -i onepiece_demo.mp4 -map 0:v:0 -map 1:a:0 -c:a copy -c:v libx264 -r 23.98 -pix_fmt yuv420p output_w_audio.mp4
    # Step 1: Extract
ffmpeg -i onepiece_demo.mp4 -qscale:v 1 -qmin 1 -qmax 1 -vsync 0 tmp_frames/frame%08d.png

# Step 2: Inference
./realesrgan-ncnn-vulkan.exe -i tmp_frames -o out_frames -n realesr-animevideov3 -s 2 -f jpg

# Step 3: Merge with audio
ffmpeg -r 23.98 -i out_frames/frame%08d.jpg -i onepiece_demo.mp4 -map 0:v:0 -map 1:a:0 -c:a copy -c:v libx264 -r 23.98 -pix_fmt yuv420p output_w_audio.mp4