Stable Diffusion WebUI Forge

The implementation supports a wide variety of pretrained models, including:

• EfficientNet variants: NoisyStudent (B0-B7, L2), AdvProp (B0-B8), standard EfficientNet (B0-B8), EdgeTPU (S, M, L), CondConv, and Lite.
• MixNet: Including MixNet-XL, L, M, and S.
• MobileNet variants: MobileNet-V3 (Large, Small, Minimalistic), MobileNet-V2 (various scales like 100, 110d, 120d, 140).
• Other architectures: MNASNet (B1, A1, Small), FBNet-C, and Single-Path NAS.

EfficientNet weights ported from TensorFlow checkpoints require specific preprocessing to match original accuracy.

Key Requirements:

• Mean and Std: For EfficientNet AdvProp (AP), EfficientNet EdgeTPU, EfficientNet-CondConv, EfficientNet-Lite, and MobileNet-V3 models, use Inception-style mean and standard deviation: mean=0.5, std=0.5.
• TF Preprocessing: Enabling the TensorFlow preprocessing pipeline via the --tf-preprocessing flag during validation can improve scores by 0.1-0.5%, bringing results closer to the original TensorFlow implementation.
• Padding and Scaling: Accuracy depends on using SAME convolution padding equivalents, specific crop factors, and image scaling as defined in the model's configuration.

MiDaS computes depth from a single image using the following ROS topic interface:

• Input Topic (image_topic): Expects a sensor_msgs/Image message of type RGB8. The image can have any shape.
• Output Topic (midas_topic): Emits a sensor_msgs/Image message of type TYPE_32FC1. This contains inverse relative depth maps in the range [0 - 255] with the original image size and channels=1.

To use MiDaS for monocular depth estimation, follow these setup steps:

1. Install Dependencies: Use Conda to create the environment from the provided YAML file.

   conda env create -f environment.yaml
   conda activate midas-py310

2. Optional Components:

   • Next-ViT model: If using Next-ViT, add the submodule:

     git submodule add https://github.com/isl-org/Next-ViT midas/external/next_vit

   • OpenVINO: For Intel CPU inference using the small legacy model, install OpenVINO:

     pip install openvino

3. Download Weights: Download the desired model weights and place them in the weights folder.

# No single runnable setup command, but provides the sequence:
conda env create -f environment.yaml
conda activate midas-py310

If you prefer using Docker with NVIDIA GPU support:

1. Build the image:

   docker build -t midas .

2. Run inference: Mount your local input, output, and weights directories to the container:

   docker run --rm --gpus all -v $PWD/input:/opt/MiDaS/input -v $PWD/output:/opt/MiDaS/output -v $PWD/weights:/opt/MiDaS/weights midas

Use the validate.py script to run validation on ImageNet datasets. Different models require different arguments for image size, crop percentage, and preprocessing.

Standard validation (e.g., tf_efficientnet_b5):

python validate.py /path/to/imagenet/validation/ --model tf_efficientnet_b5 -b 64 --img-size 456 --crop-pct 0.934 --interpolation bicubic

Validation with TensorFlow preprocessing:

python validate.py /path/to/imagenet/validation/ --model tf_efficientnet_b5 -b 64 --img-size 456 --tf-preprocessing

Validation for models with Inception preprocessing (e.g., EfficientNet-B8 AdvProp):

python validate.py /path/to/imagenet/validation/ --model tf_efficientnet_b8_ap -b 48 --num-gpu 2 --img-size 672 --crop-pct 0.954 --mean 0.5 --std 0.5

Running the Node

To start the MiDaS node, execute the launch script from your catkin workspace:

~/catkin_ws/src/launch_midas_cpp.sh

Testing with Video

To verify the installation using a video file:

1. Place a video file named test.mp4 into the ~/catkin_ws/src/ directory.
2. Run the MiDaS node: ~/catkin_ws/src/launch_midas_cpp.sh.
3. In a separate terminal, run the test nodes:

   cd ~/catkin_ws/src && ./run_talker_listener_test.sh

   Wait approximately 30 seconds for the test to complete.

Note: If you need to use Python 2 instead of Python 3, run: sed -i 's/python3/python2/' ~/catkin_ws/src/midas_cpp/scripts/*.py

# Run the midas node
~/catkin_ws/src/launch_midas_cpp.sh

# Run test nodes in another terminal
cd ~/catkin_ws/src && ./run_talker_listener_test.sh

EfficientNets can be installed via pip after setting up a Conda environment with PyTorch. The package is named geffnet.

Recommended Environment Setup:

1. Create and activate a Conda environment.
2. Install PyTorch, torchvision, and cudatoolkit using the official PyTorch instructions.
3. Install the package using pip:

pip install geffnet

conda create -n torch-env
conda activate torch-env
conda install -c pytorch pytorch torchvision cudatoolkit=10.2
pip install geffnet

To extend the Detectron2 layers with a new operation (Op), follow these steps:

1. Create a new directory for the operation.
2. Implement the new ops within that directory.
3. Declare the Python interface for the new operation in vision.cpp.