VisionReasoner

VisionReasoner reformulates various visual perception and reasoning tasks into a unified framework. The currently supported task categories include:

• Detection: Includes tasks like Object Detection, Visual Grounding, Face Detection, Anomaly Detection, and Open Vocabulary Object Detection.
• Segmentation: Includes Semantic Segmentation, Instance Segmentation, Lane Detection, Medical Image Segmentation, and Referring Expression Segmentation.
• Counting: Includes Object Counting, Crowd Counting, Density Estimation, and Pedestrian Detection.
• VQA (Visual Question Answering): Includes Image Captioning, Visual Reasoning, Relational Reasoning, and standard Classification/Question Answering.

Enabling --hybrid_mode allows the framework to intelligently switch between direct detection (using YOLO-World) for simple queries and reasoning-based approaches for complex queries. This optimizes for speed on simple tasks while maintaining accuracy on complex ones.

Simple Query (Direct Detection):

python vision_reasoner/inference.py --image_path "assets/crowd.png" --query "person" --hybrid_mode 

Complex Query (Reasoning-based):

python vision_reasoner/inference.py --image_path "assets/crowd.png" --query "the person who is facing to the camera" --hybrid_mode 

To install VisionReasoner, clone the repository, set up a Conda environment with Python 3.12, and install the required dependencies including torch and torchvision.

git clone https://github.com/JIA-Lab-research/VisionReasoner.git
cd VisionReasoner
conda create -n visionreasoner_test python=3.12
conda activate visionreasoner_test
pip3 install torch torchvision
pip install -r requirements.txt

Use the provided evaluation scripts to test VisionReasoner on segmentation, detection, or counting tasks. Scripts accept either a HuggingFace dataset path or a local filesystem path.

To use a local model instead of downloading from HuggingFace, provide the --model_path and --task_router_model_path arguments.

# Using HuggingFace dataset paths
bash evaluation/eval_segmentation.sh Ricky06662/refcoco_val

# Using local dataset paths
bash evaluation/eval_segmentation.sh /path/to/your/local/refcoco_val

# Using local model paths
bash evaluation/eval_segmentation.sh [dataset_path] \
  --model_path /path/to/local/VisionReasoner-7B \
  --task_router_model_path /path/to/local/TaskRouter-1.5B

Before running inference, you must download the pretrained models. Use git lfs to clone the VisionReasoner-7B and TaskRouter-1.5B models from Hugging Face into a pretrained_models directory.

If you encounter connection issues with Hugging Face, you can set the following environment variable: export HF_ENDPOINT=https://hf-mirror.com

mkdir pretrained_models
cd pretrained_models
git lfs install
git clone https://huggingface.co/Ricky06662/VisionReasoner-7B
git clone https://huggingface.co/Ricky06662/TaskRouter-1.5B

For tasks requiring complex visual reasoning (e.g., answering questions about spatial relationships or object functions), provide the --image_path and a reasoning-based --query. The thinking process is printed to the command line, and visual results are saved to result_visualization.png.

python vision_reasoner/inference.py --image_path "assets/stand_higher.png" --query "find what can make the woman stand higher?"

To perform detection or segmentation, provide an --image_path and a --query describing the target. The results (e.g., bounding boxes or masks) will be saved to result_visualization.png.

python vision_reasoner/inference.py --image_path "assets/donuts.png" --query "please segment the donuts"

The framework supports image generation using the gpt-image-1 model. You must provide an OpenAI API key via the --generation_model_name flag and enable --generation_mode.

Text-to-image generation:

python vision_reasoner/inference.py --image_prompt "Draw a image of a cute dog." --generation_model_name [your openAI api key] --generation_mode 

Image reference generation:

python vision_reasoner/inference.py  --refer_image_path "assets/dog.png" --image_prompt "Generate a cute dog in a forest" --generation_model_name [your openAI api key] --generation_mode 

For naive Visual Question Answering (VQA) or captioning, provide the --image_path and the --query. The final answer is returned directly in the command line without a detailed reasoning process.

python vision_reasoner/inference.py --image_path "assets/company_name.png" --query "What is name of the company?"

The following shell scripts are available for evaluating different vision tasks:

• eval_segmentation.sh: Evaluates segmentation performance on RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg datasets. If the dataset includes bounding box ground truth, it also outputs detection metrics.
• eval_coco.sh: Evaluates detection performance on the COCO dataset.
• eval_count.sh: Evaluates counting performance on counting benchmarks.
• eval_segmentation_with_nonobj.sh: Specifically for evaluation on ViSurf (note: uses different checkpoints than standard VisionReasoner).