OmniParser

OmniTool is a system designed to control a Windows 11 VM using OmniParser and a vision model. It consists of three main components:

1. omniparserserver: A FastAPI server running OmniParser V2, used for visual parsing.
2. omnibox: A Windows 11 VM running inside a Docker container, providing the computer use API.
3. gradio: A user interface to provide commands and monitor the reasoning and execution process on OmniBox.

Deployment Recommendation: For optimal performance, run omnibox and gradio on a CPU machine, and run omniparserserver on a GPU server.

The Gradio UI allows you to interact with the AI agent. Ensure your omniparserserver and omnibox are running before starting.

1. Navigate to the gradio directory:

   cd OmniParser/omnitool/gradio

2. Activate your conda environment:

   conda activate omni

3. Start the Gradio server, specifying the URLs for the Windows host and the OmniParser server:

   python app.py --windows_host_url localhost:8006 --omniparser_server_url localhost:8000

4. Open the URL provided in the terminal output, enter your API Key, and begin using the agent.

To evaluate OmniParser on the ScreenSpot Pro (ss pro) benchmark, you can use the adapted evaluation scripts provided in this repository. This setup is designed to be compatible with the original ScreenSpot Pro official repository.

Steps to use the adapted evaluation:

1. Replace the model file: Take the script eval/ss_pro_gpt4o_omniv2.py from this repository and use it to replace the existing models/gpt4x.py file in the original ScreenSpot Pro repository. This script contains the specific prompts used for the OmniParser evaluation.
2. Review results: Inference results for ScreenSpot Pro using the GPT4o + OmniParser v2 combination are available in eval/logs_sspro_omniv2.json.

# Note: This guide describes a file replacement workflow in an external repository.
# 1. Copy eval/ss_pro_gpt4o_omniv2.py
# 2. Paste into [original ss pro repo]/models/gpt4x.py

Follow these steps to set up the OmniParser V2 FastAPI server using Conda:

1. Create and activate a Conda environment with Python 3.12:

   conda create -n "omni" python==3.12
   conda activate omni

2. Install dependencies:

   pip install -r requirements.txt

3. Download the required weights (Detector and Caption weights):

   huggingface-cli download microsoft/OmniParser-v2.0 icon_detect_v3/model.pt --revision refs/pr/37 --local-dir weights
   rm -rf weights/icon_caption weights/icon_caption_florence
   huggingface-cli download microsoft/OmniParser-v2.0 --local-dir weights --repo-type model --include "icon_caption/*"
   mv weights/icon_caption weights/icon_caption_florence

4. Start the server:

   cd OmniParser/omnitool/omniparserserver
   python -m omniparserserver

The Gradio interface allows you to adjust several parameters for the AI agent's behavior:

• Model: Select from supported models including omniparser + gpt-4o, omniparser + o1, omniparser + R1, omniparser + qwen2.5vl, and Anthropic's claude-3-5-sonnet-20241022.
• N most recent screenshots: A slider to control how many recent screenshots are sent to the model (0-10).
• API Provider: Select the provider for your LLM (e.g., openai, anthropic, groq, dashscope, bedrock, vertex).
• API Key: Enter the required API key for the selected provider.

Note: Changing the Model may automatically update the available API Provider options and the required API Key placeholder.

The OmniTool interface is provided via a Gradio web application. You can launch it from the command line, specifying the URLs for the Windows host (where the VNC session is running) and the OmniParser server.

To run the application, use the following command:

python app.py --windows_host_url localhost:8006 --omniparser_server_url localhost:8000

By default, the app launches on 127.0.0.1:7888.

To use OmniParser, you need to initialize a YOLO model for object detection (SOM) and a caption model for semantic labeling. You can choose between two caption model types: fine-tuned blip2 or florence2.

import torch
from util.utils import get_yolo_model, get_caption_model_processor

device = 'cuda' # or 'cpu'

# Initialize the SOM (Set of Marks) model using YOLO
som_model = get_yolo_model(device=device)

# Initialize the caption model processor
# Options for model_name include 'florence2'
caption_model_processor = get_caption_model_processor(
    model_name="florence2", 
    model_name_or_path="weights/icon_caption_florence", 
    device=device
)

The OmniParser OCR library (Paddle) requires the C++ Redistributable on Windows. If you encounter this error:

1. Install the C++ Redistributable on your Windows host.
2. Rerun the dependency installation:

   pip install -r requirements.txt

If Gradio returns a 'Windows Host is not responding' error, the server inside the VM is likely unavailable.

Diagnostic Steps:

1. Verify the VM server status by running this command inside the container:

   docker exec -it omni-windows bash -c "curl http://localhost:5000/probe"

2. If the VM is still in its initial setup phase, wait for the terminal window in the NoVNC viewer to disappear.

Resolution Steps:

1. Wait up to 10 minutes.
2. Restart the VM:

   ./manage_vm.sh stop
   ./manage_vm.sh start

3. If issues persist, delete the VM but keep the storage folder, then run create again:

   ./manage_vm.sh delete
   ./manage_vm.sh create

4. Factory Reset: If all else fails, delete the VM, manually delete the vm/win11storage folder, and run ./manage_vm.sh create again.