UI-TARS models output coordinates in a specific format (e.g., click(start_box='(197,525)')). To use these coordinates for actual GUI interaction, you must map the model's output coordinates back to the original image dimensions. This is necessary because the model operates on a resized version of the image that adheres to specific pixel constraints and divisibility rules.
Workflow
- Parse Coordinates: Extract the raw
x and y values from the model's text response using regular expressions. - Determine Resized Dimensions: Use the
smart_resize logic to calculate the dimensions (new_width, new_height) the model actually saw. This logic ensures dimensions are divisible by IMAGE_FACTOR (default 28) and stay within MIN_PIXELS and MAX_PIXELS bounds. - Map to Original Image: Calculate the actual pixel position on the original image using the ratio of the model's output to the resized dimensions.
Note: For a complete implementation of action space parsing, refer to the uitars_agent.py in the OSWorld repository.
from PIL import Image
import math
# Constants used by the model's vision processing
IMAGE_FACTOR = 28
MIN_PIXELS = 100 * 28 * 28
MAX_PIXELS = 16384 * 28 * 28
MAX_RATIO = 200
def round_by_factor(number: int, factor: int) -> int:
return round(number / factor) * factor
def ceil_by_factor(number: int, factor: int) -> int:
return math.ceil(number / factor) * factor
def floor_by_factor(number: int, factor: int) -> int:
return math.floor(number / factor) * factor
def smart_resize(
height: int, width: int, factor: int = IMAGE_FACTOR, min_pixels: int = MIN_PIXELS, max_pixels: int = MAX_PIXELS
) -> tuple[int, int]:
if max(height, width) / min(height, width) > MAX_RATIO:
raise ValueError(f"absolute aspect ratio must be smaller than {MAX_RATIO}")
h_bar = max(factor, round_by_factor(height, factor))
w_bar = max(factor, round_by_factor(width, factor))
if h_bar * w_bar > max_pixels:
beta = math.sqrt((height * width) / max_pixels)
h_bar = floor_by_factor(height / beta, factor)
w_bar = floor_by_factor(width / beta, factor)
elif h_bar * w_bar < min_pixels:
beta = math.sqrt(min_pixels / (height * width))
h_bar = ceil_by_factor(height * beta, factor)
w_bar = ceil_by_factor(width * beta, factor)
return h_bar, w_bar
# Example usage for mapping
img = Image.open('./data/coordinate_process_image.png')
width, height = img.size
model_output_width, model_output_height = 197, 525
new_height, new_width = smart_resize(height, width)
# Map model coordinates back to original image scale
new_coordinate = (int(model_output_width/new_width * width), int(model_output_height/new_height * height))
print(f'Mapped Coordinate: {new_coordinate}')
