Agent S Framework

Agent S2.5 uses a grounding function set_cell_values that requires the soffice CLI and the unotools library. Because the WAA VM defaults to a GIMP Python 2.x environment, you must configure the system to use LibreOffice's Python (3.10+) instead.

Step 1: Capture existing dependencies

1. Connect to the VM.
2. Open a terminal and navigate to the Desktop directory.
3. Run pip freeze > requirements.txt to save current libraries.

Step 2: Configure System Path

1. Find the LibreOffice Python path using where python inside the VM.
2. Open Environment Variables in the Windows Search bar.
3. Edit the Path variable under System variables.
4. Add the LibreOffice Python path and ensure it is above the GIMP Python path.
5. Verify by running soffice and checking if import uno works in a Python shell.

Step 3: Install dependencies for LibreOffice Python

1. Since LibreOffice Python lacks pip, download get-pip.py from https://bootstrap.pypa.io/get-pip.py.
2. Run python get-pip.py (ensuring you are using the LibreOffice Python).
3. Install the captured requirements: pip install -r requirements.txt.

Step 4: Update WAA Source Code

In the WAA repository, locate the Python controller and update the command list to use the absolute path of the LibreOffice Python to ensure the Flask server uses the correct version:

Change: command_list = ["python", "-c", self.pkgs_prefix.format(command=command)] To: command_list = ["/absolute/path/to/libreoffice/python", "-c", self.pkgs_prefix.format(command=command)]

# Inside VM Desktop directory
pip freeze > requirements.txt

# After installing pip for LibreOffice Python
pip install -r requirements.txt

Agent S2 uses the set_cell_values grounding function, which requires soffice CLI and the unotools library. Because the default VM Python is a GIMP Python (2.x), you must configure the environment to use LibreOffice's Python (3.10+).

Step 1: Prepare Requirements

1. Connect to the VM.
2. Open a terminal and run pip freeze > requirements.txt from the Desktop directory to capture current dependencies.

Step 2: Configure System Path

1. Find the LibreOffice python.exe path (use where python inside the VM).
2. Open Environment Variables in the Windows search bar.
3. Edit the Path under System variables.
4. Add the LibreOffice Python path to the list, ensuring it is above the GIMP Python path.
5. Verify by running soffice and checking if import uno works in a Python script.

Step 3: Install Dependencies for LibreOffice Python

1. Download get-pip.py from https://bootstrap.pypa.io/get-pip.py.
2. Run python get-pip.py using the LibreOffice Python executable.
3. Install the captured requirements: pip install -r requirements.txt.

Step 4: Update WAA Code

In the WAA repository, locate the file src/win-arena-container/client/desktop_env/controllers/python.py and update the command_list assignment to use the absolute path to the LibreOffice Python executable:

# Change this:
command_list = ["python", "-c", self.pkgs_prefix.format(command=command)]

# To this:
command_list = ["absolute/path/to/libreoffice/python", "-c", self.pkgs_prefix.format(command=command)]

To integrate Agent S3 into your own Python applications, use the gui_agents SDK. The workflow involves defining engine parameters for both the main model and the grounding model, initializing an OSWorldACI grounding agent, and then creating an AgentS3 instance.

Core Components

• AgentS3: The main agent class.
• OSWorldACI: The grounding agent that translates actions into executable Python code.
• LocalEnv: (Optional) Enables the local coding environment.

Implementation Steps

1. Define engine_params for the main model (provider, model, base_url, api_key, temperature).
2. Define engine_params_for_grounding for the grounding model (engine_type, model, base_url, api_key, grounding_width, grounding_height).
3. Initialize OSWorldACI with the engine parameters and platform.
4. Initialize AgentS3 with the grounding agent and platform.
5. Call agent.predict(instruction, observation) where observation is a dictionary containing a screenshot (as bytes).

import pyautogui
import io
from gui_agents.s3.agents.agent_s import AgentS3
from gui_agents.s3.agents.grounding import OSWorldACI
from gui_agents.s3.utils.local_env import LocalEnv
from dotenv import load_dotenv

load_dotenv()

# 1. Setup Parameters
current_platform = "linux"  # "darwin", "windows"

engine_params = {
  "engine_type": "openai",
  "model": "gpt-5-2025-08-07",
  "api_key": "YOUR_API_KEY"
}

engine_params_for_grounding = {
  "engine_type": "huggingface",
  "model": "ui-tars-1.5-7b",
  "base_url": "http://localhost:8080",
  "grounding_width": 1920,
  "grounding_height": 1080,
}

# 2. Initialize Agents
enable_local_env = False
local_env = LocalEnv() if enable_local_env else None

grounding_agent = OSWorldACI(
    env=local_env,
    platform=current_platform,
    engine_params_for_generation=engine_params,
    engine_params_for_grounding=engine_params_for_grounding,
    width=1920,
    height=1080
)

agent = AgentS3(
    engine_params,
    grounding_agent,
    platform=current_platform,
    max_trajectory_length=8,
    enable_reflection=True
)

# 3. Run Inference
screenshot = pyautogui.screenshot()
buffered = io.BytesIO()
screenshot.save(buffered, format="PNG")
screenshot_bytes = buffered.getvalue()

obs = {
  "screenshot": screenshot_bytes,
}

instruction = "Close VS Code"
info, action = agent.predict(instruction=instruction, observation=obs)

# Execute the returned action
exec(action[0])