Open Computer Use

After setting up your environment, use poetry to install dependencies and start the web interface.

Standard Start

This will start the agent and prompt you for instructions in the web interface:

poetry install
poetry run start

Start with an initial prompt

You can pass a specific instruction immediately using the --prompt flag:

poetry run start --prompt "use the web browser to get the current weather in sf"

The display stream of the sandbox should become visible a few seconds after the program starts.

To enable tool use, pass a functions dictionary to the call method. The dictionary should follow this structure:

functions = {
    "function_name": {
        "description": "A description of what the tool does",
        "params": {
            "param_name": "Description of the parameter"
        }
    }
}

When the model decides to use a tool, the call method returns a list of tool call objects. Each object contains:

• type: "function"
• name: The name of the tool
• parameters: A dictionary of the arguments provided by the model.

When the agent needs to interact with a specific UI element (via click, double_click, or right_click), it follows a grounding workflow:

1. Screenshot: It captures the current screen.
2. Grounding: It calls a grounding_model with the user's query and the screenshot to find the (x, y) coordinates.
3. Visualization: It draws a large dot on the screenshot at the target position and saves it for logging.
4. Action: It moves the mouse to the coordinates and executes the requested click command (e.g., self.sandbox.left_click).

The application requires an E2B API key to function. This should be set as an environment variable.

Environment Variable:

• E2B_API_KEY: Your E2B API key.

You can provide this in a .env file, which is automatically loaded by the application.

The agent requires three specific model assignments to function: a grounding_model, a vision_model, and an action_model. These are instantiated using classes from the os_computer_use.providers module.

• grounding_model: Used for grounding tasks. Available providers include OSAtlasProvider and ShowUIProvider.
• vision_model: Used for visual processing. Supported providers include FireworksProvider, OpenAIProvider, AnthropicProvider, MoonshotProvider, GroqProvider, and OpenRouterProvider.
• action_model: Used for executing actions. Supported providers include FireworksProvider, OpenAIProvider, AnthropicProvider, MoonshotProvider, MistralProvider, and GroqProvider.

When selecting a provider, you must pass the specific model identifier (e.g., `

The Browser class provides an interface to open and close a webview window in a separate process. This allows for non-blocking browser interaction.

Key methods:

• open(url, width=None, height=None): Launches a new browser window at the specified URL. You can optionally provide width and height (defaults to 1024x768).
• close(): Sends a close command to the webview process and waits for it to terminate.

The run(instruction) method is the main entrypoint to start an autonomous computer-use session. It takes a natural language instruction representing the goal.

Workflow:

1. Appends the objective to the agent's message history.
2. Enters a loop where it uses an action_model to decide on tool calls.
3. Uses a vision_model to analyze screenshots of the current state.
4. Executes requested tools (like click, type_text, or run_command) via the sandbox.
5. Continues until the model calls the stop tool or the loop terminates.

Note: The method automatically sets the sandbox timeout to 60 seconds during the loop to prevent idle timeouts.

You can add new capabilities to the SandboxAgent by using the @tool decorator on methods within the class. The decorator registers the method's name, description, and parameters into the tools dictionary used by the LLM.

Example:

@tool(description="Custom tool description", params={"arg": "arg description"})
def my_custom_tool(self, arg):
    # implementation
    return "result"

The OpenAIBaseProvider uses the OpenAI Python SDK. It supports:

• Image handling: Automatically detects image types (e.g., PNG, JPEG) using Pillow and encodes them as Base64 data URLs.
• Tool parsing: Includes a fallback mechanism using regex to manually parse JSON tool calls if the provider returns them as unparsed text in the message content.