Computer Use AI SDK

The mcp-bridge.ts utility acts as a proxy that allows an MCP client (communicating via stdio) to interact with a Rust-based MCP server that is running in HTTP mode. This is useful when your core logic is implemented in a language like Rust and exposed via an HTTP API, but you need to expose it to MCP clients that only support the Standard Input/Output transport.

To use this bridge, you must:

1. Point the RUST_BINARY constant to your compiled Rust server executable.
2. Ensure the Rust server is configured to listen on a local HTTP endpoint (e.g., http://127.0.0.1:8080).
3. Configure the server.setRequestHandler to forward specific MCP requests to the corresponding Rust HTTP endpoints using fetch.

// Example of forwarding an MCP request to a Rust HTTP endpoint
server.setRequestHandler(someHandler, async (request) => {
  const response = await fetch("http://127.0.0.1:8080/api/click-by-index", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(request.params),
  });

  const data = await response.json();
  return data;
});

The AccessibilityEngine trait defines the interface for platform-specific accessibility engines (macOS, Windows, and Linux). It provides methods to interact with the OS-level UI tree, find elements, and manage applications.

Key capabilities include:

• UI Navigation: Accessing the root element, finding focused elements, or searching for elements using a Selector.
• Application Management: Listing running applications, finding an application by name, or opening a new application.
• Web Interaction: Opening a URL in a specific browser or the system default.

Implementations are platform-specific, so certain methods (like get_element_by_id) may only be available on specific operating systems like Windows.

/// The common trait that all platform-specific engines must implement
pub trait AccessibilityEngine: Send + Sync {
    fn get_root_element(&self) -> UIElement;
    fn get_focused_element(&self) -> Result<UIElement, AutomationError>;
    fn find_element(&self, selector: &Selector, root: Option<&UIElement>) -> Result<UIElement, AutomationError>;
    fn open_application(&self, app_name: &str) -> Result<UIElement, AutomationError>;
    fn open_url(&self, url: &str, browser: Option<&str>) -> Result<UIElement, AutomationError>;
    // ... other methods
}

This project implements the Model Context Protocol (MCP). All communication follows the JSON-RPC pattern using these core types:

• MCPRequest: The standard envelope containing jsonrpc, id, method, and params.
• MCPResponse: The standard successful response containing result.
• MCPErrorResponse: The error response containing an MCPError object.
• MCPError: Contains a numeric code and a message.

Tool execution is handled via ExecuteToolFunctionParams, which specifies the function name and its arguments (as a JSON Value).

#[derive(Debug, Deserialize, Serialize)]
pub struct MCPRequest {
    pub jsonrpc: String,
    pub id: Value,
    pub method: String,
    pub params: Option<Value>,
}

#[derive(Debug, Deserialize, Serialize)]
pub struct MCPResponse {
    pub jsonrpc: String,
    pub id: Value,
    pub result: Value,
}

#[derive(Debug, Deserialize, Serialize)]
pub struct MCPErrorResponse {
    pub jsonrpc: String,
    pub id: Value,
    pub error: MCPError,
}

#[derive(Debug, Deserialize, Serialize)]
pub struct MCPError {
    pub code: i32,
    pub message: String,
    pub data: Option<Value>,
}

#[derive(Debug, Deserialize, Serialize)]
pub struct ExecuteToolFunctionParams {
    pub function: String,
    pub arguments: Value,
}

When a tool function is successfully executed via executeToolFunction, the server returns a JSON-RPC result containing the action outcome and the updated UI state.

Common fields in the result object include:

• ui_state_changed: A boolean indicating if the UI responded to the action.
• elements: An updated list of interactable elements (crucial for the next step in an agentic loop).
• action_timestamp: An RFC3339 formatted string of when the action occurred.
• evaluation_hints: A list of strings suggesting how the client should verify the success of the action (e.g., "check if text field contains the typed content").
• Action-specific results:
  • click: { "success": boolean, "message": string }
  • type_action: { "success": boolean, "message": string }
  • press_key: { "success": boolean, "message": string }
  • application: { "success": boolean, "message": string }
  • url: { "success": boolean, "message": string }
  • input: { "success": boolean }