Hyperagent Documentation

HyperAgent operates on a repetitive loop to accomplish complex tasks. The execution flow follows these steps:

1. DOM State Extraction: The agent calls getDom(page) to find interactive elements, draws a numbered canvas overlay on the page, and captures a screenshot.
2. Message Building: It constructs a prompt containing the system instructions, the user's task, previous step context, a text representation of the DOM (with numbered indices), and the screenshot.
3. LLM Invocation: The agent sends the prompt to the LLM, requesting a structured list of actions (e.g., clickElement at index 5).
4. Action Execution: The agent iterates through the requested actions, executing them via Playwright (e.g., using a CSS path locator derived from the element index) and waits 2 seconds between actions.
5. Iteration: The loop repeats until the task is complete, the maximum number of steps is reached, or the task is cancelled.

HyperAgent provides two distinct modes of operation for interacting with a page:

🎯 page.perform() - Single Granular Actions

Best for: Specific, low-latency actions like clicking buttons or filling forms.

• Mechanism: Uses the accessibility tree (text-based DOM analysis) without screenshots.
• Pros: Fast, cheap (single LLM call), and reliable.
• Note: page.aiAction() is a deprecated alias for page.perform().

🧠 page.ai() - Complex Multi-Step Tasks

Best for: Workflows that require visual context or multiple steps to complete.

• Mechanism: Can use screenshots with element overlays for visual understanding.
• Pros: Context-aware, adaptive to page state, and handles multi-step logic automatically.
• Parameters:
  • useDomCache (boolean): Reuse DOM snapshots for speed.
  • enableVisualMode (boolean): Enable screenshots and overlays (defaults to false).

🎨 Mix and Match

You can combine these for optimal performance: use perform() for simple interactions and ai() for complex reasoning or multi-step navigation.

const page = await agent.newPage();
await page.goto("https://example.com/login");

// Fast, reliable single actions
await page.perform("fill email with user@example.com");
await page.perform("fill password with mypassword");
await page.perform("click the login button");

// Complex task with multiple steps handled automatically
await page.ai("search for flights from Miami to New Orleans on July 16", {
  useDomCache: true,
});

The Hyperbrowser agent uses a multi-phase pipeline to synchronize the Chrome DevTools Protocol (CDP) state with the DOM structure required for LLM reasoning. This ensures that when an agent interacts with a page, it can correctly identify elements even within complex nested iframes or Out-of-Process Iframes (OOPIFs).

Phase 1: Initialization & Event Listener Setup

When a task starts (via page.ai() or page.aiAction()), the system performs the following:

1. Ensures Frame Contexts: Calls ensureFrameContexts.ready() to prepare the environment.
2. Initializes CDP Client & FrameContextManager: Sets up the connection and the manager responsible for tracking frame lifecycles.
3. Enables CDP Domains: Enables Page, DOM, and Runtime domains.
4. Attaches Event Listeners: Monitors Page.frameAttached, Page.frameNavigated, Runtime.executionContextCreated, and others to track dynamic changes.
5. Builds Initial Frame Graph: Uses Page.getFrameTree() to enumerate existing frames and populates the FrameGraph and sessions Map.

Phase 2: DOM Capture & Multi-Path Discovery

To build an accessible DOM representation (getA11yDOM()), the system:

1. Performs DFS Traversal: Uses DOM.getDocument({ depth: -1, pierce: true }) to traverse the entire document tree.
2. Assigns frameIndex: Indices are assigned based on DOM document order during the Depth-First Search (DFS) traversal, starting from 1 for the first discovered iframe. This index is independent of the order in which CDP events were received.
3. Maps Identifiers: Creates mappings between tagName, backendNodeId, and xpath using a composite key format (e.g., frameIndex-backendNodeId).

In the Hyperbrowser CDP integration, DOM.getFrameOwner is the critical method used to link a CDP frameId to a DOM backendNodeId.

• The Problem: DOM.getDocument returns the backendNodeId of an <iframe> element but does not include its frameId. CDP events like Page.frameAttached provide the frameId but do not specify which DOM node represents that frame.
• The Solution: By calling DOM.getFrameOwner({ frameId }), the system retrieves the backendNodeId associated with that specific frame. This allows the FrameContextManager to match event-driven frame data with the static DOM tree discovered during traversal.

// The key method that links frameId to backendNodeId
private async populateFrameOwner(session: CDPSession, frameId: string): Promise<void> {
  // Call DOM.getFrameOwner with the frameId to get backendNodeId
  const owner = await session.send("DOM.getFrameOwner", { frameId });
  
  // Store backendNodeId in the frame record
  this.graph.upsertFrame({
    frameId,
    backendNodeId: owner.backendNodeId  // ✅ THIS is the link!
  });
}

The A11yDOMState is the complete data structure assembled and sent to the LLM to enable agentic interaction. It contains both human-readable/LLM-friendly data and technical mapping data required to resolve element actions back to the browser.

Key Components:

• simplified (string): A text representation of the accessibility tree (e.g., [0-15] button 'Login') used by the LLM to identify elements.
• elements (Map<EncodedId, AccessibilityNode>): A map of unique identifiers to full accessibility nodes containing roles, names, and metadata.
• backendNodeMap (Record<EncodedId, number>): Maps the EncodedId to the raw backendDOMNodeId for low-level CDP commands.
• xpathMap (Record<EncodedId, string>): Maps the EncodedId to the element's XPath (relative to its frame).
• frameMap (Map<number, IframeInfo>): Maps frame indices to their metadata, including frameId, executionContextId, and sessionId (crucial for OOPIF/cross-origin frames).
• boundingBoxMap (Map<EncodedId, DOMRect>): (Optional) Provides visual coordinates for elements, translated to the main viewport coordinates.
• metrics: Metadata about the capture, such as totalElements, frameCount, and captureTimeMs.

HyperAgent relies on a sophisticated infrastructure to map AI-identified elements to actual Chrome DevTools Protocol (CDP) commands, especially when dealing with iframes and multiple execution contexts.

Core Components

• getCDPClient(page): Lazily creates a PlaywrightCDPClient shared across the page to ensure all features reuse the same sessions.
• FrameContextManager: The authoritative source for mapping encoded element IDs (e.g., 3-283) to specific frames, CDP sessions, and execution contexts. It tracks Page.frameAttached, Page.frameDetached, and Runtime.executionContextCreated events.
• resolveElement(encodedId, ...): Converts an AI-generated ID into a resolvable CDP element by looking up the backendNodeMap, xpathMap, and frameMap managed by the FrameContextManager.
• dispatchCDPAction(method, args, ...): The final step that executes the actual browser command (like click or type) using the resolved CDP element and its bounding box.

To act on an element identified by an LLM (e.g., an encodedId like "1-42"), the agent must resolve that ID through several internal mapping layers. If any piece of this chain is missing, element resolution will fail.

The resolution chain follows this logic:

1. Identify the DOM Node: Use backendNodeMap["1-42"] to find the backendDOMNodeId (e.g., 42).
2. Identify the XPath: Use xpathMap["1-42"] to get the relative XPath (e.g., "//button[1]") as a fallback if the node becomes stale.
3. Identify the Frame Context: Use frameMap.get(frameIndex) to retrieve the IframeInfo. This provides the frameId (for session lookup), the executionContextId (required for XPath evaluation), and the sessionId (to determine which CDP connection to use).
4. Retrieve the CDP Session: Use frameContextManager.getFrameSession(frameId) to obtain the actual CDPSession required to execute the command.

A critical challenge in the integration is that DOM.getDocument does not provide a frameId for same-origin iframes. The agent relies on specific CDP events and API calls to bridge this gap: