DOCSEARCHbyAlgolia
Submit Docs

OpenBrowser AI

repository·main·Indexed 42 days ago

https://github.com/billy-enrizky/openbrowser-ai

An AI-powered browser automation framework using a CodeAgent architecture, LangGraph, and the Chrome DevTools Protocol (CDP). It enables LLMs to write and execute Python code to autonomously navigate, interact with, and extract data from websites. Includes a FastAPI-based backend with REST and WebSocket APIs, VNC support for live browser viewing, and specialized demo applications for ad generation, message scheduling, and news monitoring.

Tokens
90.4K
Snippets
275
Records
387
Agent score
47%

What's inside OpenBrowser

  2. Use OpenBrowser Stress Tests to evaluate automation agents

    main

    OpenBrowser Stress Tests is a collection of difficult test scenarios designed to evaluate the capabilities of browser automation agents. The tests include a wide variety of form implementations, ranging from standard HTML to complex frameworks and difficult UI patterns (like Shadow DOM or Nested Iframes).

    Each test scenario includes:

    • All standard HTML form input types
    • Date and time pickers
    • Character-restricted fields
    • Disabled fields
    • Form validation
    • Submit buttons
    • A success message: the secret is: dumbledore

  3. Compare OpenBrowser against other automation tools

    main

    OpenBrowser provides benchmark comparisons to evaluate its performance against other browser automation and MCP server implementations. There are two primary benchmark suites available:

    1. CLI Benchmark: A 4-way comparison of CLI tools driven by an LLM via a single Bash tool (dated 2026-03-15).
    2. MCP Benchmark: A 3-way comparison of MCP (Model Context Protocol) servers acting as tool providers.

  4. Available User Interfaces for OpenBrowser

    main

    OpenBrowser provides three distinct user interfaces depending on your preferred interaction method:

    1. Terminal (CLI): Uses command_line.py to parse arguments for direct command-line execution.
    2. Gradio: Uses gradio_demo.py to launch an interactive web-based UI via the Gradio framework.
    3. Streamlit: Uses streamlit_demo.py to run a web interface via the Streamlit framework.

    Choose the interface that best fits your workflow, whether you need a headless CLI or a visual interactive dashboard.

    # Terminal usage
python command_line.py

# Gradio usage
python gradio_demo.py

# Streamlit usage
python -m streamlit run streamlit_demo.py

  5. Overview of OpenBrowser Skills

    main

    The Claude Code plugin includes 6 built-in skills that provide guided workflows. These skills are triggered automatically when a user's request matches their description. Each skill uses the openbrowser-ai -c CLI-first approach for automation.

    | Skill | Description |
|-------|-------------|
| `web-scraping` | Extract structured data, handle pagination, and multi-tab scraping |
| `form-filling` | Fill web forms, handle login/registration, and multi-step wizards |
| `e2e-testing` | Test web applications by simulating interactions and verifying outcomes |
| `page-analysis` | Analyze page content, structure, metadata, and interactive elements |
| `accessibility-audit` | Audit pages for WCAG compliance, headings, labels, and ARIA |
| `file-download` | Download files (PDFs, CSVs, images) using authenticated sessions |

  6. Overview of the Accessibility Audit skill

    main

    The accessibility-audit skill audits web pages for accessibility issues, WCAG 2.1 compliance, and screen reader compatibility. It uses Python code execution to perform checks on:

    • Heading structure
    • Form labels
    • Image alt text
    • ARIA attributes
    • Landmark regions
    • Keyboard navigation

    Triggering the skill

    The skill is automatically triggered when a user asks to:

    • "check accessibility"
    • "run an a11y audit"
    • "test WCAG compliance"
    • "check screen reader support"
    • "audit ARIA attributes"
    • "verify keyboard navigation"
    • "find accessibility issues"
    • "check for missing alt text or labels"

  7. What is Code-Use Mode

    main
    Code-Use Mode is a notebook-like code execution system for browser automation. Instead of selecting from a predefined set of actions, an LLM writes Python code that executes within a persistent namespace. This environment provides direct access to browser control functions and allows the agent to use standard Python libraries (like pandas, numpy, BeautifulSoup, etc.) to process, loop, and filter data during the automation process.

  8. Optimize Deep Research agent performance and parallelism

    main

    To ensure the Deep Research skill operates efficiently, follow these architectural constraints:

    • Parallelism via Single Messages: To achieve true parallel execution, the orchestrator must include all $N$ Agent tool calls within a single message. Sequential Agent calls spread across multiple messages will run serially.
    • One Tab per Sub-agent: Each sub-agent must own exactly one browser tab. Do not pass new_tab=True to the navigate() function after the initial call, as multiple tabs per sub-agent will serialize navigation at the CDP layer and negate parallelism benefits.
    • Headless Mode: The daemon starts in headless mode by default (OPENBROWSER_HEADLESS=true). If you need to debug with a visible browser window, override this by setting export OPENBROWSER_HEADLESS=false.
    • Avoid -p mode: This skill uses the -c daemon path, which does not require an API key. Do not use the -p flag, as it triggers the full Browser Agent loop requiring OpenAI, Anthropic, or Google API keys.

  9. Access Agent state and methods inside hooks

    main

    When a lifecycle hook is executing, you have full access to the Agent instance. This allows you to inspect or manipulate the agent's environment in real-time.

    Key Agent attributes available in hooks:

    • Task Management: agent.task (view current task), agent.add_new_task(...) (queue new tasks).
    • Tools & Actions: agent.tools (access Tools() and Registry()). You can manually execute actions via agent.tools.registry.execute_action('action_name', params, browser_session=agent.browser_session).
    • Configuration & Context: agent.settings (initial config), agent.context (user-provided context), agent.sensitive_data (dictionary for sensitive info).
    • LLM & State: agent.llm (the LLM object), agent.state (internal thoughts, outputs, actions).
    • History: agent.history provides access to:
      • agent.history.model_thoughts()
      • agent.history.model_outputs()
      • agent.history.model_actions()
      • agent.history.extracted_content()
      • agent.history.urls()
    • Browser Session: agent.browser_session provides direct access to the BrowserSession and CDP interface:
      • agent.browser_session.get_or_create_cdp_session()
      • agent.browser_session.get_tabs()
      • agent.browser_session.get_current_page_url()
      • agent.browser_session.get_current_page_title()
    • Flow Control: agent.pause() and agent.resume() to control execution flow.

  10. Understand the Browser State Format

    main

    When you receive a response from the agent, the browser state includes:

    • URL & DOM: A compressed DOM tree where interactive elements are marked with indices like [i_123].
    • Loading Status: A list of pending network requests (filtered for ads/tracking) showing URL, duration, and resource type.
    • Element Markers:
      • [i_123]: Interactive elements (buttons, inputs, links).
      • #shadow: Shadow DOM boundaries.
      • #iframe-content: Iframe content boundaries.
      • scroll="...": Indicates scrollable elements and their current position.

    CRITICAL: All indices [i_index] become invalid immediately after any navigation, click, or DOM update. You must re-query the state to get new indices.

  11. Connect the frontend to a backend

    main

    The OpenBrowser Frontend communicates with the backend using a combination of standard HTTP requests and WebSockets for real-time updates (like log streaming and chat messages).

    To establish a connection:

    1. Ensure your backend is deployed and supports WebSocket connections.
    2. Configure the NEXT_PUBLIC_API_URL and NEXT_PUBLIC_WS_URL environment variables to match your backend's address.
    3. The connection is managed via the useWebSocket.ts hook and is established automatically when the application loads.

  12. Export format: Python Script (.py)

    main

    The .py export format is ideal for production deployment, automation, and version control. It generates a complete, standalone runnable script that includes:

    • All necessary imports.
    • A main() function containing the full automation logic.
    • JavaScript code blocks converted into Python string variables.
    • Proper indentation for all executed code.
    • Browser lifecycle management (browser.start() and browser.stop()).

    To use the exported script, simply run python <script_name>.py.

    # Generated from openbrowser code-use session
import asyncio
import json
from openbrowser import BrowserSession
from openbrowser.code_use import create_namespace

async def main():
    # Initialize browser and namespace
    browser = BrowserSession()
    await browser.start()
    
    # Create namespace with all browser control functions
    namespace = create_namespace(browser)
    
    # Extract functions from namespace for direct access
    navigate = namespace["navigate"]
    click = namespace["click"]
    evaluate = namespace["evaluate"]
    # ... other functions
    
    # JavaScript Code Block: extract_products
    extract_products = """(function(){
        return Array.from(document.querySelectorAll('.product')).map(product => ({
            name: product.querySelector('.name')?.textContent,
            price: product.querySelector('.price')?.textContent
        }));
    })()"""
    
    # Cell 1
    await navigate('https://example.com')
    
    # Cell 2
    products = await evaluate(extract_products)
    print(f"Found {len(products)} products")
    
    await browser.stop()

if __name__ == '__main__':
    asyncio.run(main())