rayobrowse

repository·main·Indexed 41 days ago

https://github.com/rayobyte-data/rayobrowse

A self-hosted, Chromium-based stealth browser for high-scale web scraping and automation. It provides coherent device fingerprints to bypass advanced bot detection and supports the Chrome DevTools Protocol (CDP) for integration with tools like OpenClaw, Scrapy, and Playwright. Available as a self-hosted Docker service or a managed Cloud version.

Tokens
5.9K
Snippets
14
Records
24
Agent score
53%

What's inside rayobrowse

  1. Prohibited use by Covered Anti-Bot Companies

    main

    Certain companies and platforms (defined as Covered Anti-Bot Companies) are expressly prohibited from using the Binary or any related rayobrowse materials (documentation, examples, telemetry, etc.) for any Anti-Bot Purpose.

    Anti-Bot Purpose includes: detecting, fingerprinting, blocking, rate limiting, classifying, profiling, reverse engineering, benchmarking, or researching to improve bot detection, WAF, CAPTCHA, or fraud prevention systems.

    Examples of covered entities include (but are not limited to):

    • Security/WAF/Bot Management: Cloudflare, Akamai, DataDome, Kasada, Imperva, Fastly, F5, HUMAN Security, PerimeterX, Arkose Labs, FingerprintJS, etc.
    • Major Platforms: Google (reCAPTCHA, Cloud Armor), AWS (WAF), Microsoft (Azure WAF), Meta (Facebook/Instagram), X (Twitter), TikTok, LinkedIn, Reddit, etc.
    • E-commerce/Travel/Services: eBay, Etsy, Amazon, Booking.com, Airbnb, Ticketmaster, Spotify, Netflix, etc.
  2. Understand rayobrowse licensing

    main

    The project uses a split licensing model:

    1. Repository Wrapper, Examples, Docs, and Config Files: Licensed under the MIT License.
    2. rayobrowse Browser Binary: Governed by a separate license. This license allows most commercial and hobby use but restricts illegal/unethical use and certain commercial resale categories.

    Refer to LICENSE and BROWSER_BINARY_LICENSE.md for full details.

  3. Restricted commercial uses of the Browser Binary

    main

    You are prohibited from using the Binary (directly or indirectly) for the following without a separate written agreement from Rayobyte:

    1. Restricted Browser Solutions: Products that sell, resell, host, or provide browser automation, browser APIs, browser infrastructure, hosted browsers, anti-detect browsers, or stealth browsers to third parties.
    2. Restricted Scraping APIs: Products where third parties call an API or hosted workflow to receive data from web scraping, crawling, extraction, or monitoring services.

    For commercial licensing inquiries regarding these categories, contact sales@rayobyte.com.

  4. Prohibited illegal and unethical uses

    main

    The Binary may not be used for illegal, harmful, or unethical activities, including:

    • Unauthorized access to systems, accounts, or networks.
    • Credential stuffing, phishing, fraud, or malware distribution.
    • Harassment, stalking, or unauthorized collection of sensitive personal information.
    • Infringing on intellectual property or violating privacy and data protection laws.
  5. Legal and Ethical usage of rayobrowse

    main

    rayobrowse must only be used for legal and ethical automation and web data collection.

    Prohibited activities include:

    • Credential attacks
    • Fraud
    • Spam
    • Unauthorized access
    • Harassment
    • Malware or other harmful activity

    Rayobyte is a partner of the EWDCI.

  6. Understand the rayobrowse Browser Binary License

    main

    The rayobrowse project uses two different licenses:

    1. Repository Wrapper: The source files, examples, documentation, and configuration files in this repository are licensed under the MIT License (see LICENSE).
    2. Browser Binary: The actual browser executable (Chromium-based), stealth engine, and bundled binary artifacts are governed by the rayobrowse Browser Binary License.

    If you are using rayobrowse in a commercial, regulated, or high-risk setting, you are responsible for ensuring your use complies with this specific binary license.

  7. Compare Self-Hosted and Cloud rayobrowse

    main

    rayobrowse offers two deployment models: Self-Hosted (free and unlimited) and Cloud (managed scaling and orchestration).

    Key Differences

    FeatureSelf-Hosted (Free)Cloud
    Core EngineFull stealth engine, Fingerprint spoofing, CDP compatible, Desktop fingerprints, Headless/Headful, Proxy supportFull stealth engine, Fingerprint spoofing, Real canvas profiles, CDP compatible, Desktop fingerprints, Headless/Headful, Managed or BYO proxies
    InfrastructureManual managementCentral dashboard, Auto-scaling, Zombie cleanup, OOM protection, Session queuing
    NetworkBring your own proxiesManaged proxies (Datacenter, ISP, Residential)
    AdvancedLimited real-world fingerprintsTens of thousands of fingerprints, Android mobile fingerprints, Persistent profiles (Beta)

    Connection Model

    The connection model is identical for both versions: you call the /connect endpoint to receive a CDP (Chrome DevTools Protocol) URL, which you then use to automate the browser via tools like Playwright or Puppeteer.

  8. Integrate Scrapy with rayobrowse

    main

    To use Scrapy with a stealth browser that passes modern bot detection, combine Scrapy with scrapy-playwright and use rayobrowse as the browser provider.

    rayobrowse provides a Chromium session with a realistic device fingerprint and standard CDP support. The integration works by calling rayobrowse's HTTP /connect endpoint to obtain a CDP WebSocket URL, which is then passed to scrapy-playwright via the PLAYWRIGHT_CDP_URL setting.

    Scrapy spider
      |
      | scrapy-playwright download handler
      v
    Playwright  ---CDP--->  returned rayobrowse CDP URL  --->  stealth Chromium
  9. Configure OpenClaw to use a rayobrowse profile

    main

    OpenClaw requires a profile containing a session-specific CDP URL. Because the CDP URL changes every time a new session is requested, you must update the profile whenever you want a fresh browser session.

    Using OpenClaw CLI

    First, request a new URL:

    CDP_URL=$(curl -fsS "http://localhost:9222/connect?headless=true&os=windows")

    Then, configure the rayobrowse profile:

    openclaw config set browser.enabled true
    openclaw config set browser.defaultProfile rayobrowse
    openclaw config set "browser.profiles.rayobrowse.cdpUrl" "$CDP_URL"

    Manual Configuration

    Alternatively, edit ~/.openclaw/openclaw.json directly:

    {
      browser: {
        enabled: true,
        defaultProfile: "rayobrowse",
        profiles: {
          rayobrowse: {
            cdpUrl: "ws://localhost:9222/cdp/br_59245e8658532863",
          },
        },
      },
    }
    CDP_URL=$(curl -fsS "http://localhost:9222/connect?headless=true&os=windows")
    
    openclaw config set browser.enabled true
    openclaw config set browser.defaultProfile rayobrowse
    openclaw config set "browser.profiles.rayobrowse.cdpUrl" "$CDP_URL"
  10. Use rayobrowse Cloud

    main

    rayobrowse Cloud is available in early access. The integration flow is identical to the local version, but you must use the cloud endpoint and provide an API key via the x-api-key header.

    Endpoint: https://cloud.rayobrowse.com

    To request access, contact sales@rayobyte.com.

    CDP_URL=$(curl -fsS "https://cloud.rayobrowse.com/connect?headless=true&os=windows" \
      -H "x-api-key: YOUR_API_KEY")
  11. Quick Start with rayobrowse via Docker

    main

    To run rayobrowse locally, you must have Docker installed. The project uses Docker to provide a reproducible environment containing the Chromium binary, stealth fingerprint engine, and HTTP daemon.

    Follow these steps to start the service:

    1. Clone the repository.
    2. Start the container using Docker Compose.
    3. Verify the service is healthy by hitting the /health endpoint.
    git clone https://github.com/rayobyte-data/rayobrowse.git
    cd rayobrowse
    docker compose up -d
    curl http://localhost:9222/health