Browsers Benchmark Toolkit

repository·main·Indexed 43 days ago

https://github.com/techinz/browsers-benchmark

A toolkit for testing browser automation engines against modern web protection systems like Cloudflare, DataDome, and Akamai. It measures bypass success rates, performance metrics (CPU/memory), fingerprinting resistance via reCAPTCHA and CreepJS scores, and network analysis. Supports various engines including Playwright, Selenium, Camoufox, Patchright, and AdsPower.

Tokens
6K
Snippets
12
Records
18
Agent score
56%

What's inside Browsers Benchmark

  1. Overview of Browser Engine Benchmark

    main

    The Browser Engine Benchmark is a toolkit designed to test browser automation engines against modern web protection systems (e.g., Cloudflare, DataDome, Akamai). It evaluates engines based on four primary dimensions:

    • Bypass Success Rate: How effectively the engine bypasses major bot detection systems.
    • Performance Metrics: Resource consumption including memory usage, CPU usage, and page load times.
    • Fingerprinting Resistance: Measured via reCAPTCHA scores and CreepJS trust ratings.
    • Network Analysis: Validation of proxy effectiveness, including IP detection and WebRTC leak testing.
  2. Interpret Browser Benchmark Results Summary

    main

    The Browser Benchmark Results Summary provides a comparative analysis of different browser engines and automation tools based on several key performance and stealth metrics. This summary is used to evaluate which engine is best suited for specific automation tasks, such as bypassing bot detection or minimizing resource consumption.

    Key metrics included in the summary are:

    • Overall Bypass Rate (%): The percentage of successful protection system bypasses.
    • Resource Usage Comparison: Memory usage (MB) and CPU usage (%).
    • Recaptcha Scores: A score between 0 and 1 indicating the likelihood of passing Recaptcha challenges.
    • CreepJS Scores: Includes Trust Score (%), Bot Score (%), and WebRTC IP detection.
    • IP (Ipify): The public IP address detected for the engine.
  3. Supported Browser Engines

    main

    The benchmark supports a wide variety of automation engines, categorized by their licensing and architecture:

    Proprietary

    • AdsPower: Anti-detect browser with per-profile fingerprinting. Requirement: Requires the AdsPower desktop app running locally and an API key.

    Open-Source (Playwright-based)

    • Camoufox
    • Patchright
    • Playwright Stealth (tf-playwright-stealth)
    • CloakBrowser

    Open-Source (Other Frameworks)

    • Playwright: Microsoft's standard framework (Chrome, Firefox, Safari).
    • Selenium: Standard automation framework (tested without proxies).
    • Seleniumbase: Professional toolkit for web automation.
    • NoDriver: Supports SOCKS5 proxies only.
    • ZenDriver: Based on NoDriver.
  4. Understanding Benchmark Metrics

    main

    When reviewing benchmark results, use the following criteria to evaluate engine quality:

    reCAPTCHA Scores

    Measured via antcpt.com/score_detector. Scores range from 0.0 to 1.0:

    • 1.0: Very likely a human.
    • 0.0: Very likely a bot.
    • >= 0.7: Easier to pass reCAPTCHA 2 challenges.
    • < 0.3: Likely to trigger difficult reCAPTCHA 2 challenges.
    • Note: nan indicates the website stopped working during the test.

    CreepJS Scores

    Evaluates fingerprinting trust.

    • Trust/Bot Scores: Currently disabled in some versions of CreepJS.
    • WebRTC IP: If the WebRTC IP differs from your proxy IP, no leakage occurred. If it matches your real IP, the engine has leaked your identity.

    IP Validation (Ipify)

    • Success: The detected IP matches your proxy IP.
    • Failure: The detected IP matches your real/local IP.
  5. How to add new test targets

    main

    To add a new website or protection system to the benchmark, follow these three steps:

    1. Define the Target

    Modify config/benchmark_targets.py to add a new Target object:

    Target(
        name="custom_site",
        url="https://example.com",
        check_function="check_custom_bypass",
        description="Custom site protection test"
    )

    2. Implement the Check Function

    Create a function in utils/targets/check_bypass (e.g., custom_bypass.py) that accepts a BrowserEngine and returns a boolean indicating if the bypass was successful (e.g., True if no captcha was found).

    from engines.base import BrowserEngine
    
    async def check_custom_bypass(engine: BrowserEngine) -> bool:
        # Use engine.locator to find elements
        element_found, element_html = await engine.locator('//div[@class="captcha"]')
        return not element_found

    3. Register the Checker

    Add your new function to the checkers dictionary in config/benchmark_targets.py within the BypassTargetsSettings class:

    checkers: Dict[str, Callable] = Field(
        default_factory=lambda: {
            "check_cloudflare_bypass": check_cloudflare_bypass,
            "check_custom_bypass": check_custom_bypass,
        }
    )
    from engines.base import BrowserEngine
    
    async def check_custom_bypass(engine: BrowserEngine) -> bool:
        element_found, element_html = await engine.locator('//div[@class="captcha"]')
        return not element_found
  6. Proxy Requirements for Accurate Benchmarking

    main

    To ensure benchmark results reflect the browser engine's capabilities rather than network reputation, using a clean proxy is essential.

    Failure to use a clean proxy can lead to inaccurate data due to:

    • IP Reputation: Your local or datacenter IP might already be flagged by security systems.
    • Clean Testing Environment: Proxies ensure you are testing the engine, not the network.
    • Rate Limiting: Repeated tests from a single IP can trigger rate limits, artificially lowering bypass success rates.
  7. How to add new browser engines

    main

    To extend the benchmark with a new automation framework, you must implement a new engine class and register it.

    1. Implement the Engine Class

    Extend the appropriate base class depending on the underlying technology:

    For generic engines:

    class CustomEngine(BrowserEngine):
        async def start(self) -> None:
            # Initialize browser
            
        async def navigate(self, url: str) -> Dict[str, Any]:
            # Navigation logic

    For Playwright-based engines: Extend PlaywrightBase.

    For Selenium-based engines: Extend SeleniumBase.

    2. Register the Engine

    Add your new class to the base_engines list in config/engines.py within the EnginesSettings class:

    base_engines = [
        {
            "class": CustomEngine,
            "params": {"headless": True, "name": "custom_engine", "browser_type": "chromium"}
        },
        # ... other engines
    ]
  8. Configure proxies for the benchmark

    main

    Proxies are highly recommended for accurate testing.

    1. Create a file named proxies.txt in the documents directory.
    2. Add proxy URLs in one of these formats:
      • http://username:password@proxy_host:port
      • http://proxy_host:port

    Important Constraints:

    • The number of proxies must be greater than or equal to the number of engines you are testing.
    • Different engines require different protocols:
      • Playwright supports only HTTP and HTTPS.
      • NoDriver supports only SOCKS5.
      • Selenium does not use proxies.
    • Ensure your proxies.txt contains a mix of protocols (e.g., HTTP/HTTPS and at least one SOCKS5) to satisfy all engines.

    Example documents/proxies.txt content:

    http://proxy1.example.com:8080
    http://proxy2.example.com:8080
    http://username:password@proxy3.example.com:8080
    http://username:password@proxy4.example.com:8080
    socks5://username:password@proxy5.example.com:8080
    http://proxy1.example.com:8080
    http://username:password@proxy3.example.com:8080
    socks5://username:password@proxy5.example.com:8080
  9. Install the Browser Engine Benchmark

    main

    Follow these steps to set up the environment and install the necessary browser engines.

    1. Clone and Python Setup

    git clone https://github.com/techinz/browsers-benchmark.git
    cd browsers-benchmark
    python -m venv venv
    source venv/bin/activate  # On Windows: venv\Scripts\activate
    pip install -r requirements.txt

    2. Install Browser Engines

    Depending on which engines you want to test, run the corresponding commands:

    Playwright

    playwright install
    # On Linux also run:
    playwright install-deps

    Camoufox

    # Windows
    camoufox fetch
    
    # Linux
    python -m camoufox fetch
    sudo apt install -y libgtk-3-0 libx11-xcb1 libasound2

    Patchright

    patchright install chromium

    AdsPower (Optional)

    1. Install the AdsPower desktop app.
    2. Enable the local API in Settings → API.
    3. Set ADSPOWER_API_KEY in your .env file.
    4. Keep AdsPower running during the benchmark.

    3. Configuration and Execution

    cp .env.example .env
    # Edit .env with your proxy settings if needed
    python main.py
    git clone https://github.com/techinz/browsers-benchmark.git
    cd browsers-benchmark
    python -m venv venv
    source venv/bin/activate
    pip install -r requirements.txt
    playwright install
    python main.py
  10. Configure environment variables via .env

    main

    The benchmark uses a .env file for configuration. Key variables include:

    Proxy Configuration

    • PROXY_ENABLED: Set to true to enable proxy usage.
    • PROXY_FILE_PATH: Path to your proxy list (e.g., documents/proxies.txt).
    • PROXY_MAX_RETRIES: Number of retry attempts for proxy connections.

    Performance Settings

    • PAGE_LOAD_TIMEOUT_S: Timeout for page loads in seconds.
    • PAGE_STABILIZATION_DELAY_S: Delay for page stabilization in seconds.
    • MAX_RETRIES: Global retry limit.

    AdsPower Settings (Only if using AdsPower engine)

    • ADSPOWER_API_KEY: Your API key from AdsPower Settings → API.
    • ADSPOWER_BASE_URL: The local API endpoint (default: http://local.adspower.net:50325).
    • ADSPOWER_GROUP_ID: The specific AdsPower group ID to use.
    PROXY_ENABLED=true
    PROXY_FILE_PATH=documents/proxies.txt
    PROXY_MAX_RETRIES=3
    PAGE_LOAD_TIMEOUT_S=90
    PAGE_STABILIZATION_DELAY_S=5
    MAX_RETRIES=3
    ADSPOWER_API_KEY=your_api_key_here
    ADSPOWER_BASE_URL=http://local.adspower.net:50325
    ADSPOWER_GROUP_ID=
  11. Reference: Overall Bypass Rate Metrics

    main

    The Overall Bypass Rate table lists the percentage of successful bypasses for various browser engines. Higher percentages indicate better stealth capabilities against protection systems.

    | Engine | Bypass Rate (%) |
    |-----------------|----------------:|
    | patchright | 100.0 |
    | cloakbrowser | 90.0 |
    | camoufox_headless | 90.0 |
    | nodriver-chrome | 80.0 |
    | adspower | 80.0 |
    | seleniumbase-cdp-chrome | 80.0 |
    | adspower_headless | 70.0 |
    | tf-playwright-stealth-firefox | 70.0 |
    | tf-playwright-stealth-firefox_headless | 70.0 |
    | zendriver-chrome | 70.0 |
    | cloakbrowser_headless | 60.0 |
    | tf-playwright-stealth-chromium | 60.0 |
    | playwright-chrome | 60.0 |
    | playwright-firefox_headless | 60.0 |
    | playwright-firefox | 60.0 |
    | zendriver-chrome_headless | 60.0 |
    | tf-playwright-stealth-chromium_headless | 50.0 |
    | selenium-chrome__no_proxy | 50.0 |
    | playwright-chrome_headless | 40.0 |
    | nodriver-chrome_headless | 40.0 |
    | patchright_headless | 40.0 |
    | camoufox | 30.0 |
    | selenium-chrome_headless__no_proxy | 30.0 |
  12. Reference: Resource Usage Comparison Metrics

    main

    The Resource Usage Comparison table tracks the efficiency of each engine in terms of Memory (MB) and CPU (%). This is critical for scaling automation tasks.

    | Engine | Memory Usage (MB) | CPU Usage (%) |
    |-----------------|------------------:|--------------:|
    | adspower_headless | 123.0 | 4.9 |
    | adspower | 130.0 | 7.0 |
    | playwright-chrome_headless | 517.0 | 7.4 |
    | tf-playwright-stealth-chromium_headless | 522.0 | 8.2 |
    | zendriver-chrome_headless | 818.0 | 21.5 |
    | tf-playwright-stealth-chromium | 913.0 | 28.6 |
    | cloakbrowser_headless | 936.0 | 27.2 |
    | selenium-chrome_headless__no_proxy | 948.0 | 10.1 |
    | zendriver-chrome | 1009.0 | 38.8 |
    | selenium-chrome__no_proxy | 1034.0 | 29.8 |
    | tf-playwright-stealth-firefox_headless | 1034.0 | 44.7 |
    | playwright-firefox_headless | 1068.0 | 70.3 |
    | cloakbrowser | 1082.0 | 54.4 |
    | playwright-chrome | 1103.0 | 28.5 |
    | seleniumbase-cdp-chrome | 1157.0 | 51.9 |
    | playwright-firefox | 1161.0 | 75.5 |
    | camoufox | 1181.0 | 53.3 |
    | tf-playwright-stealth-firefox | 1261.0 | 67.7 |
    | nodriver-chrome_headless | 1275.0 | 24.4 |
    | patchright_headless | 1277.0 | 16.6 |
    | patchright | 1314.0 | 53.3 |
    | camoufox_headless | 1318.0 | 88.6 |
    | nodriver-chrome | 1389.0 | 47.4 |