DOCSEARCHbyAlgolia
Submit Docs

CloudflareBypassForScraping

repository·main·Indexed 25 days ago

https://github.com/sarperavci/cloudflarebypassforscraping

A tool designed to bypass Cloudflare protection during web scraping. It provides two primary modes: generating valid cf_clearance cookies and user-agents via the /cookies endpoint, or acting as a transparent proxy in Mirror Mode that mimics Chrome's TLS/JA3 fingerprints and handles challenges automatically using a patched Chromium stealth browser (CloakBrowser).

Tokens
2.6K
Snippets
13
Records
22
Agent score
82%

What's inside CloudflareBypassForScraping

  1. How Cloudflare Bypass works

    main

    The service bypasses Cloudflare using two primary mechanisms powered by a stealth browser (patched Chromium via CloakBrowser):

    1. Cookie Generation (/cookies): Solves challenges to provide the cf_clearance cookie and the required user_agent pair.
    2. Mirror Mode: Combines cookie injection with TLS/JA3 fingerprint mimicking. This addresses both the JavaScript/Turnstile challenges and the TLS fingerprinting obstacles, allowing you to use any HTTP method without needing a browser on your side.

  2. Use Mirror Mode with `x-hostname`

    main

    Mirror mode acts as a transparent proxy. You point your scraper at the bypass server, and it replays your request to the target site with Cloudflare protection already bypassed.

    How it works:

    1. It generates (or reuses) clearance cookies for the host specified in the x-hostname header.
    2. It replays your exact request (Method, Path, Query, Headers, and Body).
    3. It uses an HTTP client that mimics Chrome's TLS/JA3 fingerprint to bypass fingerprinting detection.
    4. It merges the Cloudflare cookies into the request and returns the real response.

    Headers:

    • x-hostname: The domain of the target website (e.g., protected-site.com).
    • x-proxy (Optional): A proxy URL to route the request through (e.g., http://user:pass@host:port).
    curl -X POST "http://localhost:8000/api/submit" \
  -H "x-hostname: protected-site.com" \
  -H "x-proxy: http://user:pass@host:port" \
  -H "Content-Type: application/json" \
  -d '{"key":"value"}'

  3. Quick Start with Docker

    main

    To get started, run the service using Docker. The container exposes port 8000 by default.

    Once running, you can test the service by pointing a request to the server and providing the x-hostname header to specify the target website.

    docker run -p 8000:8000 ghcr.io/sarperavci/cloudflarebypassforscraping:latest

  5. Use Request Mirroring to bypass Cloudflare

    main

    Mirror mode allows you to forward any HTTP request (GET, POST, etc.) through the bypass server. The server handles clearance-cookie generation and SSL/TLS fingerprinting.

    To use this, point your API base URL to the local server and include the x-hostname header with the target host. The first request generates and caches Cloudflare cookies; subsequent requests reuse them automatically.

    # GET request
curl "http://localhost:8000/api/data" -H "x-hostname: example-site-protected-with-cf.com"

# POST request
curl -X POST "http://localhost:8000/api/submit" \
  -H "x-hostname: cf-protected-website.com" \
  -H "Content-Type: application/json" \
  -d '{"key": "value"}'

  11. Configure Proxy exit-IP check

    main
    When using rotating residential proxies, the exit IP may change, which invalidates the cf_clearance cookie. Enabling the exit-IP check ensures that the bypasser verifies the current IP on every cache hit. If the IP has changed, the cache is invalidated and cookies are regenerated. Note that enabling this adds one HTTP request per cache hit.