scrapy-impersonate

The scrapy_impersonate.RandomBrowserMiddleware rotates the impersonated browser across chrome, firefox, safari, edge, and tor by default.

You can restrict the set of browsers used for rotation by setting the IMPERSONATE_BROWSERS setting in your Scrapy configuration (e.g., IMPERSONATE_BROWSERS = ["chrome", "firefox"]).

To enable impersonation, you must perform three steps in your Scrapy settings:

1. Replace Download Handlers: Update DOWNLOAD_HANDLERS to use scrapy_impersonate.ImpersonateDownloadHandler for both http and https.
2. Set User-Agent: Set USER_AGENT = "" (an empty string). This allows curl_cffi to automatically select the correct User-Agent matching the impersonated browser.
3. Use Asyncio Reactor: Ensure you are using the asyncio-based Twisted reactor for proper asynchronous execution by setting TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor".

DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

USER_AGENT = ""

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Install the package using pip:

pip install scrapy-impersonate

Use the impersonate_curl_options key in Request.meta to pass raw libcurl options through. Keys can be CurlOpt members or their names. These options take precedence over the handler's default settings.

A common use case is pinning header order to bypass WAFs that check for stable browser header sequences (requires curl_cffi >= 0.16.0 for HTTPHEADER_ORDER).

yield scrapy.Request(
    "https://tls.browserleaks.com/json",
    dont_filter=True,
    meta={
        "impersonate": "chrome",
        "impersonate_curl_options": {
            "HTTPHEADER_ORDER": "Host,Connection,User-Agent,Accept,Referer",
        },
    },
)

Use the impersonate_args key in Request.meta to pass arguments directly to curl_cffi.

Warning: allow_redirects is forced to False by the handler so that Scrapy manages redirects. Enabling it via impersonate_args can lead to redirect-based SSRF vulnerabilities.

Commonly used arguments:

• http_version: Set to "v3" to use HTTP/3.
• doh_url: Resolve DNS over HTTPS (requires curl_cffi >= 0.16.0).
• interface: Bind the request to a specific network interface or local source address.
• extra_fp: Fine-tune fingerprint details on top of the impersonated browser.

yield scrapy.Request(
    "https://tls.browserleaks.com/json",
    dont_filter=True,
    meta={
        "impersonate": "chrome124",
        "impersonate_args": {
            "verify": False,
            "timeout": 10,
        },
    },
)