Selenium Wire

Selenium Wire allows you to modify network traffic on the fly using interceptors.

• Request Interceptor: A function that accepts a single argument (request). Set it via driver.request_interceptor.
• Response Interceptor: A function that accepts two arguments (request, response). Set it via driver.response_interceptor.

Important Patterns:

• Replacing Headers: Since request.headers allows duplicates, always use del request.headers['header-name'] before setting a new value to ensure you are replacing rather than appending.
• Updating Parameters: Request parameters are held in a regular dictionary. To update them, you must read the dictionary, modify it, and write it back to request.params.
• Updating JSON Bodies: For POST requests with JSON, you must decode the request.body (bytes) to a string, parse it, modify it, and then re-encode it back to bytes. Remember to update the Content-Length header manually after modifying the body.

def interceptor(request):
    params = request.params
    params['foo'] = 'bar'
    request.params = params

driver.request_interceptor = interceptor
driver.get(...)

# To unset an interceptor:
del driver.request_interceptor
del driver.response_interceptor

You can configure Selenium Wire to use an upstream proxy server by passing a proxy dictionary to the seleniumwire_options argument of the webdriver.

Supported configurations include:

• HTTP/HTTPS Proxies: Specify URLs for http and https schemes.
• Basic Authentication: Include credentials directly in the URL (e.g., https://user:pass@host:port).
• Custom Authorization: Use the custom_authorization key within the proxy dictionary to provide non-Basic authentication headers, such as Bearer tokens.
• SOCKS Proxies: Use socks5, socks4, or socks5h schemes. Use socks5h to perform DNS resolution on the proxy server instead of the client.

Alternatively, you can set the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables.

# HTTP Proxy with Basic Auth
options = {
    'proxy': {
        'http': 'http://192.168.10.100:8888',
        'https': 'https://user:pass@192.168.10.100:8888',
        'no_proxy': 'localhost,127.0.0.1'
    }
}
driver = webdriver.Chrome(seleniumwire_options=options)

# SOCKS Proxy
options = {
    'proxy': {
        'http': 'socks5://user:pass@192.168.10.100:8888',
        'https': 'socks5://user:pass@192.168.10.100:8888',
        'no_proxy': 'localhost,127.0.0.1'
    }
}
driver = webdriver.Chrome(seleniumwire_options=options)

# Custom Proxy-Authorization (e.g., Bearer token)
options = {
    'proxy': {
        'https': 'https://192.168.10.100:8888',
        'custom_authorization': 'Bearer mytoken123'
    }
}

To improve performance or reduce overhead, you can restrict what Selenium Wire captures using several methods:

1. driver.scopes: A list of regular expressions. Only URLs matching these patterns will be captured. Unmatched requests still pass through the proxy but are not stored.
2. seleniumwire_options['disable_capture']: Disables request interception and storage entirely. Interceptors will not execute.
3. seleniumwire_options['exclude_hosts']: A list of hostnames that will bypass the Selenium Wire proxy entirely, going direct from the browser to the server.
4. request.abort(): Blocking specific request types (like images) via an interceptor to speed up page loads.

# Using scopes
driver.scopes = [
    '.*stackoverflow.*',
    '.*github.*'
]

# Using exclude_hosts
options = {
    'exclude_hosts': ['host1.com', 'host2.com']
}
driver = webdriver.Chrome(seleniumwire_options=options)

# Disabling capture
options = {
    'disable_capture': True
}
driver = webdriver.Chrome(seleniumwire_options=options)

Selenium Wire stores captured requests/responses in a .seleniumwire sub-folder within the system temp directory by default. You can customize this behavior via seleniumwire_options:

• request_storage_base_dir: Set a custom base directory for the .seleniumwire folder.
• request_storage: Set to 'memory' to store requests in RAM instead of on disk. This is useful for short-lived environments like Docker.
• request_storage_max_size: When using 'memory' mode, use this to limit the number of requests stored. Once the limit is reached, older requests are discarded.

# Custom disk storage
options = {
    'request_storage_base_dir': '/my/storage/folder'
}
driver = webdriver.Chrome(seleniumwire_options=options)

# In-memory storage with size limit
options = {
    'request_storage': 'memory',
    'request_storage_max_size': 100
}
driver = webdriver.Chrome(seleniumwire_options=options)