TLS Requests

repository·main·Indexed 41 days ago

https://github.com/thewebscraping/tls-requests

A high-performance Python library for making secure, browser-like HTTP/TLS fingerprint requests. It is designed to bypass anti-bot protections like Cloudflare Bot Fight Mode by mimicking browser TLS fingerprints and synchronizing headers. Features include support for HTTP/1.1, HTTP/2, and HTTP/3, an AsyncClient for high-concurrency workloads, proxy rotation via ProxyRotator, and flexible authentication schemes.

Tokens
23.4K
Snippets
83
Records
96
Agent score
49%

What's inside TLS Requests

  1. Overview of TLS Requests capabilities

    main

    TLS Requests is a high-performance Python HTTP client built on top of tls-client. It is designed to mimic browser TLS fingerprints to bypass anti-bot systems.

    Core Capabilities:

    • Browser-like TLS Fingerprinting: Mimics real browsers to avoid detection.
    • Dynamic Header Synchronization: Automatically aligns User-Agent and sec-ch-ua with the client_identifier.
    • High Performance: Uses a Go-based backend with Protocol Racing (Happy Eyeballs) enabled by default.
    • Protocol Support: Supports HTTP/1.1, HTTP/2, and HTTP/3 (Alpha).
    • Concurrency: Supports both synchronous and asynchronous request patterns.

    Additional Features:

    • Proxy Support: Handles Socks5, HTTP, and HTTPS proxies.
    • Session Management: Built-in cookie management.
    • Authentication: Supports Basic and Function authentication.
    • Content Handling: Automatic decoding for gzip and brotli.
    • Advanced Options: Supports protocol_racing, allow_http, and stream_id.
  2. How event hooks work in tls_requests

    main

    Event hooks allow you to execute custom logic during the HTTP request/response lifecycle. They are useful for logging, monitoring, tracing, or pre/post-processing data.

    There are two types of hooks:

    1. Request Hook: Executed after the request is fully prepared but before it is sent to the network. It receives the request object as its only argument.
    2. Response Hook: Triggered after the response is received from the network but before it is returned to the caller. It receives the response object.

    Key Rules for Hooks:

    • Always Use Lists: Hooks must be registered as lists of callables, even if you are only adding a single function.
    • Order Matters: Hooks are executed in the order they are registered.
    • Accessing Content: When using a response hook, use .read() (or await .aread() in async contexts) to access response.content before the hook returns.
    import tls_requests
    
    def log_request(request):
        print(f"Request event: {request.method} {request.url}")
    
    def log_response(response):
        print(f"Response event: {response.status_code} for {response.url}")
    
    client = tls_requests.Client(hooks={
        'request': [log_request],
        'response': [log_response]
    })
  3. Use the Client class for persistent sessions

    main

    The tls_requests.Client class is the primary interface for making synchronous HTTP requests. It is functionally equivalent to requests.Session from the requests library.

    Using a Client is recommended over top-level functions (like tls_requests.get()) because it:

    • Performance: Reuses underlying TLS sessions and network connections.
    • State Management: Automatically manages cookies and authentication across multiple requests.
    • Consistency: Applies shared headers, proxies, and timeouts to every request made with that client instance.
    import tls_requests
    
    with tls_requests.Client() as client:
        response = client.get("https://httpbin.org/get")
        print(response.status_code)
  4. Merge headers and cookies between Client and Request levels

    main

    When providing headers or cookies at both the Client level and the individual request level, they are merged. If a conflict exists, the request-level values override the client-level values.

    import tls_requests
    
    client_headers = {'X-Auth': 'client'}
    request_headers = {'X-Custom': 'request'}
    
    with tls_requests.Client(headers=client_headers) as client:
        response = client.get("https://httpbin.org/get", headers=request_headers)
        # Resulting headers will contain both 'X-Auth': 'client' and 'X-Custom': 'request'
        print(response.request.headers['X-Auth'])  # 'client'
        print(response.request.headers['X-Custom'])  # 'request'
  5. Implement custom authentication by subclassing Auth

    main

    For complex authentication schemes, subclass tls_requests.Auth and implement the build_auth method. The build_auth method must accept a tls_requests.Request object and return the modified tls_requests.Request object. This is the recommended way to create reusable authentication classes.

    import tls_requests
    
    class BearerAuth(tls_requests.Auth):
        def __init__(self, token):
            self.token = token
    
        def build_auth(self, request: tls_requests.Request) -> tls_requests.Request:
            request.headers["Authorization"] = f"Bearer {self.token}"
            return request
  6. Emulate browser TLS fingerprints with client_identifier

    main

    You can emulate specific browser TLS fingerprints by providing a client_identifier to the request methods. This spoofing is handled automatically at the protocol level, allowing you to bypass fingerprint-based detection.

    By default, tls_requests uses the chrome_133 profile. If no identifier is provided, the library uses this default configuration.

    import tls_requests
    
    # Using the default profile (chrome_133)
    response = tls_requests.get("https://httpbin.org/get")
    
    # Using a specific profile
    response = tls_requests.get("https://httpbin.org/get", client_identifier="firefox_132")
  7. Proxy usage considerations

    main

    When working with proxies in tls_requests, keep the following in mind:

    • Global vs. Per-Request: Proxies are typically set on the Client instance, but you can also pass a proxy argument to individual request methods (e.g., client.get(url, proxy=...)) to override the client default.
    • HTTPS Support: Both HTTP and SOCKS5 proxies correctly handle HTTPS traffic (HTTP proxies use the CONNECT method).
    • Format: The proxy parameter expects a valid string. If you use a Proxy object from tls_requests.models, it will be automatically converted to the required string format.
  8. Manually download TLS binaries

    main

    If your environment has restricted internet access or you require a specific version of the underlying tls-client library, you can trigger the download manually using TLSLibrary.download(). This is useful for ensuring the binary is present before your main application logic runs.

    from tls_requests import TLSLibrary
    
    # Download a specific version
    TLSLibrary.download(version='1.13.1')
  9. Use AsyncClient for asynchronous HTTP requests

    main

    The AsyncClient provides full support for asynchronous HTTP requests, making it suitable for high-concurrency workloads and integration with frameworks like FastAPI. It mirrors the synchronous Client API but requires the await keyword for all network operations.

    To use it, wrap your logic in an async function and use the async with context manager to ensure TLS sessions are properly closed.

    import asyncio
    import tls_requests
    
    async def main():
        async with tls_requests.AsyncClient() as client:
            response = await client.get("https://httpbin.org/get")
            print(f"Status: {response.status_code}")
            print(f"Data: {response.json()}")
    
    if __name__ == "__main__":
        asyncio.run(main())
  10. Rotate proxies with ProxyRotator

    main

    Proxy rotation is not enabled by default. You must provide a list of proxies to the proxy parameter to enable it.

    Proxy Rotation Strategies

    • Weighted Strategy (Default): When you pass a list of proxy strings directly to the Client, the library uses a weighted strategy. The Client automatically provides performance feedback (success/failure, latency) to the rotator, prioritizing proxies that perform well.
    • Custom Strategy: You can use ProxyRotator.from_file(proxy_list, strategy="...") to specify a different strategy, such as round_robin.

    Proxy String Formats

    • Standard: http://user:pass@host:port or socks5://host:port
    • Weighted/Regional: http://user:pass@host:port|weight|region (e.g., http://user:pass@proxy.example.com:8084|1.0|US)
    import tls_requests
    
    proxy_list = [
        "http://user1:pass1@proxy.example.com:8080",
        "http://user2:pass2@proxy.example.com:8081",
        "socks5://proxy.example.com:8082",
        "proxy.example.com:8083",  # (defaults to http)
        "http://user:pass@proxy.example.com:8084|1.0|US",  # http://user:pass@host:port|weight|region
    ]
    
    # Provide a list to enable proxy rotation.
    with tls_requests.Client(proxy=proxy_list) as client:
        response = client.get("https://httpbin.org/get")
  11. Register hooks during Client initialization

    main

    You can define hooks at the moment of creation by passing a hooks dictionary to the tls_requests.Client or tls_requests.AsyncClient constructor. The dictionary keys must be 'request' or 'response', and the values must be lists of callable functions.

    import tls_requests
    
    def log_request(request):
        pass
    
    def log_response(response):
        pass
    
    def raise_on_4xx_5xx(response):
        response.raise_for_status()
    
    client = tls_requests.Client(hooks={
        'request': [log_request],
        'response': [log_response, raise_on_4xx_5xx],
    })
  12. Dynamically update hooks on an existing client

    main

    You can modify hooks after a client has been initialized by accessing the .hooks property. You can add specific event hooks, update them, or replace the entire dictionary.

    import tls_requests
    
    def log_request(request):
        pass
    
    def log_response(response):
        pass
    
    def raise_on_4xx_5xx(response):
        response.raise_for_status()
    
    client = tls_requests.Client()
    
    # Add a request hook
    client.hooks['request'] = [log_request]
    
    # Add a response hook
    client.hooks['response'] = [log_response]
    
    # Completely replace hooks
    client.hooks = {
        'request': [log_request],
        'response': [raise_on_4xx_5xx],
    }