httpcloak

repository·main·Indexed 22 days ago

https://github.com/sardanioss/httpcloak

A multi-language library designed to make HTTP requests indistinguishable from real browser traffic. It emulates the entire fingerprinting surface, including TLS handshakes, HTTP/2 and HTTP/3 transport parameters, and complex header ordering to bypass advanced bot detection. Features include browser presets (Chrome, Firefox, Safari), Encrypted Client Hello (ECH) support, proxy configuration (HTTP, SOCKS5, MASQUE), and a .NET implementation providing Session and HttpCloakHandler for HttpClient integration.

Tokens
164.4K
Snippets
423
Records
643
Agent score
77%

What's inside httpcloak

  1. Understand HTTP protocol support in httpcloak

    main

    httpcloak supports HTTP/1.1, HTTP/2, and HTTP/3 (QUIC). The library includes logic to automatically negotiate the best available protocol or allows you to force a specific version.

    • HTTP/1.1: Used when forced or when negotiation fails.
    • HTTP/2: The default for most modern hosts, utilizing SETTINGS and PRIORITY frames on the wire.
    • HTTP/3 (QUIC): Implemented via sardanioss/quic-go, running over UDP and supporting 0-RTT (Zero Round-Trip Time) for faster connections.
  2. Available language bindings for httpcloak

    main

    httpcloak provides consistent wire behavior across four different language surfaces by using a shared cgo-built library. While Go is the native implementation, Python, Node.js, and .NET use wrappers that call into this same core library to ensure identical HTTP/TLS/Header emulation.

    • Go: The native implementation using idiomatic Go.
    • Python: A wrapper designed to mimic the requests library API.
    • Node.js: A wrapper backed by koffi that supports both ESM and CommonJS.
    • .NET: A P/Invoke wrapper compatible with .NET 8+.
  3. What is an httpcloak preset?

    main

    A preset is a complete fingerprint bundle for a specific browser version on a specific platform. When you select a preset, httpcloak emulates the following layers to match a real browser's wire signature:

    • TLS Layer: ClientHello (cipher list, extension list, supported groups, signature algorithms, ALPN, cert compression).
    • HTTP/2 Layer: SETTINGS values, WINDOW_UPDATE, and pseudo-header order.
    • Header Layer: Default HTTP headers in the exact order used by Chrome, Firefox, or Safari.
    • Priority Layer: RFC 7540 stream priorities and the RFC 9218 priority table per Sec-Fetch-Dest.
    • HTTP/3 / QUIC Layer: Transport parameters (only on supported presets).
    • TCP/IP Layer: Fingerprint hints like TTL, MSS, and window size for OS-level matching.

    By picking a preset by name, your outgoing requests will match the wire bytes of the targeted browser.

  4. Overview of Advanced TLS features

    main

    For specialized networking requirements, httpcloak provides several advanced TLS knobs:

    • ECH (Encrypted Client Hello): Enabled by default; use WithDisableECH to opt out.
    • Speculative TLS: Pipelines CONNECT and ClientHello to save one RTT on proxied dials.
    • TLS Keylog: Dumps SSLKEYLOGFILE for inspection in Wireshark.
    • Domain Fronting: Configures SNI to differ from the Host header.
    • Cert Pinning: Pins a server's certificate or public key at the application layer.
    • Insecure Skip Verify: Skips certificate verification (use only for dev/MITM testing, never in production).
    • Session Cache: Allows plugging in a Redis (or other) backend for distributed TLS ticket resumption.
  5. Create a patch preset using `based_on`

    main

    Instead of a full dump-and-edit, you can create a lightweight JSON patch that inherits from an existing preset. This is ideal for simple modifications like updating a User-Agent while keeping all other TLS and HTTP/2 settings identical.

    Note: Use the full describe -> mutate -> load flow if you need to override a field to its 'zero value' (e.g., clearing a header). In a based_on patch, setting a field to its zero value is treated as if the field was not set at all, meaning it will continue to inherit the value from the parent.

    Example Patch JSON

    {
      "version": 1,
      "preset": {
        "name": "my-chrome-mutant",
        "based_on": "chrome-148-windows",
        "headers": {
          "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/200.0.0.0 Safari/537.36"
        }
      }
    }
  6. Concurrency and Threading in HTTPCloak

    main

    HTTPCloak is designed for high-concurrency environments with the following characteristics:

    • Thread Safety: The Session object is goroutine-safe. Internal state like the cookie jar, transport map, and TLS cache are protected by mutexes.
    • Session Forking: Using Fork(n) returns new sessions that share the same cookie jar mutex as the parent. While concurrent requests across forks will contend for the cookie jar, they will use independent network connections.
    • Context Propagation: Per-request context.Context is propagated down to the transport layer. Canceling the context will cancel in-flight reads, handshakes, or dials.
  7. How to handle field order in Multipart requests

    main

    Multipart requests preserve the order in which fields are listed. While most servers parse fields into a map and ignore order, some strict servers require specific ordering:

    • CSRF/Tokens: Some servers require security tokens (like CSRF) to appear before file uploads.
    • Validation: Some servers expect a specific sequence defined in their form validation logic.

    Troubleshooting Tip: If you receive "field missing" or "invalid form" errors despite the data being correct, try reordering your fields. A common pattern is to place text fields (like tokens) first and the file upload last.

  8. Use Source-IP binding with SOCKS5

    main

    You can use WithLocalAddress in conjunction with SOCKS5. This binds the local socket on your machine to a specific local IP address before connecting to the SOCKS5 server.

    Note: This does not change the egress IP seen by the target website; the proxy still uses its own IP for the upstream connection. This is useful only if your machine has multiple public IPs and you want to control which one initiates the connection to the proxy server.

  9. Optimize proxy latency with Speculative TLS

    main

    By default, an HTTP CONNECT proxy requires two round-trips (TCP handshake + CONNECT exchange) before the TLS handshake begins.

    httpcloak can reduce this by one full round-trip by pipelining the CONNECT request with the inner ClientHello. This is achieved using WithEnableSpeculativeTLS().

    Warning: This is disabled by default because some older proxies (e.g., certain Squid configurations) may fail if they receive bytes before the 200 Connection established response is fully read. Test this with your specific provider before deploying.

    s := httpcloak.NewSession("chrome-latest",
        httpcloak.WithSessionTCPProxy("http://user:pass@proxy.example.com:8080"),
        httpcloak.WithEnableSpeculativeTLS(),
    )
  10. Compare Save/Load vs Marshal/Unmarshal

    main

    Choose the method based on your storage target:

    MethodTargetUse Case
    Save(path) / LoadSession(path)Local FilesystemLong-running scrapers, daemons, or CLI tools that benefit from a single file on disk.
    Marshal() / UnmarshalSession(data)Non-filesystem (DB, Network, Memory)Storing in a database column, shipping to workers via gRPC/Kafka, or using in read-only containers.

    Note: Save automatically applies 0600 file permissions. Marshal provides raw bytes/strings, leaving encryption and storage responsibility to the caller.