tls-client

repository·master·Indexed 23 days ago

https://github.com/bogdanfinn/tls-client

A specialized HTTP client library designed to bypass anti-bot detection systems by mimicking the TLS and HTTP/3 fingerprints of real web browsers (e.g., Chrome, Firefox, Safari). It supports HTTP/1.1, HTTP/2, and HTTP/3 with automatic negotiation, custom header ordering, WebSocket support, and certificate pinning. The library is available in Go, JavaScript (Node.js), Python, and C# via FFI.

Tokens
8.2K
Snippets
14
Records
49
Agent score
82%

What's inside tls-client

  1. What is TLS-Client and why use it?

    master

    TLS-Client is an HTTP client library designed to bypass TLS fingerprinting. While changing the User-Agent header can mimic a browser, many modern servers use TLS fingerprinting to detect the underlying client. TLS-Client allows you to specify a specific browser and version (e.g., Chrome, Firefox, Safari) to mimic their TLS and HTTP/3 fingerprints accurately.

    Key features include:

    • Protocol Support: HTTP/1.1, HTTP/2, and HTTP/3 with automatic negotiation and 'Happy Eyeballs' protocol racing.
    • Fingerprinting: Mimics browser TLS and HTTP/3 (QUIC) fingerprints.
    • Advanced Control: Custom header ordering, WebSocket support, and certificate pinning.
    • Connectivity: Support for HTTP and SOCKS5 proxies.
    • Management: Built-in cookie jar management and bandwidth tracking.
    • Cross-Language: Available in Go, JavaScript (Node.js), Python, and C# via FFI.
  2. How to use TLS-Client in Go

    master

    To use TLS-Client in Go, you initialize a CookieJar, define HttpClientOptions (such as timeouts, client profiles, and redirect settings), and then create the client using tls_client.NewHttpClient. You can then use the client to perform standard HTTP operations like Do, Get, Post, or Head using requests from the github.com/bogdanfinn/fhttp package.

    package main
    
    import (
    	"fmt"
    	"io"
    	"log"
    
    	http "github.com/bogdanfinn/fhttp"
    	ls_client "github.com/bogdanfinn/tls-client"
    	"github.com/bogdanfinn/tls-client/profiles"
    )
    
    func main() {
    	jar := tls_client.NewCookieJar()
    	options := []tls_client.HttpClientOption{
    		tls_client.WithTimeoutSeconds(30),
    		tls_client.WithClientProfile(profiles.Chrome_144),
    		tls_client.WithNotFollowRedirects(),
    		tls_client.WithCookieJar(jar), // create cookieJar instance and pass it as argument
    	}
    
    	client, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(), options...)
    	if err != nil {
    		log.Println(err)
    		return
    	}
    
    	req, err := http.NewRequest(http.MethodGet, "https://tls.peet.ws/api/all", nil)
    	if err != nil {
    		log.Println(err)
    		return
    	}
    
    	req.Header = http.Header{
    		"accept":                    {"*/*"},
    		"accept-language":           {"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"},
    		"user-agent":                {"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"},
    		http.HeaderOrderKey: {
    			"accept",
    			"accept-language",
    			"user-agent",
    		},
    	}
    
    	resp, err := client.Do(req)
    	if err != nil {
    		log.Println(err)
    		return
    	}
    
    	defer resp.Body.Close()
    
    	log.Println(fmt.Sprintf("status code: %d", resp.StatusCode))
    
    	readBytes, err := io.ReadAll(resp.Body)
    	if err != nil {
    		log.Println(err)
    		return
    	}
    
    	log.Println(string(readBytes))
    }
  3. Configure the HTTP client using HttpClientOption

    master
    The tls-client library uses a functional options pattern to configure the HTTP client. You can pass multiple HttpClientOption functions to your client constructor to set up proxies, timeouts, TLS profiles, hooks, and more. Each option modifies a central httpClientConfig object.
  4. Manage sessions and cookies

    master

    The client supports session-based persistence. You can retrieve cookies from an active session or destroy a session to free resources.

    Retrieve Cookies Use getCookiesFromSession or getCookiesFromSessionAsync with a TLSClientFetchCookiesForSessionRequestPayload containing the sessionId and the target url. This returns a TLSClientFetchCookiesForSessionResponse containing an array of Cookie objects.

    Destroy Session Use destroySession or destroySessionAsync with a TLSClientReleaseSessionPayload containing the sessionId. This returns a TLSClientReleaseSessionResponse indicating success or failure.

  5. How JA3 fingerprinting and ClientHelloSpec work together

    master

    To bypass TLS fingerprinting, you must match a target client's TLS handshake characteristics. This library provides a way to translate a standard JA3 string into a tls.ClientHelloSpec used by the utls engine.

    The Process

    1. JA3 String Parsing: The JA3 string is split into components: Ciphers, Extensions, Curves, and Point Formats.
    2. Parameter Mapping: Since a JA3 string often only contains IDs, you must provide additional context (like supportedSignatureAlgorithms or supportedVersions) to fully reconstruct the ClientHelloSpec required for a valid handshake.
    3. Extension Construction: The factory maps the parsed IDs and your provided parameters into specific tls.TLSExtension implementations (e.g., tls.SignatureAlgorithmsExtension, tls.KeyShareExtension, tls.ALPNExtension).
    4. Spec Generation: The resulting ClientHelloSpec contains the CipherSuites, Extensions, and GetSessionID function needed to impersonate the target client.
  6. Inject custom headers into CONNECT requests via Context

    master

    When using a connectDialer (returned by newConnectDialer), you can inject dynamic headers into the outgoing CONNECT request by using context.WithValue.

    To do this, wrap an http.Header in a context using the ContextKeyHeader{} type as the key. The connectDialer will inspect the context during DialContext, and these headers will override any colliding headers in the dialer's DefaultHeader.

  7. HttpClient Interface Reference

    master

    The HttpClient interface extends the standard Go net/http client with specialized methods for cookie management, proxy configuration, and fingerprinting-related access. It is the primary interface used to interact with the library.

    type HttpClient interface {
        GetCookies(u *url.URL) []*http.Cookie
        SetCookies(u *url.URL, cookies []*http.Cookie)
        SetCookieJar(jar http.CookieJar)
        GetCookieJar() http.CookieJar
        SetProxy(proxyUrl string) error
        GetProxy() string
        SetFollowRedirect(followRedirect bool)
        GetFollowRedirect() bool
        CloseIdleConnections()
        Do(req *http.Request) (*http.Response, error)
        Get(url string) (resp *http.Response, err error)
        Head(url string) (resp *http.Response, err error)
        Post(url, contentType string, body io.Reader) (resp *http.Response, err error)
    
        GetBandwidthTracker() bandwidth.BandwidthTracker
        GetDialer() proxy.ContextDialer
        GetTLSDialer() TLSDialerFunc
    }
  8. Configure a TLSClientRequestPayload

    master

    The TLSClientRequestPayload object defines the parameters for an HTTP request. It allows for fine-grained control over TLS fingerprinting, proxy settings, and HTTP behavior.

    Key configuration options:

    • requestUrl: The target URL.
    • requestMethod: One of 'GET' | 'POST' | 'PATCH' | 'PUT' | 'DELETE'.
    • requestBody: The string representation of the request body.
    • requestCookies: An optional array of cookie objects [{ [key: string]: string }].
    • tlsClientIdentifier: An optional identifier for a pre-defined TLS fingerprint (e.g., 'chrome_103').
    • followRedirects: Boolean to control redirect behavior.
    • insecureSkipVerify: Boolean to skip TLS certificate verification.
    • isByteResponse: Boolean to indicate if the response should be treated as bytes.
    • withoutCookieJar: Boolean to bypass the internal cookie jar.
    • withRandomTLSExtensionOrder: Boolean to randomize TLS extension order.
    • timeoutSeconds: Number of seconds before the request times out.
    • sessionId: The ID of an existing session to reuse.
    • proxyUrl: The proxy URL to use for the request.
    • headers: An object containing custom HTTP headers { [key: string]: string }.
    • headerOrder: An array of strings defining the order of headers.
    • customTlsClient: An object for providing a manual, highly specific TLS configuration (JA3, H2 settings, etc.).
  9. Configure HTTP/2 connection reuse in connectDialer

    master
    The connectDialer supports connection reuse for HTTP/2. When EnableH2ConnReuse is set to true, the dialer will attempt to reuse an existing cached HTTP/2 connection if it is still valid (CanTakeNewRequest). This reduces the overhead of repeated CONNECT requests over HTTP/2.
  10. Configure TLS and Network behavior

    master

    The client provides fine-grained control over the TLS handshake and network layer:

    • WithClientProfile(clientProfile profiles.ClientProfile): Sets the TLS fingerprint/profile (e.g., Chrome, Firefox).
    • WithRandomTLSExtensionOrder(): Randomizes the order of TLS extensions in the ClientHello.
    • WithInsecureSkipVerify(): Skips SSL certificate verification.
    • WithCertificatePinning(certificatePins map[string][]string, handlerFunc BadPinHandlerFunc): Enables SSL Pinning. certificatePins is a map where the key is the host and the value is a slice of valid pins. handlerFunc is called if a bad pin is detected.
    • WithServerNameOverwrite(serverName string): Overwrites the server name used for certificate verification and ClientHello (requires WithInsecureSkipVerify to work properly).
    • WithForceHttp1(): Forces the client to use HTTP/1.1.
    • WithDisableHttp3(): Disables HTTP/3 (falls back to HTTP/2).
    • WithProtocolRacing(): Races HTTP/3 (QUIC) and HTTP/2 (TCP) connections simultaneously, using the one that connects first.
    • WithDisableIPV4() / WithDisableIPV6(): Forces the connection to use only IPv6 or IPv4 respectively.
  11. Configure Request Timeouts and Redirects

    master

    Control the lifecycle of requests with these options:

    • WithTimeoutSeconds(timeout int): Sets a hard deadline for the entire request lifecycle in seconds.
    • WithTimeoutMilliseconds(timeout int): Sets a hard deadline in milliseconds.
    • WithNotFollowRedirects(): Disables automatic following of HTTP redirects.
    • WithCustomRedirectFunc(redirectFunc func(req *http.Request, via []*http.Request) error): Provides a custom function to handle redirects, similar to net/http's CheckRedirect.
  12. Implement http.RoundTripper with tls_client

    master

    The roundTripper struct implements the standard Go http.RoundTripper interface. It manages connection pooling, TLS fingerprinting via utls, and protocol negotiation (HTTP/1.1, HTTP/2, and HTTP/3). It automatically selects the appropriate transport based on the negotiated protocol during the TLS handshake.

    Key capabilities include:

    • Protocol Racing: If enabled, it can race HTTP/3 against other protocols to find the fastest connection.
    • Certificate Pinning: Supports validating server certificates against provided pins.
    • Custom Dialing: Allows providing a custom proxy.ContextDialer for proxy support or custom network logic.
    • Connection Management: Provides CloseIdleConnections() to clean up cached transports.