thermoptic

repository·main·Indexed 21 days ago

https://github.com/mandatoryprogrammer/thermoptic

An HTTP proxy (version 1.0.0) that cloaks request JA4+ fingerprints (JA4, JA4H, JA4X, JA4T) by replaying requests through a real Chrome/Chromium browser instance via the Chrome Debugging Protocol (CDP). It allows HTTP clients like curl or python-requests to produce fingerprints identical to a real browser, featuring a hook framework for JavaScript fingerprinting mitigation and an Xpra Web UI for manual browser interaction.

Tokens
6.9K
Snippets
10
Records
35
Agent score
76%

What's inside thermoptic

  1. What is thermoptic?

    main
    thermoptic is an HTTP proxy designed to bypass fingerprinting-based blocking (such as JA4+) by replaying requests through a real Chrome/Chromium browser session. This ensures that your HTTP client (e.g., curl, requests) produces fingerprints that are byte-for-byte identical to a real web browser. It also provides a hook framework to mitigate JavaScript-based fingerprinting and supports hybrid scraping workflows where you can combine low-level HTTP clients with a real browser.
  2. Use lifecycle hooks to extend thermoptic

    main

    You can inject custom Node.js logic into the thermoptic lifecycle using file paths provided via environment variables. This allows you to automate browser interactions or manipulate requests.

    • ON_START_HOOK_FILE_PATH: Runs once when the proxy starts. The proxy will block listening until this script finishes. Use this for tasks like clicking through Cloudflare JavaScript challenges.
    • BEFORE_REQUEST_HOOK_FILE_PATH: Runs before a request is proxied. Use this to ensure the browser passes specific checks before an HTTP request is made.
    • AFTER_REQUEST_HOOK_FILE_PATH: Runs after a request is proxied. Use this for post-request cleanup, such as removing specific cookies set by the client via the Cookie header.
  3. Understand the risks of Chrome Debugging Protocol (CDP) dependencies

    main
    thermoptic relies heavily on Chrome and the Chrome Debugging Protocol (CDP). Because the CDP has certain limitations—such as the inability to perform a Page.navigate while simultaneously catching request details with Fetch.requestPausedthermoptic employs internal workarounds (hacks) to mimic direct user navigation. These workarounds carry a risk of breakage if Chrome's internal behavior or the CDP implementation changes in future versions.
  4. How thermoptic handles cookies

    main

    thermoptic loads the browser with the cookies specified in your client's Cookie header. This ensures that the server cannot fingerprint the user based on cookie ordering or other metadata.

    Warning: These cookies persist in the browser context after the request is executed. If you require cookie cleanup, you must implement a custom thermoptic hook.

  5. Use thermoptic hooks for JavaScript fingerprinting mitigation

    main

    To mitigate JavaScript-based fingerprinting (common in WAFs like Cloudflare), thermoptic provides a hook framework. This allows you to automate the browser to solve challenges or capture artifacts before/after requests.

    Supported hook types include:

    • before-request
    • after-request
    • on-start

    An example of a Cloudflare Turnstile solving hook can be found in the repository's ./hooks/onstart.js file.

  6. Avoid the Runtime API when automating browsers

    main

    When writing automation for anti-bot services, avoid using the Chrome Debugging Protocol (CDP) Runtime API (e.g., Runtime.enable).

    Why? Using the Runtime API to inject or execute JavaScript can trigger an object-serialization nuance that makes your scraping immediately detectable to advanced anti-bot services like Cloudflare. Instead, use the DOM and Input APIs to interact with the page as a user would.

  7. How thermoptic cloaking works

    main

    Thermoptic achieves browser-parity by using the Chrome Debugging Protocol (CDP) to puppet a real browser instance. The process follows these steps:

    1. An HTTP request is sent to the thermoptic proxy from a client (e.g., curl).
    2. thermoptic analyzes the request to determine the intended browser behavior (e.g., manual URL visit, form submission, or fetch()).
    3. thermoptic uses CDP to command the browser to mock the request exactly as it would occur in a real session.
    4. The request is executed by the browser's full network stack (TCP, TLS, HTTP).
    5. thermoptic captures the browser's HTTP response and returns it to the original client.

    Because the request is made by the actual browser, the resulting JA4+ fingerprints are indistinguishable from a real user.

  8. Configure the Cloudflare Turnstile hook in docker-compose.yml

    main

    To enable a custom hook (like the Turnstile bypass) in a thermoptic Docker deployment, set the ON_START_HOOK_FILE_PATH environment variable to the path of your hook file within the container. Ensure the hook file is mounted via a volume.

    thermoptic:
      build: .
      volumes:
        - ./ssl:/work/cassl
        - ./hooks:/work/hooks
      depends_on:
        - chrome
      ports:
        - 1234:1234
      environment:
        HTTP_PROXY_PORT: 1234
        CHROME_DEBUGGING_PORT: 3003
        CHROME_DEBUGGING_HOST: chrome
        PROXY_USERNAME: changeme
        PROXY_PASSWORD: changeme
        # Path to your custom hook file
        ON_START_HOOK_FILE_PATH: /work/hooks/onstart.js
  9. Use thermoptic hooks for custom browser automation

    main

    To handle advanced JavaScript fingerprinting or manual verification steps (like CAPTCHAs), you can use thermoptic hooks. These allow you to execute custom JavaScript using the Chrome Debugging Protocol (CDP) at specific lifecycle stages.

    Available hook stages via environment variables:

    • ON_START_HOOK_FILE_PATH: Executed when the browser first starts.
    • BEFORE_REQUEST_HOOK_FILE_PATH: Executed before a request is proxied.
    • AFTER_REQUEST_HOOK_FILE_PATH: Executed after a request has finished being proxied.

    Each hook receives a cdp object, which is an instance of a connected browser via the chrome-remote-interface interface.

    // `cdp` is an instance of a connected browser, use it to run your browser actions
    export async function hook(cdp) {
        console.log(`[STATUS] Browser start hook called successfully!`);
    }
  10. Security considerations for thermoptic

    main

    Usage Warning

    thermoptic is designed to be used only with HTTP clients that you explicitly trust. Do not expose the proxy to untrusted users.

    Vulnerability Reporting

    If you discover a security vulnerability, please report it to mandatory@ Gmail.

  11. Control the Dockerized Chrome browser via Xpra Web UI

    main

    You can manually interact with the Dockerized Chrome browser using the Xpra web interface available at http://127.0.0.1:14111. This is useful for:

    • Authentication: Logging into websites (e.g., Reddit) manually so that subsequent requests made via thermoptic (using curl or other clients) are automatically authenticated.
    • Debugging: Testing your custom thermoptic hooks and observing how websites behave.
  12. Bypass Cloudflare Turnstile using thermoptic hooks

    main

    To bypass Cloudflare Turnstile CAPTCHAs, you can write a thermoptic hook that runs on proxy start. This hook uses the Chrome Debugging Protocol (CDP) to interact with the page directly, avoiding the detection risks associated with high-level automation frameworks like Puppeteer or using the Runtime API (which can trigger object-serialization detection).

    Instead of injecting JavaScript via Runtime.evaluate, use the Target, Page, DOM, and Input CDP APIs to find the CAPTCHA element and simulate a human-like mouse click.

    Key Strategy:

    1. Use DOM.querySelectorAll to find candidate elements.
    2. Use DOM.getAttributes to identify the Cloudflare container (e.g., looking for specific style attributes).
    3. Use DOM.getBoxModel to calculate the element's coordinates.
    4. Use Input.dispatchMouseEvent with mousePressed and mouseReleased to perform the click.
    5. Crucial: Add 'fuzziness' (random small offsets) to your click coordinates and timing to avoid appearing robotic.
    // Example logic for finding and clicking the Turnstile checkbox
    const { nodeIds: divNodeIds } = await DOM.querySelectorAll({
        nodeId: documentNodeId,
        selector: 'div'
    });
    
    // ... logic to find targetNodeId via attributes ...
    
    const { model } = await DOM.getBoxModel({ nodeId: targetNodeId });
    const click_x = (x_top_left + 25) + x_fuzz;
    const click_y = ((y_top_left + y_bottom_left) / 2) + y_fuzz;
    
    await Input.dispatchMouseEvent({
        type: 'mousePressed',
        x: click_x,
        y: click_y,
        button: 'left',
        clickCount: 1
    });
    
    await Input.dispatchMouseEvent({
        type: 'mouseReleased',
        x: click_x,
        y: click_y,
        button: 'left',
        clickCount: 1
    });