impit

repository·master·Indexed 19 days ago

https://github.com/apify/impit

A library for browser impersonation that allows developers to make web requests appearing to originate from real browsers. It supports HTTP/1.1, HTTP/2, and HTTP/3 using patched networking libraries. impit provides a fetch-compatible API for Node.js and a builder-pattern API for Rust, supporting the emulation of browsers like Chrome and Firefox, TLS fingerprint switching, and custom cookie management.

Tokens
17.9K
Snippets
56
Records
77
Agent score
65%

What's inside impit

  1. Install the `impit` Node.js package

    master

    Install impit using npm. The root package automatically detects your platform and installs the appropriate prebuilt binary (supporting Linux, macOS, and Windows across x86_64 and arm64 architectures).

    npm install impit
  2. Use `impit` to impersonate a browser

    master

    The impit module allows you to switch TLS fingerprints and HTTP headers to impersonate a specific browser while maintaining a fetch-compatible API.

    To use it, instantiate the Impit class with a configuration object and call the .fetch() method. The .fetch() method behaves identically to the built-in Node.js fetch function.

    import { Impit } from 'impit';
    
    // Set up the Impit instance
    const impit = new Impit({
        browser: "chrome", // or "firefox"
        proxyUrl: "http://localhost:8080",
        ignoreTlsErrors: true,
    });
    
    // Use the `fetch` method as you would with the built-in `fetch` function
    const response = await impit.fetch("https://example.com");
    
    console.log(response.status);
    console.log(response.headers);
    console.log(await response.text());
  3. Install impit in a Rust project

    master

    To use impit in a Rust project, you must add it as a dependency and patch rustls and h2 to use the versions provided by the impit repository. Without these patches, the project will not build.

    Additionally, you must build your project with the specific rustflag --cfg reqwest_unstable because impit utilizes unstable reqwest features, such as HTTP/3 support.

    [dependencies]
    impit = { git="https://github.com/apify/impit.git", branch="master" }
    
    [patch.crates-io]
    rustls = { git="https://github.com/apify/rustls.git" }
    h2 = { git="https://github.com/apify/h2.git" }

    Build command requirement:

    rustflags = "--cfg reqwest_unstable"

  4. Use the impit Python module

    master

    The impit Python module provides high-level clients and standalone functions for browser impersonation. You can use the Client or AsyncClient classes for managed sessions, or use standalone HTTP method functions for one-off requests.

    Core Classes

    • Client: Synchronous client for managing requests.
    • AsyncClient: Asynchronous client for managing requests.
    • ImpitPyResponse: The response object returned by requests.

    Standalone HTTP Methods

    You can perform requests without manually instantiating a client using the following functions:

    • get(url, ...)
    • post(url, ...)
    • put(url, ...)
    • head(url, ...)
    • patch(url, ...)
    • delete(url, ...)
    • options(url, ...)
    • trace(url, ...)
    • stream(method, url, ...): For streaming response content.

    Common Request Parameters

    All request functions and methods accept the following optional arguments:

    • content: bytes (raw content)
    • data: Request body data
    • headers: dict[str, str]
    • timeout: float or str (e.g., '10s'). Defaults to USE_CLIENT_DEFAULT.
    • force_http3: bool
    • cookie_jar: A Python object representing a cookie jar.
    • cookies: dict of cookies.
    • follow_redirects: bool
    • max_redirects: int
    • proxy: str (proxy URL)

    To use the default timeout settings, use the USE_CLIENT_DEFAULT constant.

    import impit
    
    # Using a standalone GET request
    response = impit.get("https://example.com", timeout=5.0)
    print(response.status_code)
    
    # Using the Client class
    client = impit.Client(proxy="http://myproxy:8080")
    response = client.get("https://example.com")
    
    # Streaming a request
    stream = impit.stream("GET", "https://example.com/large-file")
    for chunk in stream:
        print(chunk)
  5. How HttpHeaders resolves header priority

    master

    When iterating over HttpHeaders, the library resolves headers using a specific priority and deduplication logic:

    1. Custom Headers First: Headers provided via with_custom_headers have the highest priority.
    2. Fingerprint Headers Second: If a BrowserFingerprint is present, its headers are added after custom headers.
    3. Deduplication: If a header name appears multiple times (e.g., once in custom headers and once in the fingerprint), only the first occurrence (the one with higher priority) is kept. The comparison is case-insensitive.
    4. Empty Values: Headers with empty string values are filtered out and not included in the final set.
  6. How AsyncClient handles cookies and redirects

    master

    AsyncClient manages stateful interactions through two primary mechanisms:

    You can initialize the client with a persistent cookie_jar or by providing an initial set of cookies. If a cookie_jar is provided, the client will use it to store and send cookies across requests. If cookies are provided, they are converted into a PythonCookieJar for the client's use.

    Redirect Behavior

    By default, AsyncClient uses ManualRedirect behavior, meaning it will not automatically follow redirects. To enable automatic following, set follow_redirects=True during initialization. You can control the depth of this behavior using the max_redirects parameter (defaults to 20).

  7. How Response state and streaming work

    master

    The Response object manages an internal state that determines how data is accessed:

    1. Unread: The response is in streaming mode. Data has not been fetched yet. Calling iter_bytes(), aiter_bytes(), read(), or content will transition the state to Read or Streaming.
    2. Read: The entire content has been loaded into memory. Accessing content or text is fast and returns cached data.
    3. Streaming: The response is currently being consumed via an iterator.
    4. StreamingClosed: The stream has been exhausted or closed.

    Important Lifecycle Rules

    • Single Consumption: Once a stream is consumed (via iteration or a full read), you cannot consume it again. Attempting to do so will raise an error (e.g., StreamConsumed).
    • Closing: Closing a response (via close() or aclose()) prevents further reading and transitions the state to StreamingClosed if it was streaming.
  8. Configure cookie management with a cookie jar

    master

    To enable automatic cookie handling in Impit, pass a cookieJar object in the constructor. The Impit class will automatically call getCookieString(url) before requests to include cookies in the headers, and call setCookie(cookie, url) when receiving Set-Cookie headers in responses.

    Required Cookie Jar Interface:

    • getCookieString(url: string): Promise<string>: Returns a cookie string for the given URL.
    • setCookie(cookie: string, url: string): Promise<void>: Saves a cookie for the given URL.
    const impit = new Impit({
        cookieJar: {
            getCookieString: async (url) => 'key=value',
            setCookie: async (cookie, url) => { /* implementation */ }
        }
    });
  9. Use the Impit class for browser impersonation

    master

    The Impit class is the primary interface for making HTTP requests with browser impersonation. One Impit instance represents a single user agent, meaning all requests made by that instance share the same configuration, resources (such as the cookie jar and connection pool), and settings.

    To use it, instantiate Impit and use the .fetch() method, which is designed to be API-compatible with the standard Web Fetch API.

    import { Impit } from 'impit';
    
    const impit = new Impit();
    const response = await impit.fetch('https://example.com');
    console.log(await response.text());
  10. Quickstart: Use impit in Rust

    master

    To use impit for browser impersonation, you need to initialize an Impit instance using a builder pattern. You can specify a browser fingerprint (e.g., from fingerprint::database) and enable features like HTTP/3. The Impit struct is generic over a cookie jar type (e.g., reqwest::cookie::Jar).

    use impit::{impit::Impit, fingerprint::database as fingerprints};
    use reqwest::cookie::Jar;
    
    #[tokio::main]
    async fn main() {
       let impit = Impit::<Jar>::builder()
           .with_fingerprint(fingerprints::firefox_144::fingerprint())
           .with_http3()
           .build()
           .unwrap();
    
       let response = impit.get(String::from("https://example.com"), None, None).await;
    
       match response {
           Ok(response) => {
               println!("{}", response.text().await.unwrap());
           }
           Err(e) => {
               println!("{:#?}", e);
           }
       }
    }