fingerprint-suite

repository·master·Indexed 25 days ago

https://github.com/apify/fingerprint-suite

A modular toolkit for browser fingerprint generation and injection designed to help web scrapers avoid detection. Version 2.1.86 includes packages such as header-generator for realistic HTTP headers, fingerprint-generator for consistent browser fingerprints, fingerprint-injector for Playwright and Puppeteer integration, and generative-bayesian-network for sampling from distribution definitions. It also includes apify_fingerprint_datapoints for accessing raw fingerprint datafiles.

Tokens
8.5K
Snippets
21
Records
59
Agent score
81%

What's inside fingerprint-suite

  1. Overview of fingerprint-suite packages

    master

    The fingerprint-suite is a modular toolkit designed for generating and injecting realistic browser fingerprints to help scrapers avoid detection. It is composed of several specialized npm packages:

    • header-generator: Generates configurable, realistic HTTP headers.
    • fingerprint-generator: Generates realistic browser fingerprints that affect both HTTP headers and browser JavaScript APIs.
    • fingerprint-injector: Injects browser fingerprints into managed browser instances (Playwright or Puppeteer).
    • generative-bayesian-network: A fast implementation of a Bayesian generative network used for fingerprint generation.
  2. How the BayesianNetwork class works

    master

    The BayesianNetwork class is used to randomly sample from a distribution defined by a JSON network definition.

    Network Definition Structure

    The definition is a JSON object containing a nodes array. Each node object must include:

    • name: The unique identifier for the node.
    • values: An array of possible values for that node.
    • parentNames: An array of names of nodes that this node depends on.
    • conditionalProbabilities: The probability distribution. This can be provided upfront in the JSON, or calculated later from data.

    Workflow

    1. Initialize: Create an instance with new BayesianNetwork(networkDefinition).
    2. Configure Probabilities (Optional): If the JSON doesn't include probabilities, use .setProbabilitiesAccordingToData(dataframe) with a Danfo.js dataframe.
    3. Persist: Save the configured network using .saveNetworkDefinition(networkDefinitionFilePath).
    4. Sample: Generate data using .generateSample() or .generateConsistentSampleWhenPossible().
    {
        "nodes": [
            {
                "name": "ParentNode",
                "values": ["A", "B", "C"],
                "parentNames": [],
                "conditionalProbabilities": {
                    "A": 0.1,
                    "B": 0.8,
                    "C": 0.1
                }
            },
            {
                "name": "ChildNode",
                "values": [".", ",", "!", "?"],
                "parentNames": ["ParentNode"],
                "conditionalProbabilities": {
                    "A": {
                        ".": 0.7,
                        "!": 0.3
                    },
                    "B": {
                        ",": 0.3,
                        "?": 0.7
                    },
                    "C": {
                        ".": 0.5,
                        "?": 0.5
                    }
                }
            }
        ]
    }
  3. Use HeaderGenerator to generate browser-like headers

    master

    To generate headers, create an instance of the HeaderGenerator class. You can provide global configuration via HeaderGeneratorOptions in the constructor.

    When calling .getHeaders(), you can pass an optional options object to override the global configuration for that specific call. The method generates a random, realistic set of headers, excluding request-dependent headers like Host or HTTP/2 pseudo-headers.

    import { HeaderGenerator } from 'header-generator';
    
    // Initialize with global options
    let headerGenerator = new HeaderGenerator({
        browsers: [
            { name: 'firefox', minVersion: 90 },
            { name: 'chrome', minVersion: 110 },
            'safari',
        ],
        devices: ['desktop'],
        operatingSystems: ['windows'],
    });
    
    // Get headers with specific overrides for this call
    let headers = headerGenerator.getHeaders({
        operatingSystems: ['linux'],
        locales: ['en-US', 'en'],
    });
  4. Inject fingerprints into Playwright using newInjectedContext

    master

    To camouflage a Playwright-managed Chromium instance, use the newInjectedContext function from fingerprint-injector. This function allows you to pass fingerprintOptions to constrain the generated fingerprint (e.g., by device type or operating system) and newContextOptions to pass standard Playwright context configurations.

    import { chromium } from 'playwright';
    import { newInjectedContext } from 'fingerprint-injector';
    
    (async () => {
        const browser = await chromium.launch({ headless: false });
        const context = await newInjectedContext(browser, {
            // Constraints for the generated fingerprint (optional)
            fingerprintOptions: {
                devices: ['mobile'],
                operatingSystems: ['ios'],
            },
            // Playwright's newContext() options (optional, random example for illustration)
            newContextOptions: {
                geolocation: {
                    latitude: 51.50853,
                    longitude: -0.12574,
                },
            },
        });
    
        const page = await context.newPage();
        // ... your code using `page` here
    })();
  5. Inject fingerprints into Puppeteer using newInjectedPage

    master

    To camouflage a Puppeteer-managed browser instance, use the newInjectedPage function from fingerprint-injector. This function creates a new page and injects a fingerprint based on the provided fingerprintOptions.

    import puppeteer from 'puppeteer';
    import { newInjectedPage } from 'fingerprint-injector';
    
    (async () => {
        const browser = await puppeteer.launch({ headless: false });
        const page = await newInjectedPage(browser, {
            // constraints for the generated fingerprint
            fingerprintOptions: {
                devices: ['mobile'],
                operatingSystems: ['ios'],
            },
        });
    
        // ... your code using `page` here
        await page.goto('https://example.com');
    })();
  6. HeaderGeneratorOptions configuration

    master

    Configuration object used to control the header generation process.

    ParamTypeDescription
    browsers(BrowserSpecification|string)[]?List of BrowserSpecification objects or strings (chrome, firefox, safari).
    browserListQuerystring?A Browserslist query. If provided, the browsers array is ignored.
    operatingSystemsstring[]?List of OS: windows, macos, linux, android, ios.
    devicesstring[]?List of device types: desktop, mobile.
    localesstring[]?Up to 10 language tags for the Accept-Language header (e.g., en, en-US, de).
    httpVersionstring?HTTP version for header generation: 1 or 2. Default is 2.
  7. HeaderGenerator class API

    master

    The HeaderGenerator class is the primary interface for generating realistic browser headers.

    new HeaderGenerator(options)

    Creates a new instance.

    • Param options: HeaderGeneratorOptions (default generation options).

    .getHeaders(options, requestDependentHeaders)

    Generates a single set of ordered headers.

    • Param options: HeaderGeneratorOptions (overrides for this specific call).
    • Param requestDependentHeaders: Record<string, any> (known values for headers dependent on the specific request, such as Host. These are merged into the result).

    .orderHeaders(headers, order)

    Returns a new object containing the provided headers in a specific order.

    • Param headers: object (the headers to order).
    • Param order: string[]? (an array of ordered header names; if omitted, order is deduced from the user-agent).
  8. Configure HeaderGeneratorOptions

    master

    The HeaderGeneratorOptions interface defines how headers are generated.

    Key options include:

    • browsers: An array of BrowserSpecification objects or BrowserName strings (e.g., 'chrome', 'edge', 'firefox', 'safari').
    • browserListQuery: A string query for Browserslist. If provided, the browsers array is ignored.
    • operatingSystems: An array of OperatingSystem values.
    • devices: An array of Device values (e.g., ['desktop']).
    • locales: An array of language strings (e.g., ['en-US', 'de']) used for the Accept-Language header.
    • httpVersion: Either '1' or '2'. Defaults to '2'.
    • strict: If true, the generator throws an error if it cannot satisfy all constraints instead of relaxing them.
    export interface HeaderGeneratorOptions {
        browsers: BrowsersType;
        browserListQuery: string;
        operatingSystems: OperatingSystem[];
        devices: Device[];
        locales: string[];
        httpVersion: HttpVersion;
        strict: boolean;
    }