got-scraping

The module is ESM only. You must use import expressions or the import() method.

If your project is ESM:

import { gotScraping } from 'got-scraping';

If you are using CommonJS and cannot migrate to ESM, import it within an async context using dynamic imports:

let gotScraping;

async function fetchWithGotScraping(url) {
    gotScraping ??= (await import('got-scraping')).gotScraping;

    return gotScraping.get(url);
}

Requirement: Node.js >=16 is required.

Install the package using npm. Note that got-scraping is End-of-Life (EOL) and will no longer receive updates. For new projects, it is recommended to use impit instead.

$ npm install got-scraping

If you encounter the error: RequestError: Client network socket disconnected before secure TLS connection was established

This often means the server does not support the provided TLS settings. To resolve this, try changing the ciphers parameter to either undefined or a custom value.

To manually override the automatically generated browser-like headers, pass a headers object in the request options. This is useful for adding specific headers like referer which might be necessary for certain sites.

const response = await gotScraping({
    url: 'https://apify.com/',
    headers: {
        'user-agent': 'test',
    },
});

You can request JSON responses by setting responseType: 'json'.

Note: Header generation is optimized for HTML content types. When using JSON mode, you may need to manually adjust headers to better match browser behavior for API requests.

const response = await gotScraping({
    responseType: 'json',
    url: 'https://api.apify.com/v2/browser-info',
});

The gotScraping instance is built using got.extend(), meaning it supports all standard got features. It is designed to send browser-like requests out of the box to help blend in with website traffic.

import { gotScraping } from 'got-scraping';

gotScraping
    .get('https://apify.com')
    .then( ({ body }) => console.log(body));

The sessionToken option is a non-primitive unique object used to describe the current session.

• If undefined (default), new headers are generated for every request.
• If a sessionToken is provided, headers generated with that token will remain consistent across requests.

When using gotScraping.paginate, you provide an object of type ExtendedPaginationOptions.

Key properties include:

• pagination: An object of type ExtendedPaginationOptions<ElementType, BodyType>.
• paginate: A function (data: PaginateData<BodyType, ElementType>) => OptionsInit | false used to determine the next request parameters based on the current page's data. If it returns false, pagination stops.
• countLimit: (Inherited from PaginationOptions) used to limit the number of items processed.

Provide a proxyUrl string to route requests through an HTTP, HTTPS, or HTTP/2 based proxy. Got Scraping automatically detects the protocol supported by the proxy server.

import { gotScraping } from 'got-scraping';

gotScraping
    .get({
        url: 'https://apify.com',
        proxyUrl: 'http://usernamed:password@myproxy.com:1234',
    })
    .then(({ body }) => console.log(body));

If you provide a proxyUrl with an unsupported protocol, the library throws a ProxyError. Supported protocols are http: and https:.

If you encounter this error, ensure your proxy URL starts with http:// or https://.

The sessionDataHook is a middleware hook used to associate session-specific data with a request's context. It uses the sessionToken found in options.context as a key to retrieve or initialize a data object from an internal WeakMap.

When the hook is executed, it attaches a sessionData object to options.context. This allows you to persist and retrieve state (like cookies or proxy settings) across multiple requests that share the same sessionToken.

import { sessionDataHook } from 'got-scraping';

// Example usage within a got request configuration
const response = await got({
    url: 'https://example.com',
    context: {
        sessionToken: 'unique-session-id-123',
        // sessionData will be automatically attached/retrieved by the hook
    },
    hooks: {
        beforeRequest: [sessionDataHook]
    }
});