Apify SDK for JavaScript/TypeScript

Apify provides base Docker images to simplify running headless browsers in Docker. Images are available in two flavors:

1. Full images: Preinstall apify, crawlee, and typescript. Use these for quick execution without maintaining a package.json or when you want to rely on specific preinstalled versions.
2. -slim variants: Only ship the browser automation library (e.g., puppeteer, playwright, or camoufox-js) or no npm packages at all (e.g., apify/actor-node:24-slim). These are smaller and faster to pull. Your package.json serves as the single source of truth for dependencies.

Available Image Families:

• apify/actor-node
• apify/actor-node-puppeteer-chrome
• apify/actor-node-playwright (includes chrome, firefox, webkit, and camoufox variants)

Recommendation: Use the -slim variant unless you have a specific reason not to.

# No preinstalled packages, bring your own dependencies.
FROM apify/actor-node:24-slim

SDK v1 replaces PuppeteerPool with BrowserPool to support both Puppeteer and Playwright. BrowserPool manages the lifecycle of browsers and provides several key features:

Accessing the BrowserPool

You can access the browserPool instance on the crawler object in both PuppeteerCrawler and PlaywrightCrawler.

Using BrowserController

The BrowserController is a class provided within the crawling context that offers a unified API for both Puppeteer and Playwright.

Important: To ensure graceful shutdowns and compatibility across both libraries, use browserController instead of calling page.browser().close() or using library-specific methods like page.setCookies().

• To close a browser: Use await browserController.close().
• To set cookies: Use await browserController.setCookies(page, cookies).

Lifecycle Hooks

You can configure lifecycle hooks via browserPoolOptions in your crawler configuration. This allows you to run code (like preLaunchHooks) before a browser is launched, using the pageId to retrieve the relevant context.

const crawler = new Apify.PuppeteerCrawler({
    browserPoolOptions: {
        retireBrowserAfterPageCount: 10,
        preLaunchHooks: [
            async (pageId, launchContext) => {
                const { request } = crawler.crawlingContexts.get(pageId);
                if (request.userData.useHeadful === true) {
                    launchContext.launchOptions.headless = false;
                }
            },
        ],
    },
});

const handlePageFunction = async ({ page, browserController }) => {
    // Correct usage for graceful shutdown and cross-library compatibility
    await browserController.close();
    
    const cookies = [{ name: 'example', value: '123' }];
    await browserController.setCookies(page, cookies);
};

Key-value stores are used for saving and reading data records or files (e.g., screenshots, PDFs, or crawler state). Each record is identified by a unique key and a MIME content type.

Default Key-Value Store

Each actor run has a default key-value store. By convention:

• Actor input is stored under the INPUT key.
• Actor output is stored under the OUTPUT key.

Accessing Key-Value Stores

You can use the KeyValueStore class or the convenience methods on the Actor class:

• Actor.getValue(key): Retrieve a value.
• Actor.setValue(key, value): Save a value.
• Actor.openKeyValueStore(name): Open a specific named store.

Local Storage Structure

In local development, files are stored at: {APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}

Note: The default store ID is default, but can be overridden using the APIFY_DEFAULT_KEY_VALUE_STORE_ID environment variable.

// Get actor input from the default key-value store
const input = await Actor.getInput();

// Write actor output to the default key-value store.
await Actor.setValue('OUTPUT', { myResult: 123 });

// Open a named key-value store
const store = await Actor.openKeyValueStore('some-name');

// Write record. JavaScript objects are automatically converted to JSON.
await store.setValue('some-key', { foo: 'bar' });

// Read record. JSON is automatically parsed; text is returned as string; others as binary buffer.
const value = await store.getValue('some-key');

// Delete record
await store.setValue('some-key', null);

Actors on the Apify platform may receive signals to shut down (e.g., when a user aborts a run or the platform migrates the actor to a different server).

• aborting event: The SDK calls Actor.exit(), which waits for cleanup handlers before exiting.
• migrating event: The SDK calls Actor.reboot(), which waits for cleanup handlers and then restarts the Actor on a new worker.

You can register cleanup logic using the persistState event. You can also configure a delay before shutdown via gracefulShutdownDelayMillis in Actor.init() or disable automatic handling by setting gracefulShutdown: false.

import { Actor } from 'apify';

await Actor.init({
    gracefulShutdownDelayMillis: 5000, // Wait 5 seconds before exiting/rebooting
});

// Register cleanup handler — runs on both exit and reboot
Actor.on('persistState', async () => {
    console.log('Saving state...');
    await Actor.setValue('STATE', myState);
});

// Your Actor logic here

await Actor.exit();

A RequestQueue is a storage of URLs used for deep crawling, where you start with initial URLs and recursively follow links to discover new ones. It supports both breadth-first and depth-first crawling orders.

Key characteristics:

• Dynamic: You can add or remove requests during the run.
• Default Queue: Each actor run has a default request queue with the ID default. You can override this ID by setting the APIFY_DEFAULT_REQUEST_QUEUE_ID environment variable.
• Implementation: In local development, it is emulated using a SQLite database located at {APIFY_LOCAL_STORAGE_DIR}/request_queues/{QUEUE_ID}/db.sqlite.

Use the RequestQueue class to manage these URLs.

// Open the default request queue associated with the actor run
const requestQueue = await RequestQueue.open();

// Enqueue the initial request
await requestQueue.addRequest({ url: 'https://example.com' });

// The crawler will automatically process requests from the queue
const crawler = new CheerioCrawler({
    requestQueue,
    handlePageFunction: async ({ $, request }) => {
        // Add new request to the queue
        await requestQueue.addRequest({ url: 'https://example.com/new-page' });
        // Add links found on page to the queue
        await Actor.utils.enqueueLinks({ $, requestQueue });
    },
});

In SDK v1, handler arguments (like handlePageFunction or handleFailedRequestFunction) no longer receive separate, unique objects for every invocation. Instead, they all receive a single, shared object called the Crawling Context.

This change allows you to track values across different function invocations because the context object is consistent. You can identify specific contexts using the id property and access other running contexts via the crawler.crawlingContexts Map.

let masterContextId;
const handlePageFunction = async ({ id, page, request, crawler }) => {
    if (request.userData.masterPage) {
        masterContextId = id;
        // Prepare the master page.
    } else {
        // Access a different context using its ID
        const masterContext = crawler.crawlingContexts.get(masterContextId);
        const masterPage = masterContext.page;
        const masterRequest = masterContext.request;
    }
};