Apify SDK for Python

The complete documentation for the Apify SDK for Python is hosted at docs.apify.com/sdk/python.

Key documentation sections include:

• Overview: Explains the SDK, Actors, and core components.
• Quick start: Instructions for creating, running, and deploying your first Python Actor.
• Concepts: Detailed information on Actor lifecycle, input, storages, events, proxy management, interacting with other Actors, webhooks, accessing the Apify API, logging, configuration, and pay-per-event.
• Guides: Integration guides for BeautifulSoup, Parsel, Playwright, Selenium, Crawlee, Scrapy, Scrapling, Crawl4AI, and Browser Use. Also covers using uv, Pydantic for input validation, running web servers, building MCP servers, and hosting AI agents.
• Upgrading: Migration guides for moving between major versions.
• API reference: Full technical reference for every class and method.

The ApifyStorageClient offers two modes for accessing request queues via the request_queue_access argument:

• 'single' (default): Optimized for a single consumer. It is faster and cheaper but does not support concurrent consumption by multiple clients.
• 'shared': Supports multiple consumers working on the same queue simultaneously. Use this when splitting a large crawl across parallel Actor runs or using a producer-worker pattern.

To use the shared mode, you must set it as the cloud client of the SmartApifyStorageClient in the service locator before entering the Actor context.

# Example of setting up a shared request queue
# (Requires importing the appropriate client from crawlee)
from apify import Actor
from apify.storage_clients import SmartApifyStorageClient
from crawlee.storage_clients import ApifyStorageClient

# In the service locator setup (before Actor.init or Actor context)
# This is a conceptual representation of the pattern described
# where ApifyStorageClient is configured with request_queue_access='shared'

When the Apify platform migrates an Actor or performs a graceful abort, the ApifyGracefulStopExtension and ApifyScheduler work together to prevent data loss and duplication:

1. Detection: The integration reacts to MIGRATING or ABORTING events.
2. Scheduler Behavior: The ApifyScheduler stops handing out new requests to Scrapy.
3. Cleanup: It waits for in-flight requests (including callbacks and item pipelines) to finish, then marks them as handled in the Apify Request Queue.
4. Resumption: On the next run, the scheduler resumes from the pending requests in the queue. Only requests that were not finished during the abort are re-downloaded.

Note on Start URLs: By default, Spider.start() yields URLs with dont_filter=True, which the integration maps to always_enqueue=True. This causes restarted runs to crawl start URLs again. To ensure start URLs are deduplicated against the request queue, override start() and yield plain requests instead.

When starting another Actor using Actor.call, you can control how its logs appear in your current Actor's logs using the logger argument:

1. Default (logger=default): Logs from the called Actor are automatically redirected to the parent Actor's logs with default formatting. Entries include cyan-colored text identifying the called Actor's name and run ID.
2. Disable Redirection (logger=None): No logs from the called Actor will be redirected to the parent.
3. Custom Logger: Pass your own Logger instance to define custom formatters, filters, or handlers for the redirected logs.

from apify import Actor

# 1. Default redirection
Actor.call('other-actor-name')

# 2. No redirection
Actor.call('other-actor-name', logger=None)

# 3. Custom logger redirection
import logging
custom_logger = logging.getLogger('custom')
Actor.call('other-actor-name', logger=custom_logger)

There are two primary methods for providing an LLM to your agent:

1. Apify OpenRouter Proxy (Recommended)

Use the OpenAI-compatible endpoint at https://openrouter.apify.actor/api/v1. This allows you to access the full OpenRouter model catalog without managing individual provider API keys.

• Authentication: The proxy uses the APIFY_TOKEN automatically injected into the Actor run.
• Billing: Token usage is billed against your Apify account.
• Model Selection: Change the model input to any OpenRouter model slug (e.g., openai/gpt-5.4-mini).

2. Direct Provider Integration

Call providers like OpenAI directly using your own API key.

• Security: Do not hardcode keys. Read them from environment variables.
• Setup: On Apify, set these as secret environment variables. Locally, export them in your shell.