DOCSEARCHbyAlgolia
Submit Docs

Scrapinghub Python Client

repository·master·Indexed 41 days ago

https://github.com/scrapinghub/python-scrapinghub

A Python client for the Scrapinghub API (Python 3.10+) that allows developers to programmatically manage projects, spiders, and job executions. It provides tools to run jobs, access scraped items, manage job metadata, handle API exceptions, and interact with collections, logs, and Frontiers.

Tokens
6.2K
Snippets
26
Records
36
Agent score
52%

What's inside Scrapinghub

  1. How spiders work within a project

    master

    A project contains one or more web crawlers called "spiders". Spiders are accessible via the .spiders attribute of a Project instance.

    • Use project.spiders.list() to get a list of dictionaries containing spider metadata (like id, type, and version). Note that the id in the dictionary corresponds to the spider's name.
    • Use project.spiders.get(spider_name) to retrieve a specific Spider instance.
    # List spiders in a project
spiders_list = project.spiders.list()

# Get a specific spider instance
sprider = project.spiders.get('spider2')

  2. How jobs and jobs collections work

    master

    A "job" is a single run of a spider. Jobs are managed via a Jobs collection object, which is available as a .jobs attribute on both Project and Spider instances.

    • Project-level jobs: Access all jobs within a project via project.jobs.
    • Spider-level jobs: Access only jobs for a specific spider via spider.jobs.

    Jobs can be run, iterated over, filtered, and managed (canceled/deleted) through these collections.

  3. How projects work in Scrapinghub

    master

    Projects are the top-level containers in Scrapinghub. You can access them through the client.projects attribute.

    • Use client.projects.list() to get a list of numeric project IDs.
    • Use client.projects.summary() to get a high-level overview of all projects, including counts for finished, running, and pending jobs, and whether the project has_capacity.
    • Use client.get_project(project_id) or the shortcut client.projects.get(project_id) to obtain a Project instance for a specific ID.
    # List project IDs
project_ids = client.projects.list()

# Get project summary
summary = client.projects.summary()

# Get a specific project instance
project = client.get_project(123)

  8. Install the scrapinghub Python library

    master

    You can install the scrapinghub library using pip.

    For standard installation:

    pip install scrapinghub

    It is highly recommended to install the library with MessagePack support to achieve better response times and improved bandwidth usage:

    pip install scrapinghub[msgpack]
    pip install scrapinghub[msgpack]

  9. Instantiate the ScrapinghubClient

    master

    To interact with the Scrapinghub API, create an instance of ScrapinghubClient. You can provide your API key directly as a string during instantiation.

    Your Scrapy Cloud API key can be found at the bottom of your settings page at https://app.zyte.com/o/settings.

    from scrapinghub import ScrapinghubClient

apikey = '84c87545607a4bc0****************' # your API key as a string
client = ScrapinghubClient(apikey)

  10. Configure API keys via environment variables or .env files

    master

    If you do not provide an explicit API key to ScrapinghubClient(), the client will automatically attempt to read one of the following environment variables:

    • SH_APIKEY
    • SHUB_APIKEY (alias)
    • SHUB_JOBAUTH (alias)

    Using a .env file

    You can store your key in a .env file. By default, the client searches for the nearest .env file in the current directory and walks up parent directories.

    To specify a custom path for your environment file, use the dotenv_path argument.

    Note: Variables already set in your system environment take precedence over values found in a .env file. Only the three specific keys mentioned above are recognized; all other variables in the file are ignored.

    # .env file content
SH_APIKEY=84c87545607a4bc0****************
    # Using default environment discovery
client = ScrapinghubClient() 

# Using a specific .env file path
client = ScrapinghubClient(dotenv_path='/path/to/myenv')

  11. Initialize the HubstorageClient

    master

    The HubstorageClient is the primary entry point for interacting with Scrapinghub storage services. You can initialize it with an authentication token and optional configuration for endpoints and retry policies.

    Configuration Options:

    • auth (str): The client authentication token.
    • endpoint (str, optional): The API root address. Defaults to the SHUB_STORAGE environment variable or https://storage.scrapinghub.com/.
    • connection_timeout (int): Timeout for a single request in seconds. Defaults to 60.0.
    • max_retries (int): Number of times idempotent requests may be retried.
    • max_retry_time (int): Maximum time in seconds to spend retrying a request.
    • use_msgpack (bool): Enables/disables msgpack serialization. If True, it requires the msgpack library to be installed.
    • jobq_endpoint (str, optional): The JobQ API root address. Defaults to the SHUB_JOBQ environment variable or the main endpoint value.

    Retry Behavior: By default, the client retries requests up to 3 times within a 60-second window. If you provide max_retries without max_retry_time, it will retry exactly that many times regardless of duration. If you provide max_retry_time, the client will attempt to distribute retries within that time window using exponential backoff.

    from hubstorage import HubstorageClient

client = HubstorageClient(
    auth='YOUR_AUTH_TOKEN',
    max_retries=3,
    max_retry_time=60
)