DOCSEARCHbyAlgolia
Submit Docs

scrapyd

repository·master·Indexed 25 days ago

https://github.com/scrapy/scrapyd

A deployment service for Scrapy spiders (version 1.6.0) that allows users to upload Scrapy projects and control spider execution via a JSON-based HTTP API. It includes a CLI for deployment, a hierarchical configuration system, and support for managing project versions, scheduling crawl jobs, and monitoring job status.

Tokens
8.4K
Snippets
21
Records
79
Agent score
81%

What's inside scrapyd

  2. Manage Scrapy projects and versions in Scrapyd

    master

    Scrapyd allows you to manage multiple Scrapy projects, where each project can contain multiple versions. When you schedule a spider to crawl, Scrapyd uses the latest version by default.

    Version Selection Logic:

    • If version names are standard Python version specifiers (e.g., 1.0, 1.0rc1), they are sorted according to Python versioning rules.
    • Otherwise, the latest version is determined by alphabetical order (the alphabetically greatest name).

  3. Understand how Scrapyd operates

    master

    Scrapyd operates as a server (typically a daemon) that handles requests via an API and a web interface. Its primary functions are uploading projects and scheduling crawls.

    When a crawl is scheduled, Scrapyd spawns a process that executes the equivalent of:

    scrapy crawl myspider

    Scrapyd manages multiple processes in parallel and controls the number of concurrent processes. For advanced configuration of the process launcher, refer to the config-launcher documentation.

  5. Add Custom Webservices to Scrapyd

    master

    You can add new API endpoints by defining them in the [services] section of your configuration file. Map a JSON filename to a Python module/class.

    Example configuration:

    [services]
mywebservice.json = amodule.anothermodule.MyWebService

    To remove a default webservice, set its name to an empty string:

    [services]
daemonstatus.json =
    [services]
mywebservice.json = amodule.anothermodule.MyWebService

  7. Upload a project to Scrapyd

    master
    To upload a project, you must build a Python egg and upload it via the addversion.json webservice. The easiest way to do this is using the scrapyd-deploy command from the scrapyd-client package. Once configured, run:
    scrapyd-deploy

  8. Create a Docker image for Scrapyd and Scrapy projects

    master

    You can containerize the Scrapyd service and your Scrapy projects using a multi-stage Dockerfile. The process involves a build-stage to create the project's .egg file using scrapyd-deploy, and a final stage based on python:alpine that installs scrapyd and the necessary system dependencies (like libxml2 and libxslt) to run the service.

    Key configuration steps in the Dockerfile:

    • Volumes: Mount /etc/scrapyd/ for configuration and /var/lib/scrapyd/ for runtime data.
    • Ports: The service exposes port 6800 by default.
    • Entrypoint: Uses scrapyd with the --pidfile= flag.
    # Build an egg of your project.
FROM python as build-stage

RUN pip install --no-cache-dir scrapyd-client

WORKDIR /workdir

COPY . .

RUN scrapyd-deploy --build-egg=myproject.egg

# Build the image.
FROM python:alpine

# Install Scrapy dependencies - and any others for your project.
RUN apk --no-cache add --virtual build-dependencies \
   gcc \
   musl-dev \
   libffi-dev \
   libressl-dev \
   libxml2-dev \
   libxslt-dev \
 \
   && pip install --no-cache-dir \
   scrapyd \
   && apk del build-dependencies \
   && apk add \
   libressl \
   libxml2 \
   libxslt

# Mount two volumes for configuration and runtime.
VOLUME /etc/scrapyd/ /var/lib/scrapyd/

COPY ./scrapyd.conf /etc/scrapyd/

RUN mkdir -p /src/eggs/myproject

COPY --from=build-stage /workdir/myproject.egg /src/eggs/myproject/1.egg

EXPOSE 6800

ENTRYPOINT ["scrapyd", "--pidfile="]

  9. Access the Scrapyd web interface

    master

    Scrapyd provides a minimal built-in web interface for monitoring running processes, accessing log files, and viewing item feeds. By default, the interface is available at http://localhost:6800/.

    If you require more advanced management features, you can use third-party tools such as:

    • ScrapydWeb
    • spider-admin-pro

  10. Run Scrapyd within a Scrapy project

    master
    You can run the scrapyd command directly from a directory that contains a scrapy.cfg file, or from any directory that has a scrapy.cfg file in one of its parent directories. This allows Scrapyd to use the settings defined in your Scrapy project.

  11. Configure Scrapyd via configuration files

    master

    Scrapyd uses a hierarchical configuration system. It always loads a default configuration file, which can be overridden by other files. When multiple configuration files are present, values in later files take priority according to the following order:

    1. c:\scrapyd\scrapyd.conf (Windows)
    2. /etc/scrapyd/scrapyd.conf (Unix)
    3. /etc/scrapyd/conf.d/* (Unix, in alphabetical order)
    4. scrapyd.conf in the current directory
    5. ~/.scrapyd.conf in the user's home directory
    6. The closest scrapy.cfg file (starting in the current directory and traversing upward)