scrapy-redis

The scrapy_redis package is organized into several submodules that handle different aspects of distributed crawling with Scrapy and Redis:

• scrapy_redis.connection: Manages Redis connections.
• scrapy_redis.dupefilter: Provides distributed duplicate filtering logic.
• scrapy_redis.pipelines: Contains pipelines for processing items via Redis.
• scrapy_redis.queue: Manages the request queues stored in Redis.
• scrapy_redis.scheduler: Implements a distributed scheduler to manage request priority and queuing.
• scrapy_redis.spiders: Provides base spider classes for distributed crawling.

To install the version from GitHub (which includes specific features like JSON-supported data in Redis), clone the repository and run the setup script.

Note: If you previously installed scrapy-redis via pip, you must uninstall it first to avoid conflicts with the JSON-supported data feature:

pip uninstall scrapy-redis

git clone https://github.com/darkrho/scrapy-redis.git
cd scrapy-redis
python setup.py install

You can run the entire example-project setup using Docker and Docker Compose. This includes starting the crawler service and scaling it to multiple instances.

1. Start the sample project (using -d for daemon mode if desired): docker-compose up

2. Scale the crawler service (e.g., to 4 instances): docker-compose scale crawler=4

To perform a broad crawl by sharing a request queue across multiple instances:

1. Ensure scrapy_redis is in your PYTHONPATH.
2. Start the initial crawler to populate/initialize the queue.
3. Resume the crawl or start additional crawler instances to increase crawling speed. Each instance will pick up requests from the shared Redis queue.
4. Start post-processing workers to consume the items from Redis.

To contribute to scrapy-redis, follow these steps to set up a local development environment using a virtualenv:

1. Fork the repository on GitHub.
2. Clone your fork locally.
3. Create and activate a virtual environment.
4. Install dependencies and the local package.
5. Create a new branch for your changes.

# Clone your fork
git clone git@github.com:your_name_here/scrapy-redis.git

# Setup virtualenv
pip install virtualenv==20.0.23
virtualenv --python=/usr/bin/python3 ~/scrapy_redis
source ~/scrapy_redis/bin/activate

# Install package
cd scrapy-redis/
pip install -r requirements-install.txt
pip install .

# Create development branch
git checkout -b name-of-your-bugfix-or-feature

Before submitting changes, ensure they pass linting (flake8), unit tests (pytest), and multi-version compatibility tests (tox).

If you encounter a No module named scrapy_redis error during testing, ensure you have installed the local package using pip install . or adjust your imports to reference the src/ directory.

To install the stable release of Scrapy-Redis, use the standard Python package installer pip.

pip install scrapy-redis

Report bugs at https://github.com/rmax/scrapy-redis/issues. When reporting, please include:

• Your operating system name and version.
• Details about your local setup helpful for troubleshooting.
• Detailed steps to reproduce the bug.

This fork supports storing JSON data in Redis, which includes url, meta, and other optional parameters. The meta field is a nested JSON object. When this data is processed, it extracts the information and sends a FormRequest containing the url, meta, and additional formdata.

In your Scrapy spider, you can access this data via the request object using request.url, request.meta, and request.cookies.

{ "url": "https://exaple.com", "meta": {"job-id":"123xsd", "start-date":"dd/mm/yy"}, "url_cookie_key":"fertxsas" }

Ensure your pull request meets these requirements:

1. Include tests for your changes.
2. Update documentation if adding functionality (include docstrings and update README.rst).
3. Verify that tests pass for all supported Python versions.

By default, all requests in scrapy-redis are persisted. If you want to clear the existing queue and start a fresh crawl, use the SCHEDULER_FLUSH_ON_START setting via the command line.

scrapy crawl dmoz -s SCHEDULER_FLUSH_ON_START=1