spidermon

Spidermon uses two primary abstractions to validate spider execution:

1. Monitors: Classes that contain monitoring logic. They are similar to test cases and use methods to validate data (e.g., checking item counts or data models). You can use @monitors.name("Name") to label your monitors and individual test methods.
2. MonitorSuites: Collections of monitors. A suite defines which monitors to run and what actions (like notifications) to perform before or after the suite executes.

To run a suite when a spider finishes, add the suite's import path to the SPIDERMON_SPIDER_CLOSE_MONITORS setting in settings.py.

# tutorial/monitors.py
from spidermon import Monitor, MonitorSuite, monitors

@monitors.name("Item count")
class ItemCountMonitor(Monitor):
    @monitors.name("Minimum number of items")
    def test_minimum_number_of_items(self):
        # Access spider stats via self.data.stats
        item_extracted = getattr(self.data.stats, "item_scraped_count", 0)
        minimum_threshold = 10
        msg = "Extracted less than {} items".format(minimum_threshold)
        self.assertTrue(item_extracted >= minimum_threshold, msg=msg)

class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        ItemCountMonitor,
    ]

A MonitorSuite groups multiple Monitor classes together and defines which actions should be executed based on the outcome of the monitoring process.

To use a suite, you must register it in your Scrapy settings.py using either SPIDERMON_SPIDER_OPEN_MONITORS (to run at spider start) or SPIDERMON_SPIDER_CLOSE_MONITORS (to run at spider finish).

# monitors.py
from spidermon.core.suites import MonitorSuite

class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        # list of Monitor classes
    ]

    monitors_finished_actions = [
        # actions to execute when suite finishes its execution
    ]

    monitors_failed_actions = [
        # actions to execute when suite finishes its execution with a failed monitor
    ]

# settings.py
SPIDERMON_SPIDER_OPEN_MONITORS = (
    # list of monitor suites to be executed when the spider starts
)

SPIDERMON_SPIDER_CLOSE_MONITORS = (
    # list of monitor suites to be executed when the spider finishes
)

The spidermon.contrib.monitors.mixins.ValidationMonitorMixin allows you to create monitors that check for validation errors in job stats.

Method Groups

1. Missing Required Fields:

   • check_missing_required_fields / check_missing_required_field: Checks if the count of missing required fields is below a threshold.
   • check_missing_required_fields_percent / check_missing_required_field_percent: Checks if the ratio of missing required fields is below a threshold.

2. General Field Errors:

   • check_fields_errors / check_field_errors: Checks if the count of errors in specified fields is below a threshold.
   • check_fields_errors_percent / check_field_errors_percent: Checks if the ratio of errors in specified fields is below a threshold.

Important Notes

• Naming Convention: Methods ending in _field take a single field name. Methods ending in _fields take a list of field names.
• Ratios vs Percentages: For *_percent methods, pass the ratio (e.g., 0.15 for 15%), not the integer percentage.
• Field List Handling: By default, *_fields methods combine error counts for all fields instead of checking them individually. To change this, set the correct_field_list_handling monitor attribute.

To send Spidermon notifications to Telegram using Telegram Actions, you must create a Telegram bot, obtain its authorization token, and configure your Scrapy project settings with the token and recipient IDs.

Setup Steps

1. Create a Bot: Create a Telegram bot via the Telegram Bot API to receive your Bot Authorization Token.
2. Identify Recipients: Obtain the chat_id or group_id for the users, groups, or channels where you want notifications sent.
   • Tip: You can use [@GroupIDbot](https://t.me/GroupIDbot) to find these IDs by forwarding a message from the target user or group to the bot.
3. Add Bot to Destination:
   • For Groups: Add your created bot as a member of the group.
   • For Channels: Add your created bot as an administrator of the channel.
   • For Private Users: The user must first start a conversation with the bot by sending the /start command.
4. Configure Scrapy Settings: Add the credentials to your settings.py file.

# settings.py
SPIDERMON_TELEGRAM_SENDER_TOKEN = "YOUR_BOT_AUTHORIZATION_TOKEN"
SPIDERMON_TELEGRAM_RECIPIENTS = ["chat_id", "group_id", "@channelname"]