FEAPDER Documentation

FEAPLAT (a combination of feapder and platform) is a crawler management system designed for high stability and elastic scaling. Unlike traditional systems where worker nodes are persistent, FEAPLAT dynamically generates workers for specific task instances. Once a task is completed, the worker is destroyed. This ensures that tasks do not interfere with each other and allows for automatic load balancing across a server cluster.

Key Features:

• Supports any Python script (including feapder and scrapy).
• Supports browser rendering via playwright or selenium (including headless mode).
• Supports server cluster management and automatic load balancing.
• Supports elastic scaling and multiple task instances (distributed scenarios).
• Supports 4 types of scheduled starts.
• Supports custom worker images (e.g., for Java or Machine Learning environments).
• One-click deployment via Docker Swarm.

FEAPDER is a modular web crawling framework. The architecture is designed to decouple task scheduling, data downloading, and data parsing through various buffers and controllers to optimize database access and concurrency.

Core Modules

• spider: The central scheduling core of the framework.
• parser_control: The template controller responsible for scheduling parser instances.
• collector: A task collector that batches tasks from the task queue into memory to reduce database access frequency and concurrency.
• parser: The data parser responsible for extracting information.
• start_request: The initial task dispatch function.
• item_buffer: A data buffer queue that performs batch storage of extracted data into the database.
• request_buffer: A request task buffer queue that performs batch storage of new request tasks into the task queue.
• request: The data downloader that encapsulates requests for internet data retrieval.
• response: A request response wrapper that supports xpath, css, and re parsing methods and automatically handles Chinese character encoding issues.

You can integrate multiple independent parsers into a single Spider to manage multiple data sources under one crawler. This is useful for projects with consistent collection cycles and requirements but different data sources.

To achieve this:

1. Rewrite your existing Spiders as Parsers: Change the inheritance from feapder.Spider to feapder.BaseParser. BaseParser contains the parsing logic but lacks scheduling capabilities.
2. Register Parsers: Use the add_parser() method on your Spider instance to add the parser classes.

Note: Spider and BatchSpider support integration, but AirSpider does not.

import feapder

# 1. Define parsers inheriting from BaseParser
class SinaNewsParser(feapder.BaseParser):
    def start_requests(self):
        yield feapder.Request("https://news.sina.com.cn/")

    def parse(self, request, response):
        title = response.xpath("//title/text()").extract_first()
        print(title)

class TencentNewsParser(feapder.BaseParser):
    def start_requests(self):
        yield feapder.Request("https://news.qq.com/")

    def parse(self, request, response):
        title = response.xpath("//title/text()").extract_first()
        print(title)

# 2. Integrate into a single Spider
spider = feapder.Spider(redis_key="feapder:test_spider_integration")
sider.add_parser(SinaNewsParser)
sider.add_parser(TencentNewsParser)

sider.start()

Use GoldUserPool for accounts with high unit costs that require strict usage limits, such as frequency or time constraints. This pool requires a Redis environment. You define GoldUser objects with parameters like max_use_times, use_interval, and login_interval. You must implement a custom login method to handle authentication.

from feapder.network.user_pool import GoldUser
from feapder.network.user_pool import GoldUserPool

users = [
    GoldUser(
        username="zhangsan",
        password="1234",
        max_use_times=10,
        use_interval=5,
    ),
    GoldUser(
            username="lisi",
            password="1234",
            max_use_times=10,
            use_interval=5,
            login_interval=50,
        ),
]

class CustomGoldUserPool(GoldUserPool):
    def login(self, user: GoldUser) -> GoldUser:
        # Implement login logic
        user.cookies = "zzzz"
        return user

user_pool = CustomGoldUserPool(
    "test:user_pool",
    users=users,
    keep_alive=True,
)

To automatically save scraped items into a MongoDB database, use MongoPipeline within your AirSpider class. You must configure the connection details in the __custom_setting__ dictionary using the following keys:

• ITEM_PIPELINES: Set to ["feapder.pipelines.mongo_pipeline.MongoPipeline"].
• MONGO_IP: MongoDB server IP.
• MONGO_PORT: MongoDB port.
• MONGO_DB: Database name.
• MONGO_USER_NAME: Username.
• MONGO_USER_PASS: Password.

When yielding an Item, specify the target collection name using the table_name attribute.

import feapder
from feapder import Item


class TestMongo(feapder.AirSpider):
    __custom_setting__ = dict(
        ITEM_PIPELINES=["feapder.pipelines.mongo_pipeline.MongoPipeline"],
        MONGO_IP="localhost",
        MONGO_PORT=27017,
        MONGO_DB="feapder",
        MONGO_USER_NAME="",
        MONGO_USER_PASS="",
    )

    def start_requests(self):
        yield feapder.Request("https://www.baidu.com")

    def parse(self, request, response):
        title = response.xpath("//title/text()").extract_first()
        item = Item()
        item.table_name = "test_mongo"  # Specify the collection name
        item.title = title
        yield item


if __name__ == "__main__":
    TestMongo().start()