funboost

Funboost is a general-purpose distributed function scheduling platform for Python. It is designed to be non-intrusive, allowing you to add distributed, high-concurrency, and high-reliability capabilities to any existing function using the @boost decorator without changing your project structure.

Core Capabilities:

• Distributed Task Queue: Supports over 40 message brokers (RabbitMQ, Kafka, Redis, MySQL CDC, etc.).
• FaaS (Function as a Service): Automatically turns decorated functions into HTTP microservices via funboost.faas.
• High-Performance Concurrency: Supports multiple modes (threading, gevent, eventlet, asyncio, single_thread) and precise QPS (Queries Per Second) control.
• Reliability: Built-in ACK (acknowledgment) mechanisms, automatic retries, and Dead Letter Queues (DLQ).
• RPC Mode: Allows synchronous or asynchronous retrieval of function execution results.

Common Use Cases:

• Asynchronous Background Tasks: Email/SMS, report generation, video transcoding.
• Microservices/FaaS: Exposing Python functions as HTTP APIs for cross-language calls (Java/Go/Frontend).
• Distributed Crawling: High-concurrency web scraping with built-in stability.
• Data Pipelines/ETL: Real-time data synchronization using MySQL CDC (Binlog).
• Scheduling: Distributed cron jobs and delayed tasks.
• IoT: Handling MQTT or TCP/UDP message streams.

The boost_spider library provides three core classes to simplify crawling tasks while leveraging funboost for distributed scheduling and concurrency:

1. RequestClient: An enhanced HTTP request class. It wraps requests.Session for cookie persistence and provides built-in support for rotating User-Agents and switching between multiple IP proxy providers automatically.
2. SpiderResponse: An enhanced response object returned by RequestClient. It allows direct parsing of HTML using .xpath(), .css(), and .re() methods without manual conversion.
3. DatasetSink: A data persistence class that allows writing Python dictionaries to various databases with a single line of code.

# Conceptual usage pattern
from boost_spider import RequestClient

client = RequestClient()
# Returns a SpiderResponse object
resp = client.request("https://example.com")

# Direct parsing via SpiderResponse
title = resp.xpath("//h1/text()")[0]

Funboost provides several concurrency pool implementations to execute tasks using different concurrency models (threads, coroutines, etc.). All pools implement a submit(fn, *args, **kwargs) API, making them similar to the concurrent.futures module.

Available pool types include:

• gevent: Coroutine-based using gevent.
• eventlet: Coroutine-based using eventlet.
• asyncio: Coroutine-based using asyncio.
• CustomThreadPoolExecutor: A variable-bounded thread pool that can automatically expand and shrink the number of threads based on task density.
• FlexibleThreadPool: The framework's default pool. It is a variable-bounded thread pool that supports both standard def functions and async def coroutines.
• FixedThreadPool: A simple fixed-size thread pool for long-running tasks that do not need to exit automatically.
• SoloExecutor: A single-threaded executor (max 1 worker).

def fun(x):
    print(x)

# Example using a pool
pool = FlexibleThreadPool(50)
pool.submit(fun, 1)

Funboost is a distributed function scheduling platform designed to be more powerful and easier to use than Celery. Key advantages include:

• Cross-Platform Support: Full support for Windows, Linux, and macOS (unlike Celery, which has limited Windows support).
• Versatile Brokers: Supports a wider range of message queues including Kafka, NSQ, MQTT, ZeroMQ, RocketMQ, and Pulsar, in addition to everything Celery/Kombu supports (Redis, RabbitMQ, etc.). It can even use memory, files, or databases as brokers.
• High Performance: Claims significantly higher performance in both publishing and consuming tasks.
• IDE Integration: Optimized for PyCharm/VSCode with full auto-completion for methods and parameters (e.g., fun.clear, fun.publish, fun.consume).
• Simplified Execution: No complex CLI commands required; tasks can be started directly via python xx.py.
• Flexible Directory Structure: No strict folder hierarchy or centralized app instance required, avoiding circular import issues common in Celery.
• Native Asyncio Support: Directly supports async def functions without manual loop management.
• Advanced Control: Supports distributed global QPS rate limiting, multi-process + multi-thread/coroutine stacking, and remote deployment via Fabric.

Funboost provides several ways to implement crawling and task scheduling, depending on your requirements for complexity, distributed capabilities, and existing code compatibility.

Recommended Approach

• boost_spider (👑 Preferred): Combines Funboost with RequestClient. It follows a FaaS (Function-as-a-Service) philosophy, offering minimal code and out-of-the-box distributed support. It is the highest-rated option across almost all dimensions including scheduling, state management, and scalability.

Alternative Approaches

• boost_scrapy: A compatibility layer for users migrating from Scrapy, using the Funboost engine with a Scrapy-style API. Recommended only if you have significant existing Scrapy logic.
• Feapder: A specialized distributed crawler framework with strong automatic database entry features.
• Scrapy: The traditional framework; lacks external task injection capabilities and is considered less flexible for modern distributed needs.
• Celery: A general-purpose distributed task queue. While powerful, it is often considered 'overkill' for simple crawling due to complex configuration and worker management.
• ThreadPool / Redis+Thread: Low-level manual implementations (using concurrent.futures or custom Redis logic). These are suitable only for single-machine learning or very simple scripts and lack robust error handling or distributed scaling.

| 方式 | 目录 | 核心技术 | 一句话评价 |
|:---|:---|:---|:---|
| **boost_spider** 👑 | `boost_spider_imp` | Funboost + RequestClient | **首选**！FaaS 降维打击，极简代码，分布式开箱即用 |
| **boost_scrapy** | `boost_scrapy_imp` | Funboost 引擎 + Scrapy 风格 API | 照顾 Scrapy 遗老的兼容层，非必要不推荐 |
| **Feapder** | `feapder_imp` | 国产分布式爬虫框架 | 优秀的垂直框架，自动入库功能很棒 |
| **Scrapy** | `scrapy_imp` | 传统 Scrapy 框架 | 曾经的王者，现在略显过时，无法外部注入任务 |
| **Celery** | `celery_imp` | Celery 分布式任务队列 | 杀鸡用牛刀，配置繁琐，worker 启动麻烦 |
| **Redis+Thread** | `threadpool_redis_crawler_imp` | 手写 Redis + ThreadPool | 400行代码实现1行功能，维护噩梦 |
| **ThreadPool** | `threadpool_crawler_imp` | Python concurrent.futures | 单机玩具，进程死任务丢，仅限学习 |