ProxyPool

ProxyPool is a Python-based proxy pool designed for web crawlers with the following capabilities:

• Multi-source Collection: Built-in support for 15+ free proxy sources with support for custom extensions.
• Automatic Validation: Automatically checks HTTP/HTTPS availability to ensure proxy quality and remove dead proxies.
• Persistent Storage: Uses Redis or SSDB for data persistence and supports cluster deployment.
• RESTful API: Ready-to-use endpoints like /get, /pop, /all, /count, and /delete.
• Docker Deployment: Supports one-command startup via docker-compose with an integrated Redis service.
• Scheduled Tasks: Driven by APScheduler to automatically maintain the proxy pool without manual intervention.

The scheduler performs a directory scan of fetcher/sources/ and reloads modules during every collection cycle. This enables hot-updates:

• New files: Automatically discovered and enabled in the next cycle.
• Modified files: The new version of the logic is loaded in the next cycle.
• Deleted files: Removed in the next cycle (it is recommended to disable them via enabled = False or PROXY_FETCHER_EXCLUDE before deletion to avoid errors).

Proxy collection uses a plugin-based architecture. To add or modify proxy sources, you interact with the fetcher/ directory:

1. Base Class: All sources must inherit from BaseFetcher in baseFetcher.py, which provides shared parsing methods.
2. Implementation: Each source is a standalone file in fetcher/sources/ (e.g., zdaye.py, kuaidaili.py). Each must implement the fetch() method and define name, url, and enabled attributes.
3. Automatic Discovery: The scheduler automatically scans the sources/ directory and loads any source where enabled=True.
4. Exclusion: You can temporarily disable specific sources by adding their name to the PROXY_FETCHER_EXCLUDE list in setting.py.

For high-performance requirements, you can bypass the HTTP API and read proxies directly from the underlying database. ProxyPool supports Redis and SSDB.

Both databases use a hash structure where the hash name is defined by the TABLE_NAME configuration key (which defaults to use_proxy).

ProxyPool uses a multi-stage validation process to ensure proxies are functional and identify their capabilities. All validation methods are defined in helper/validator.py using decorators from the ProxyValidator class. A proxy must pass all methods within a specific category for that category to be considered successful.

Validation Stages and Order:

1. preValidator: Called immediately after a proxy is fetched but before availability testing. If this fails, the proxy is discarded.
2. httpValidator: Tests the general availability of the proxy. If all httpValidator methods return True, the proxy is considered 'available' and added to the pool.
3. httpsValidator: Tests if the proxy supports HTTPS. If all httpsValidator methods return True, the proxy's https attribute is set to True; otherwise, it is set to False.

You can run ProxyPool using Docker or Docker Compose. When using docker run, ensure you pass the DB_CONN environment variable to connect to your Redis instance.

# Using Docker Image
docker pull jhao104/proxy_pool
docker run --env DB_CONN=redis://:password@ip:port/0 -p 5010:5010 jhao104/proxy_pool:latest

# Using Docker Compose
docker-compose up -d

You can add new proxy sources by creating a new .py file in the fetcher/sources/ directory. The new class must inherit from BaseFetcher, define name, url, and enabled attributes, and implement a fetch() method that yields proxies in the host:port format.

from fetcher.baseFetcher import BaseFetcher
from util.webRequest import WebRequest

class MyProxyFetcher(BaseFetcher):
    """My custom proxy source""
    name = "myproxy"
    url = "https://www.example.com/"
    enabled = True

    def fetch(self):
        r = WebRequest().get("https://www.example.com/api/proxies")
        for item in r.json:
            yield item["ip"] + ":" + item["port"]

To ensure the stability of your installation or development environment, you can run the test suite.

1. Install test dependencies:

   pip install -r requirements-test.txt

2. Run all tests using pytest:

   pytest

3. Run specific test layers:
   • Unit tests (no external dependencies): pytest tests/unit/
   • API route tests: pytest tests/api/
   • Integration tests (uses fakeredis to simulate Redis/Ssdb): pytest tests/integration/
4. Check coverage:

   pytest --cov=. --cov-report=term-missing

The project provides a docker-compose.yml file that orchestrates both the proxy_pool service and a proxy_redis service. The proxy_pool service is configured to link to proxy_redis and uses the DB_CONN environment variable to connect to the Redis container.

version: '2'
services:
  proxy_pool:
    build: .
    container_name: proxy_pool
    ports:
      - "5010:5010"
    links:
      - proxy_redis
    environment:
      DB_CONN: "redis://@proxy_redis:6379/0"
  proxy_redis:
    image: "redis"
    container_name: proxy_redis

To start the services in detached mode, run:

docker-compose up -d

To use ProxyPool, you need to clone the repository, install the Python dependencies, and configure the settings. The project consists of two main components: a schedule (the scheduler that fetches and validates proxies) and a server (the API service).

# 1. Clone the repository
git clone https://github.com/jhao104/proxy_pool.git
cd proxy_pool

# 2. Install dependencies
pip install -r requirements.txt

# 3. Start the scheduler
python proxyPool.py schedule

# 4. Start the API server (in a separate terminal)
python proxyPool.py server

To use ProxyPool, clone the repository, install the required Python dependencies, and configure the settings.

1. Clone the repository:

   git clone https://github.com/jhao104/proxy_pool.git

2. Install dependencies: Navigate to the project directory and run:

   pip install -r requirements.txt

3. Configure settings: Edit setting.py in the project root to define your API host/port, database connection, and proxy fetcher methods.