DOCSEARCHbyAlgolia
Submit Docs

proxy-scraper-checker

repository·main·Indexed 23 days ago

https://github.com/monosans/proxy-scraper-checker

A high-performance proxy scraper and checker written in Rust. It collects proxies from various sources, verifies their functionality via HTTP requests, and enriches results with geolocation and network metadata. Supports output in JSON and TXT formats, and can be deployed via pre-built binaries, Docker Compose, or Termux on Android.

Tokens
6.5K
Snippets
17
Records
38
Agent score
79%

What's inside proxy-scraper-checker

  1. Run proxy-scraper-checker using Docker Compose

    main

    You can run the tool in a containerized environment. Note that the Docker image logs to stdout instead of showing the interactive TUI.

    1. Install Docker Compose.
    2. Download: Get the proxy-scraper-checker-docker- artifact from nightly builds.
    3. Extract: Unzip to a folder and edit the config.toml.
    4. Build and Run:
      • Windows:
        docker compose build
docker compose up --no-log-prefix --remove-orphans
      • Linux/macOS:
        docker compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g) --build-arg GID=$(id -g)
docker compose up --no-log-prefix --remove-orphans

    Important: Results are saved to the ./out directory on the host machine. The output.path setting in config.toml is ignored when running in Docker.

    docker compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g)
docker compose up --no-log-prefix --remove-orphans

  2. Install and run proxy-scraper-checker via pre-built binaries

    main

    The easiest way to use the tool is to download a pre-built binary for your platform.

    1. Download: Get the archive from nightly builds. Look for artifacts starting with proxy-scraper-checker-binary- followed by your platform (e.g., Windows, Linux, macOS).
    2. Extract: Unzip the archive into a dedicated folder.
    3. Configure: Edit the config.toml file included in the archive to set your sources and preferences.
    4. Run: Execute the binary.

    Note on Configuration: The binary looks for config.toml in the current directory. You can override this by setting the PROXY_SCRAPER_CHECKER_CONFIG environment variable to point to a specific file path.

  3. Install and run on Android via Termux

    main

    For Android users, use the Termux app (installed from F-Droid, not Google Play).

    1. Install: Run the automated installation script:
      bash <(curl -fsSL 'https://raw.githubusercontent.com/monosans/proxy-scraper-checker/main/termux.sh')
    2. Configure: Use a text editor like nano to edit the config:
      nano ~/proxy-scraper-checker/config.toml
    3. Run:
      cd ~/proxy-scraper-checker && ./proxy-scraper-checker
    bash <(curl -fsSL 'https://raw.githubusercontent.com/monosans/proxy-scraper-checker/main/termux.sh')

  4. Build proxy-scraper-checker from source

    main

    To build the tool manually using the Rust toolchain:

    1. Install Rust via the official installer.
    2. Clone the repo:
      git clone https://github.com/monosans/proxy-scraper-checker.git
cd proxy-scraper-checker
    3. Build with TUI enabled:
      cargo build --features tui --release --locked
    4. Run with TUI:
      cargo run --features tui --release --locked

    Binary Locations:

    • Linux/macOS: target/release/proxy-scraper-checker
    • Windows: target\release\proxy-scraper-checker.exe
    cargo build --features tui --release --locked

  5. Manage TUI application state with AppState

    main

    The AppState struct maintains the real-time metrics displayed in the TUI. It tracks progress for:

    • IP Databases: asn_db_total, asn_db_downloaded, geo_db_total, and geo_db_downloaded.
    • Scraping: sources_total and sources_scraped (mapped by ProxyType).
    • Proxy Checking: proxies_total, proxies_checked, and proxies_working (mapped by ProxyType).

    The AppMode enum controls the high-level state of the interface:

    • Running: The application is actively scraping and checking.
    • Done: The process has completed successfully.
    • Quit: The application is shutting down.

  6. Run proxy-scraper-checker in TUI mode

    main
    If the tui feature is enabled during compilation, the application will launch a Terminal User Interface using ratatui. This mode provides a visual interface for monitoring the scraping and checking process. The TUI mode initializes a specialized logger (tui_logger) and manages terminal state via a RatatuiRestoreGuard.

  7. Understand the RetryMiddleware behavior

    main

    The RetryMiddleware provides automatic retries for transient network and server errors.

    Retry Conditions:

    • Status Codes: Retries occur on REQUEST_TIMEOUT, TOO_MANY_REQUESTS, INTERNAL_SERVER_ERROR, BAD_GATEWAY, SERVICE_UNAVAILABLE, and GATEWAY_TIMEOUT.
    • Connection Errors: Retries on connection-level errors.

    Retry Logic:

    • Max Retries: The default limit is 2 retries (DEFAULT_MAX_RETRIES).
    • Delay Calculation: Uses exponential backoff with jitter. It also respects the Retry-After header if present in the response. Supported formats for Retry-After include:
      • retry-after-ms (custom header with milliseconds).
      • Retry-After (standard header with seconds).
      • Retry-After (standard header with an HTTP-date).
    • Backoff Parameters:
      • INITIAL_RETRY_DELAY: 500ms
      • MAX_RETRY_DELAY: 8s

  9. How SourceConfig handles simple and detailed URLs

    main

    When defining urls in a scraping protocol, you can use either a simple string or a detailed object:

    1. Simple: A plain string representing the URL.
    2. Detailed: An object containing the url, optional basic_auth, and optional headers (a map of key-value pairs).

    Supported schemes for proxy URLs are http, https, socks4, and socks5.

  10. Execution Workflow of proxy-scraper-checker

    main

    The application follows a specific lifecycle to scrape and validate proxies:

    1. Configuration Loading: Loads settings via config::load_config().
    2. IP Database Download: Downloads ASN and Geo databases (if enabled) via download_ipdb.
    3. Scraping: Scrapes proxy lists from various sources using scraper::scrape_all.
    4. Checking: Validates the scraped proxies using checker::check_all.
    5. Output: Saves the validated proxies to a file via output::save_proxies.

    Depending on the enabled features, the application runs either in a Terminal User Interface (TUI) mode or a standard CLI mode.

  11. Configure the proxy-scraper-checker via TOML

    main

    The application is configured using a TOML file. By default, it looks for a file named config.toml in the current directory. You can override this location by setting the PROXY_SCRAPER_CHECKER_CONFIG environment variable.

    The configuration is divided into four main sections: debug, scraping, checking, and output.

  12. Configure proxy-scraper-checker via Docker Compose

    main

    The proxy-scraper-checker service can be deployed using Docker Compose. The configuration uses environment variables for user identity and mounts specific volumes for persistence and configuration.

    Build Arguments

    • UID: The user ID for the container build process. Defaults to 1000 if not provided.
    • GID: The group ID for the container build process. Defaults to 1000 if not provided.

    Volume Mounts

    • proxy_scraper_checker_cache: A named volume used for caching data at /home/app/.cache/proxy_scraper_checker.
    • ./out: A local directory mapped to /app/out for storing output files.
    • ./config.toml: A local configuration file mapped to /app/config.toml to provide application settings.
    services:
  app:
    build:
      args:
        - "UID=${UID:-1000}"
        - "GID=${GID:-1000}"
      pull: true
    init: true
    volumes:
      - proxy_scraper_checker_cache:/home/app/.cache/proxy_scraper_checker
      - ./out:/app/out
      - ./config.toml:/app/config.toml
volumes:
  proxy_scraper_checker_cache: