DOCSEARCHbyAlgolia
Submit Docs

CloudProxy

repository·main·Indexed 23 days ago

https://github.com/claffin/cloudproxy

A tool to manage cloud-based proxies for scraping by spinning up a pool of proxies across providers such as DigitalOcean, AWS, GCP, Hetzner, and Vultr. It features a REST API for retrieving random proxies, a web interface for monitoring and scaling, and support for rolling deployments to ensure zero-downtime proxy recycling. Version 0.6.23.

Tokens
27.7K
Snippets
74
Records
126
Agent score
82%

What's inside cloudproxy

  1. How Multi-Account Support works in CloudProxy

    main

    CloudProxy allows running multiple DigitalOcean accounts simultaneously by defining them as separate "instances".

    • The Default Instance is configured using standard variables (e.g., DIGITALOCEAN_ENABLED).
    • Additional Instances are configured by prefixing the variable name with a custom instance name (e.g., DIGITALOCEAN_USEAST_ENABLED).

    Each instance operates independently with its own proxy pool and settings.

    # Default Instance
DIGITALOCEAN_ENABLED=True
DIGITALOCEAN_ACCESS_TOKEN=your_default_token
DIGITALOCEAN_REGION=lon1
DIGITALOCEAN_MIN_SCALING=2

# Additional Instance (USEAST)
DIGITALOCEAN_USEAST_ENABLED=True
DIGITALOCEAN_USEAST_ACCESS_TOKEN=your_second_token
DIGITALOCEAN_USEAST_REGION=nyc1
DIGITALOCEAN_USEAST_MIN_SCALING=3
DIGITALOCEAN_USEAST_SIZE=s-1vcpu-1gb
DIGITALOCEAN_USEAST_DISPLAY_NAME=US East Account

  2. Configure Multi-Account Provider Support

    main

    CloudProxy allows you to manage multiple accounts for a single provider (e.g., two different DigitalOcean accounts) by using an instance-specific naming convention for environment variables: PROVIDERNAME_INSTANCENAME_VARIABLE.

    For example, to configure a second DigitalOcean account named SECONDACCOUNT:

    • DIGITALOCEAN_SECONDACCOUNT_ENABLED=True
    • DIGITALOCEAN_SECONDACCOUNT_ACCESS_TOKEN=your_second_token
    • DIGITALOCEAN_SECONDACCOUNT_REGION=nyc1
    • DIGITALOCEAN_SECONDACCOUNT_MIN_SCALING=3
    # Default DigitalOcean account
DIGITALOCEAN_ENABLED=True
DIGITALOCEAN_ACCESS_TOKEN=your_first_token
DIGITALOCEAN_DEFAULT_REGION=lon1
DIGITALOCEAN_DEFAULT_MIN_SCALING=2

# Second DigitalOcean account
DIGITALOCEAN_SECONDACCOUNT_ENABLED=True
DIGITALOCEAN_SECONDACCOUNT_ACCESS_TOKEN=your_second_token
DIGITALOCEAN_SECONDACCOUNT_REGION=nyc1
DIGITALOCEAN_SECONDACCOUNT_MIN_SCALING=3

  3. How multi-account Hetzner support works

    main

    CloudProxy allows you to run multiple Hetzner accounts simultaneously by defining them as separate "instances".

    Each instance is configured using a prefix based on an instance name. The default configuration (e.g., HETZNER_ENABLED) applies to the primary instance. For additional accounts, use the pattern HETZNER_<INSTANCENAME>_<VARIABLE>.

    Instance Configuration Pattern

    For every additional instance, you must provide:

    • HETZNER_<INSTANCENAME>_ENABLED: To enable this specific instance.
    • HETZNER_<INSTANCENAME>_API_TOKEN: The API token for this instance.

    You can also provide optional settings for each instance:

    • HETZNER_<INSTANCENAME>_MIN_SCALING
    • HETZNER_<INSTANCENAME>_MAX_SCALING
    • HETZNER_<INSTANCENAME>_SIZE
    • HETZNER_<INSTANCENAME>_LOCATION
    • HETZNER_<INSTANCENAME>_DATACENTER
    • HETZNER_<INSTANCENAME>_DISPLAY_NAME: A friendly name for the UI.

    Each instance operates independently, maintaining its own pool of proxies.

    # Example: Adding a second Hetzner account in Finland
HETZNER_FINLAND_ENABLED=True
HETZNER_FINLAND_API_TOKEN=your_second_token
HETZNER_FINLAND_LOCATION=hel1
HETZNER_FINLAND_MIN_SCALING=3
HETZNER_FINLAND_SIZE=cx11
HETZNER_FINLAND_DISPLAY_NAME=Finland Hetzner Account

  4. How Rolling Deployments work

    main

    Rolling deployments prevent service disruption during age-based proxy recycling by enforcing availability constraints.

    Standard Recycling (Disabled)

    When ROLLING_DEPLOYMENT=False, proxies reaching the age limit are immediately recycled. All aged proxies are deleted simultaneously and new ones are created, which may cause a period of reduced availability.

    Rolling Deployment (Enabled)

    When ROLLING_DEPLOYMENT=True, the system follows these steps:

    1. Identifies proxies that reached the age limit.
    2. Checks if recycling would reduce healthy proxies below ROLLING_MIN_AVAILABLE.
    3. Checks if the number of proxies currently recycling is already at ROLLING_BATCH_SIZE.
    4. If both checks pass, the proxy is recycled. If not, recycling is deferred until conditions improve.

    Example Scenario

    With AGE_LIMIT=3600, ROLLING_DEPLOYMENT=True, ROLLING_MIN_AVAILABLE=3, ROLLING_BATCH_SIZE=2, and DIGITALOCEAN_MIN_SCALING=5:

    • If all 5 droplets reach the age limit, the first 2 are recycled (batch size limit).
    • The remaining 3 stay healthy (minimum availability).
    • Once the first 2 are replaced and healthy, the next batch begins.

  5. Configure Multi-Account AWS Support

    main

    CloudProxy allows running multiple AWS accounts simultaneously. Each account is treated as a separate instance. To configure an additional account, prefix the standard AWS environment variables with a unique instance name (e.g., AWS_EU_).

    Each additional instance must include its own ENABLED, ACCESS_KEY_ID, and SECRET_ACCESS_KEY variables.

    # Example: Adding a second AWS account in a different region
AWS_EU_ENABLED=True
AWS_EU_ACCESS_KEY_ID=your_second_access_key
AWS_EU_SECRET_ACCESS_KEY=your_second_secret_key
AWS_EU_REGION=eu-west-1
AWS_EU_MIN_SCALING=1
AWS_EU_SIZE=t2.micro
AWS_EU_SPOT=True
AWS_EU_DISPLAY_NAME=EU Account

  6. Manage multiple provider instances

    main

    CloudProxy supports multiple instances of the same provider (e.g., multiple DigitalOcean accounts) by using secondary environment variable suffixes. This allows you to use different API keys or regions simultaneously.

    Use the following naming convention for secondary instances:

    • [PROVIDER]_SECONDARY_ENABLED
    • [PROVIDER]_SECONDARY_ACCESS_TOKEN
    • [PROVIDER]_SECONDARY_REGION
    • [PROVIDER]_SECONDARY_MIN_SCALING

    Access these specific instances using manager.get_provider_instance_ips(provider, instance_name).

    import os
from cloudproxy.providers import manager

# Setup the first DigitalOcean instance (default)
os.environ["DIGITALOCEAN_ENABLED"] = "True"
os.environ["DIGITALOCEAN_ACCESS_TOKEN"] = "first_token"
os.environ["DIGITALOCEAN_REGION"] = "lon1"
os.environ["DIGITALOCEAN_MIN_SCALING"] = "2"

# Setup a second DigitalOcean instance
os.environ["DIGITALOCEAN_SECONDARY_ENABLED"] = "True"
os.environ["DIGITALOCEAN_SECONDARY_ACCESS_TOKEN"] = "second_token"
os.environ["DIGITALOCEAN_SECONDARY_REGION"] = "nyc1"
os.environ["DIGITALOCEAN_SECONDARY_MIN_SCALING"] = "3"

# Initialize the manager
manager.init_schedule()

# Get all proxies from the first instance
do_proxies = manager.get_provider_instance_ips("digitalocean", "default")

# Get all proxies from the second instance
do_secondary_proxies = manager.get_provider_instance_ips("digitalocean", "secondary")

  7. How Multi-Account Vultr Support works

    main

    CloudProxy allows you to run multiple Vultr accounts simultaneously. Each account is treated as a separate "instance" with its own independent pool of proxies.

    To configure an additional account, prefix the standard Vultr environment variables with an instance name (e.g., VULTR_EUROPE_).

    Required variables per instance:

    • VULTR_{INSTANCENAME}_ENABLED
    • VULTR_{INSTANCENAME}_API_TOKEN
    • VULTR_{INSTANCENAME}_REGION

    Optional variables per instance:

    • VULTR_{INSTANCENAME}_MIN_SCALING
    • VULTR_{INSTANCENAME}_MAX_SCALING
    • VULTR_{INSTANCENAME}_PLAN
    • VULTR_{INSTANCENAME}_OS_ID
    • VULTR_{INSTANCENAME}_DISPLAY_NAME
    # European instance
VULTR_EUROPE_ENABLED=True
VULTR_EUROPE_API_TOKEN=your_second_token
VULTR_EUROPE_REGION=ams
VULTR_EUROPE_MIN_SCALING=3
VULTR_EUROPE_PLAN=vc2-1c-1gb
VULTR_EUROPE_DISPLAY_NAME=Europe Proxies

# Asia Pacific instance
VULTR_ASIA_ENABLED=True
VULTR_ASIA_API_TOKEN=your_third_token
VULTR_ASIA_REGION=sgp
VULTR_ASIA_MIN_SCALING=2
VULTR_ASIA_PLAN=vc2-1c-2gb
VULTR_ASIA_DISPLAY_NAME=Asia Proxies

  8. Configure Proxy Authentication

    main

    While the CloudProxy API itself is unauthenticated, the proxy servers themselves use authentication to restrict access.

    Authentication Methods

    1. Basic Auth: Set PROXY_USERNAME and PROXY_PASSWORD environment variables. Proxies will be accessible via: http://username:password@proxy-ip:port.
    2. IP-based Auth: Set ONLY_HOST_IP=True to restrict access to the host server's IP address.
    3. Combined Auth: Use both username/password and IP restriction for maximum security.

  9. Understand CloudProxy API Response Formats

    main

    All API responses follow a standardized structure.

    Metadata Object

    Included in every response to provide request tracking and timing.

    {
  "metadata": {
    "request_id": "123e4567-e89b-12d3-a456-426614174000",
    "timestamp": "2024-02-24T08:00:00Z"
  }
}

    Proxy Object

    Represents an individual proxy server.

    {
  "ip": "192.168.1.1",
  "port": 8899,
  "auth_enabled": true,
  "url": "http://username:password@192.168.1.1:8899",
  "provider": "digitalocean",
  "instance": "default",
  "display_name": "My DigitalOcean Instance"
}

    Provider and Instance Objects

    Providers contain configuration for a cloud account, while instances represent specific deployment groups within that provider (e.g., different regions or accounts).

  10. Configure Multi-Account GCP Support

    main

    CloudProxy allows running multiple GCP accounts simultaneously by defining separate "instances". Each instance is configured using a prefix based on the instance name.

    Pattern: GCP_{INSTANCENAME}_{VARIABLE}

    To add an additional account (e.g., named EUROPE), prefix the standard GCP variables with EUROPE_.

    # Example: Adding a second GCP account in a different zone
GCP_EUROPE_ENABLED=True
GCP_EUROPE_SA_JSON=/path/to/europe-service-account-key.json
GCP_EUROPE_PROJECT=europe-project-id
GCP_EUROPE_ZONE=europe-west1-b
GCP_EUROPE_MIN_SCALING=1
GCP_EUROPE_SIZE=e2-micro
GCP_EUROPE_DISPLAY_NAME=Europe GCP Account

  11. Set up AWS IAM permissions for CloudProxy

    main

    Required IAM Permissions

    EC2 (for managing instances):

    • DescribeInstances
    • RunInstances
    • TerminateInstances
    • CreateTags
    • DescribeImages
    • DescribeInstanceStatus
    • DescribeInstanceTypes
    • DescribeAvailabilityZones
    • DescribeSecurityGroups
    • AuthorizeSecurityGroupIngress
    • CreateSecurityGroup
    • DescribeVpcs

    Systems Manager (for configuring instances):

    • SendCommand

  12. Quick Start: Configure CloudProxy with AWS

    main

    You can quickly set up CloudProxy with AWS using Docker, an environment file, or a local Python environment. Ensure you have provided the necessary AWS credentials and enabled the AWS provider.

    # Run with Docker (recommended)
docker run -d \
  -e PROXY_USERNAME='your_username' \
  -e PROXY_PASSWORD='your_password' \
  -e AWS_ENABLED=True \
  -e AWS_ACCESS_KEY_ID="your-access-key" \
  -e AWS_SECRET_ACCESS_KEY="your-secret-key" \
  -e AWS_REGION="us-east-1" \
  -p 8000:8000 \
  laffin/cloudproxy:latest