Thumbor Deployment Recipes

To deploy Thumbor on GKE, ensure your local environment is authenticated with Google Cloud and has kubectl installed. The provided gke-*.yaml examples are tested with version 1.11.5-gke.5.

1. Environment Setup

Run the following commands to authenticate and install necessary components:

gcloud auth login
gcloud auth configure-docker
gcloud components install kubectl

2. Apply Configuration

Apply the configuration files to your cluster using kubectl:

kubectl apply -f gce-simple-ingress.yaml

3. Configure Cloud Storage Loader

These examples use the thumbor-cloud-storage loader to fetch images. You must configure the following environment variables in your deployment to match your Google Cloud project and bucket settings:

• LOADER: Set to thumbor_cloud_storage.loaders.cloud_storage_loader
• CLOUD_STORAGE_PROJECT_ID: Your Google Cloud project ID
• CLOUD_STORAGE_BUCKET_ID: Your GCS bucket ID

- name: LOADER
  value: "thumbor_cloud_storage.loaders.cloud_storage_loader"
- name: CLOUD_STORAGE_PROJECT_ID
  value: "your-project-id"
- name: CLOUD_STORAGE_BUCKET_ID
  value: "your.bucket.id"

The benchmark suite uses Locust in a distributed master/slave architecture.

Locust Master

Configured as the locust service. It coordinates the test run. It uses the following environment variables to pass configuration to the thumbor service:

• THUMBOR_DOCKER_PROCS: Number of processes for the docker environment.
• THUMBOR_NUM_PROCESSES: Number of processes for thumbor.

Key command flags used:

• --host=http://thumbor: Targets the thumbor service.
• -f /opt/scripts/benchmark.py: The benchmark script location.
• --master: Runs the instance in master mode.
• --expect-slaves 3: Specifies the expected number of slave nodes.
• --only-summary: Generates only a summary report.
• --csv=/opt/reports/...: Saves results to the reports volume.

Locust Slave

Configured as the locust-slave service. You can scale the number of slaves using docker-compose scale.

Key command flags used:

• --slave: Runs the instance in slave mode.
• --master-host=locust: Points the slave to the master service.

services:
  locust:
    # ... configuration ...
    command: "locust --host=http://thumbor -f /opt/scripts/benchmark.py --no-web -c 200 -r 4 --run-time 2m --master --expect-slaves 3 --only-summary --csv=/opt/reports/${THUMBOR_DOCKER_PROCS}-${THUMBOR_NUM_PROCESSES}"

  locust-slave:
    # ... configuration ...
    command: "locust --host=http://thumbor -f /opt/scripts/benchmark.py --no-web --slave --master-host=locust"

The ghcr.io/minimalcompact/thumbor-nginx-proxy-cache image provides an automated caching layer. You can tune the cache performance using the following environment variables:

• DEFAULT_HOST: Setting this to the same value as VIRTUAL_HOST ensures the proxy works correctly regardless of the hostname (useful for testing).
• PROXY_CACHE_SIZE: Controls the total cache size on disk (default: 10g).
• PROXY_CACHE_MEMORY_SIZE: Controls the cache memory size (default: 500m).
• PROXY_CACHE_INACTIVE: Controls the inactive cache time (default: 300m).

Important Requirements:

• You must mount /var/run/docker.sock:/tmp/docker.sock:ro so the proxy can detect containers and handle scaling.
• You should map a local directory to /var/cache/nginx to persist the cache independently of the container lifecycle.

services:
  nginx-proxy:
    image: ghcr.io/minimalcompact/thumbor-nginx-proxy-cache
    environment:
      - DEFAULT_HOST=localhost
      - PROXY_CACHE_SIZE=10g
      - PROXY_CACHE_MEMORY_SIZE=500m
      - PROXY_CACHE_INACTIVE=300m
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./cache:/var/cache/nginx
    ports:
      - "80:80"
      - "443:443"

When running Thumbor via Docker Compose for benchmarking purposes, you can tune its performance and behavior using the following environment variables:

• THUMBOR_NUM_PROCESSES: Controls the number of processes (defaults to 1).
• CORS_ALLOW_ORIGIN: Sets the allowed CORS origins (set to * in this configuration to allow any origin).
• AUTO_WEBP: Enables automatic WebP conversion if the browser's Accept headers match (set to True).
• RESULT_STORAGE: Defines the storage backend for processed results. In this benchmark setup, it uses thumbor.result_storages.no_storage to measure raw processing speed without caching.
• RESULT_STORAGE_STORES_UNSAFE: Set to True to allow the specified result storage.
• STORAGE: Defines the storage backend for source images. This configuration uses thumbor.storages.file_storage.

Additionally, a volume mapping ./data:/data is used to store cached images.

services:
  thumbor:
    image: ghcr.io/minimalcompact/thumbor
    environment:
      - THUMBOR_NUM_PROCESSES=${THUMBOR_NUM_PROCESSES:-1}
      - CORS_ALLOW_ORIGIN=*
      - AUTO_WEBP=True
      - RESULT_STORAGE=thumbor.result_storages.no_storage
      - RESULT_STORAGE_STORES_UNSAFE=True
      - STORAGE=thumbor.storages.file_storage
    volumes:
      - ./data:/data

When running the ghcr.io/minimalcompact/thumbor image, you can control its behavior using several environment variables. Key configurations include:

• VIRTUAL_HOST: The domain name for the service (e.g., thumbor.example.com). This is used by nginx-proxy to route traffic.
• THUMBOR_NUM_PROCESSES: Controls the number of processes running inside the container. This should typically align with the number of available CPU cores.
• CORS_ALLOW_ORIGIN: Defines allowed origins for CORS. Use * to allow any origin.
• AUTO_WEBP: Set to True to return WebP images if the browser's Accept headers match.
• RESULT_STORAGE: Specifies the result storage backend. Using thumbor.result_storages.no_storage offloads caching to the proxy layer.
• RESULT_STORAGE_STORES_UNSAFE: Boolean to allow unsafe storage settings.
• STORAGE: The primary storage backend (e.g., thumbor.storages.mixed_storage).
• MIXED_STORAGE_DETECTOR_STORAGE: Specifies the detector storage backend when using mixed storage.
• DETECTORS: A list of detectors to use, such as ['thumbor.detectors.queued_detector.queued_complete_detector'] to utilize all available detectors.

Redis Settings for Thumbor:

• REDIS_STORAGE_SERVER_HOST: Host for the Redis storage server.
• REDIS_STORAGE_SERVER_PORT: Port for the Redis storage server.
• REDIS_STORAGE_SERVER_DB: Database index for the Redis storage server.
• REDIS_QUEUE_SERVER_HOST: Host for the Redis queue server.
• REDIS_QUEUE_SERVER_PORT: Port for the Redis queue server.
• REDIS_QUEUE_SERVER_DB: Database index for the Redis queue server.

services:
  thumbor:
    image: ghcr.io/minimalcompact/thumbor
    environment:
      - VIRTUAL_HOST=localhost
      - THUMBOR_NUM_PROCESSES=4
      - CORS_ALLOW_ORIGIN=*
      - AUTO_WEBP=True
      - RESULT_STORAGE=thumbor.result_storages.no_storage
      - RESULT_STORAGE_STORES_UNSAFE=True
      - STORAGE=thumbor.storages.mixed_storage
      - MIXED_STORAGE_DETECTOR_STORAGE=tc_redis.storages.redis_storage
      - DETECTORS=['thumbor.detectors.queued_detector.queued_complete_detector']
      - REDIS_STORAGE_SERVER_HOST=redis
      - REDIS_STORAGE_SERVER_PORT=6379
      - REDIS_STORAGE_SERVER_DB=0
      - REDIS_QUEUE_SERVER_HOST=redis
      - REDIS_QUEUE_SERVER_PORT=6379
      - REDIS_QUEUE_SERVER_DB=0

When deploying Thumbor using Docker Compose, you can configure its behavior using environment variables. Key configuration categories include:

Scaling and Networking

• VIRTUAL_HOST: The domain name used by nginx-proxy to route traffic (e.g., thumbor.example.com).
• THUMBOR_NUM_PROCESSES: Controls the number of processes running inside a single container. This should typically align with the number of CPU cores. Note that you can also scale instances dynamically using the docker-compose scale command.

Security

• SECURITY_KEY: A required security key for signing URLs.
• ALLOW_UNSAFE_URL: Set to False to prevent unsafe URL access.
• CORS_ALLOW_ORIGIN: Defines allowed origins for Cross-Origin Resource Sharing (e.g., * for all origins).

AWS and S3 Storage

• LOADER: Specifies the loader class (e.g., tc_aws.loaders.s3_loader).
• TC_AWS_LOADER_BUCKET: The name of your S3 bucket.
• AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY: AWS credentials.
• S3_USE_SIGV4: (Optional) Set to true for buckets requiring the v4 signing algorithm.
• TC_AWS_REGION: (Optional) The AWS region (e.g., eu-central-1).

Storage and Caching

• STORAGE: The storage class (e.g., thumbor.storages.mixed_storage).
• RESULT_STORAGE: The storage class for processed images (e.g., thumbor.result_storages.no_storage).
• STORAGE_EXPIRATION_SECONDS: Expiration time for original image files in cache.
• RESULT_STORAGE_EXPIRATION_SECONDS: Expiration time for processed image files in cache.

Redis Configuration

• REDIS_STORAGE_SERVER_HOST / PORT / DB: Connection details for the storage Redis.
• REDIS_QUEUE_SERVER_HOST / PORT / DB: Connection details for the queue Redis.

services:
  thumbor:
    image: ghcr.io/minimalcompact/thumbor
    environment:
      - VIRTUAL_HOST=localhost
      - THUMBOR_NUM_PROCESSES=4
      - SECURITY_KEY=your_secret_key
      - ALLOW_UNSAFE_URL=False
      - CORS_ALLOW_ORIGIN=*
      - AWS_ACCESS_KEY_ID=your_key
      - AWS_SECRET_ACCESS_KEY=your_secret
      - TC_AWS_LOADER_BUCKET=your_bucket
      - LOADER=tc_aws.loaders.s3_loader
      - STORAGE=thumbor.storages.mixed_storage
      - RESULT_STORAGE=thumbor.result_storages.no_storage
      - REDIS_STORAGE_SERVER_HOST=redis
      - REDIS_STORAGE_SERVER_PORT=6379