Cloudtasker

GCP Cloud Tasks does not allow tasks to be scheduled more than 30 days in the future. If you need to run jobs on a monthly or quarterly basis, schedule the job to run daily and implement logic within the worker to check if the current date matches your target interval.

Monthly Job Example (Runs every day at 8am, but only executes logic on the 1st):

# Cron schedule: 0 8 * * *
class MyMonthlyWorker
  include Cloudtasker::Worker

  def perform(*args)
    return unless Time.current.day == 1
    # ... job logic
  end
end

Quarterly Job Example (Runs every day at 8am, but only executes logic on the first month of a quarter):

# Cron schedule: 0 8 * * *
class MyQuarterlyWorker
  include Cloudtasker::Worker

  def perform(*args)
    return unless Time.current.month == 1 # Jan, Apr, Jul, Oct
    return unless Time.current.day == 1
    # ... job logic
  end
end

Cloudtasker uses two distinct identifiers to help you trace jobs through logs and Google Cloud Tasks:

Workflow for debugging:

1. Find a specific task in the GCP Console using its Task name (task_id).
2. Search your logs for that task_id to find the associated job_id.
3. Use the job_id to search for all logs related to that job's entire history (including previous attempts/reschedules).

Cloudtasker Batch Jobs allow you to add sub-jobs to regular jobs, enabling you to enqueue a list of jobs and track their overall progression as a group (a "batch").

Key behaviors:

• Inheritance: Jobs added to a batch inherit the queue of the parent by default. You can override this using batch.add_to_queue.
• Completion: A batch is considered complete when all children have either successfully completed or died (exhausted all retries).
• Retries: If a child job fails, it will be retried based on your max_retries configuration. The batch remains in a pending state while retries are occurring. To prevent batches from hanging, consider reducing retries using cloudtasker_options max_retries: N on your workers.

When setting max_concurrent_dispatches in Cloud Tasks, you must ensure the total concurrency does not exceed your application's available resources (threads or database connections).

Server-based applications

Calculate your total available threads (e.g., number_of_instances * RAILS_MAX_THREADS). Ensure the sum of the concurrency of all your queues (e.g., default + critical) is less than your total threads to leave capacity for web users. Always ensure total concurrency does not exceed your database connection pool size.

Serverless applications

In serverless environments, scaling is driven by traffic (including Cloud Task requests). Since capacity is theoretically unlimited, your primary limiting factor is usually the database connection pool size. Use this limit as the capping parameter for your queue concurrency calculations.

The Storable Jobs extension (available since v0.14.0) allows you to 'park' jobs in a specific storage lane (a store_name) and 'pull' them into the active queue later when specific programmatic conditions are met.

This is a more efficient alternative to using guard logic inside a perform method (which re-enqueues jobs repeatedly and pollutes logs). Instead, you store the job arguments in Redis and only trigger the actual enqueuing when you are ready to process them.

When using Puma in cluster-mode with preload_app!, scheduling cron jobs (which triggers gRPC calls) should be delayed until after the Puma process has booted to avoid issues with forking and gRPC. Use the on_booted block in config/puma.rb to load your schedules.

# config/puma.rb

workers ENV.fetch("WEB_CONCURRENCY") { 2 }
preload_app!

on_booted do
  schedule_file = "config/cloudtasker_cron.yml"
  if File.exist?(schedule_file) && !Rails.env.test?
    Cloudtasker::Cron::Schedule.load_from_hash!(YAML.load_file(schedule_file))
  end
end

To avoid needing a public domain for local development, you can use the Cloudtasker local server. This is a Ruby daemon that mimics Google Cloud Tasks by reading jobs from Redis and sending them to your application via HTTP POST requests.

Setup

1. Set config.mode = :development in your Cloudtasker initializer.
2. Start the server using bundle exec cloudtasker or via a Procfile using foreman.

Example Procfile

web: bundle exec rails s
worker: bundle exec cloudtasker

# config/initializers/cloudtasker.rb
Cloudtasker.configure do |config|
  # Use Redis-based local processing
  config.mode = :development
end

To test your application end-to-end with actual Google Cloud Tasks, use ngrok to create a public tunnel to your local machine.

Setup

1. Start an ngrok tunnel: ngrok http 3000.
2. Configure Cloudtasker in your initializer to use :production mode, specify your GCP credentials, and set config.processor_host to your ngrok URL.
3. Start your Rails server.

# config/initializers/cloudtasker.rb
Cloudtasker.configure do |config|
  config.gcp_location_id = 'us-central1'
  config.gcp_project_id = 'my-gcp-project'
  config.gcp_queue_prefix = 'my-app'

  # Use your ngrok domain as the processor host
  config.processor_host = 'https://your-tunnel-id.ngrok.io'

  # Force Cloudtasker to use Google Cloud Tasks in development
  config.mode = :production
end