RedBeat Documentation

RedBeat stores task metadata as a JSON blob in a Redis hash. This allows applications to control task execution logic.

By default, the last_run_at field in the metadata corresponds to when RedBeat dispatched the task. However, due to queue latency, you can update this metadata with the actual execution time to ensure subsequent intervals are relative to the last successful execution rather than the last dispatch time.

{
    "last_run_at": {
        "__type__": "datetime",
        "year": 2015,
        "month": 12,
        "day": 29,
        "hour": 16,
        "minute": 45,
        "microsecond": 231
    },
    "total_run_count": 23
}

RedBeat manages task schedules using two primary Redis data structures:

1. Sorted Set (<prefix>schedule): Acts as a priority queue where task keys are sorted by their next scheduled run time (UNIX timestamp as the score).
2. Hash Key: Stores the task definition and metadata.

During each 'tick', RedBeat:

• Verifies lock ownership.
• Identifies due tasks and calculates the next tick's due tasks.
• Retrieves task definitions and metadata.
• Updates metadata and reschedules the task with its next run time.
• Enqueues due tasks for workers via async_apply.
• Calculates sleep time until the next tick.

RedBeat relies on UNIX timestamps (seconds since epoch in UTC) for its internal scheduling via the redbeat:schedule sorted set. To avoid issues with non-UTC timezones, follow these rules:

• Use Timezone-Aware Datetimes: Most datetime objects in RedBeat are timezone-aware.
• Internal Logic: The schedule is always sorted by UTC timestamps.

Key Timezone Behaviors:

• last_run_at: An aware datetime.
• now(): Returns an aware datetime.
• due_at: Returns an aware datetime.
• to_timestamp(): Accepts an aware datetime and returns seconds since epoch (UTC).
• score(): Returns seconds since epoch for due_at.

You can define static tasks using the standard Celery beat_schedule configuration option within your app configuration.

app.conf.beat_schedule = {
    'add-every-30-seconds': {
        'task': 'tasks.add',
        'schedule': 30.0,
        'args': (16, 16)
    },
}

RedBeat stores all entries in a Redis sorted set at <redbeat_key_prefix>:schedule, where the score is the next time the task is due. To enumerate the entire schedule, you must walk this sorted set and load each key using RedBeatSchedulerEntry.from_key().

Note: Do not use RedBeatScheduler.schedule to list tasks; that method is an internal optimization for the Beat process and typically only returns tasks due immediately.

from redbeat import RedBeatSchedulerEntry
from redbeat.schedulers import ensure_conf, get_redis

conf = ensure_conf(app)
# Retrieve all keys from the schedule sorted set
keys = get_redis(app).zrange(conf.schedule_key, 0, -1)
# Load each key into a RedBeatSchedulerEntry object
entries = [RedBeatSchedulerEntry.from_key(key, app=app) for key in keys]

To manually insert a new task into the RedBeat schedule, use the ZADD command on the schedule key. Assuming the default redbeat_key_prefix of 'redbeat:', the key is redbeat:schedule. The score must be the UNIX timestamp representing the next time the task should run.

# Syntax: zadd <prefix>schedule <unix_timestamp> <task_name>
zadd redbeat:schedule 0 new-task-name

If you are running Celery in development mode with the embedded beat inside a worker process, use the --scheduler flag to specify RedBeat:

celery worker --beat --scheduler redbeat.RedBeatScheduler ...

When starting the Celery Beat process, you must specify the RedBeat scheduler using the -S flag.

Replace <celery_app_file_path>.<celery_app_instance_name> with the actual path to your Celery application instance.

celery beat -A <celery_app_file_path>.<celery_app_instance_name> -S redbeat.RedBeatScheduler

RedBeat uses a Redis-based lock to ensure only one node is actively running. You can run multiple nodes as backups to provide high availability.

• Failover: If the active node fails or loses network connectivity, another node will acquire the lock after the redbeat_lock_timeout seconds has passed.
• Recovery: If a previously active node comes back online and finds it no longer holds the lock, it will exit with an error.
• Best Practice: Use a process manager like systemd or supervisord to automatically restart nodes so they can resume their role as backup nodes.

To use RedBeat as your Celery Beat scheduler, install the package using pip:

pip install celery-redbeat

RedBeat uses Redis to store its schedule. You can configure the connection using the following Celery parameters.

Note: As of version 2.4.2, falling back to broker_url or broker_transport_options is deprecated and will be removed in RedBeat 2.5.0. You should explicitly set redbeat_redis_url and redbeat_redis_options.

redbeat_redis_url = 'redis://localhost:6379/0'
redbeat_redis_options = {
    'password': 'your_password',
    'socket_timeout': 5,
    'retry_period': 60,  # Retry connection for 60 seconds; use -1 for infinite
}