huey

Huey provides integrations for popular web frameworks:

• Django: Supports native integration or via django.tasks, including an admin integration for visibility and management.
• Flask: Provides flask-peewee admin integration.
• Other Frameworks: Can use the stats extension to collect and display task statistics.

The Huey consumer consists of three main components:

1. Master Process: Coordinates the system.
2. Scheduler: Monitors the schedule and enqueues periodic tasks (e.g., cron jobs) when they are ready.
3. Workers: Listen for messages in the queue, check if tasks are revoked or scheduled for later, and execute them. Results are stored in the result store upon completion.

Depending on the worker type selected, the scheduler and workers may run in threads, processes, or greenlets.

MiniHuey provides a lightweight, in-process task queue that runs inside greenlets. It is ideal for WSGI applications performing I/O-bound tasks (like sending emails or making API requests) where you want to avoid the overhead of managing separate consumer processes.

Important Limitations:

• No Persistence: Tasks are not persisted; if the process restarts, enqueued or scheduled tasks are lost.
• No Retries/Revocation: It does not support automatic retries or dynamic task revocation.
• Gevent Requirement: It requires gevent. It is not suitable for asyncio-based applications or CPU/disk-intensive workloads.

If you need persistence or CPU-intensive processing, use the regular Huey instead.

Scheduled Tasks

Ensure the server running the consumer and the producer have synchronized clocks. Huey uses UTC by default; naive datetimes are converted from local time to UTC.

Cronjobs

Both the consumer and scheduler run in UTC by default.

Duplicate Periodic Tasks

When running multiple consumers, only one consumer should be configured to enqueue periodic tasks. All other consumers must be started with the -n or --no-periodic flag to prevent duplicate executions.

By default, Huey uses pickle for serialization. If your Redis instance is shared or network-exposed, use SignedSerializer to add an HMAC signature to every message. This prevents tampered data from being processed by rejecting it with a ValueError.

Note: This only detects tampering; it does not encrypt the data. Task arguments remain visible in Redis. For encryption, subclass Serializer and implement _serialize/_deserialize using a library like cryptography.

from huey import RedisHuey
from huey.serializer import SignedSerializer

huey = RedisHuey(
    'my-app',
    serializer=SignedSerializer(secret='my-secret-key'))

Huey supports multiple storage backends including Redis (or Valkey/Redict), Postgres, Sqlite, File-system, and in-memory. To use Redis, you must install redis-py. To use Postgres, you must install psycopg.

Initialize a Huey instance by choosing the appropriate class for your storage backend (e.g., RedisHuey, SqliteHuey, PostgresHuey, FileHuey).

from huey import RedisHuey

# Initialize RedisHuey with a name and host
huey = RedisHuey('my-app', host='redis.myapp.com')

To create a task, use the @huey.task() decorator on a function. When the function is called, it returns a Result handle immediately while the actual work is enqueued for a consumer process.

To run the consumer, use the huey_consumer CLI command, providing the import path to your huey instance (e.g., module_name.huey_instance_name).

# demo.py
from huey import SqliteHuey

huey = SqliteHuey(filename='/tmp/demo.db')

@huey.task()
def add(a, b):
    return a + b

huey_consumer demo.huey

When using Docker Compose, note the following:

• Scaling Workers: You can scale workers using docker compose up --scale worker=3. However, because scaled workers will all attempt to enqueue periodic tasks, you should run one dedicated consumer for periodic tasks and start the scaled workers with the -n or --no-periodic flag.
• Storage Backends: Multiple containers can only share a queue via network-accessible storage like Redis or Postgres. SqliteHuey and FileHuey are not recommended for multi-container setups unless using shared volumes (and even then, avoid SQLite over network filesystems).

You can install the base huey package from PyPI. The sqlite, file-system, and in-memory storage layers are included in the base installation and require no additional dependencies.

pip install huey

To run periodic tasks or use the .schedule() method, you must call mini_huey.start(). This starts the scheduler in a new green thread and returns immediately.

Critical Requirement: You must apply the gevent monkey-patch (monkey.patch_all()) before importing your application or any other modules to ensure compatibility.

# Always apply gevent monkey-patch before anything else!
from gevent import monkey; monkey.patch_all()

from my_app import app  # flask/bottle/whatever WSGI app.
from my_app import mini_huey

# Start the scheduler. Returns immediately.
mini_huey.start()

# Run the WSGI server.
from gevent.pywsgi import WSGIServer
WSGIServer(('127.0.0.1', 8000), app).serve_forever()

Use supervisord to manage the huey consumer process. If your application module is not in the default Python path, you must specify the PYTHONPATH in the environment or set the directory to your project root.

After making changes to your configuration, apply them using supervisorctl reread && supervisorctl update.