Flask-RQ

repository·main·Indexed 42 days ago

https://github.com/pallets-eco/flask-rq

An extension for Flask and Quart applications that integrates RQ (Redis Queue) to handle background job execution. It allows developers to offload long-running tasks to separate worker processes while automatically maintaining the application context for services like database connections. Features include support for multiple queues, cron-style periodic scheduling, and delayed job execution via enqueue_at and enqueue_in.

Tokens
8.7K
Snippets
41
Records
54
Agent score
51%

What's inside Flask-RQ

  1. What is Flask-RQ?

    main

    Flask-RQ is an extension for Flask and Quart that enables background job execution using RQ.

    It provides several key benefits:

    • Non-blocking execution: Queue functions to run in separate worker processes, allowing long-running tasks to run in the background without delaying web responses.
    • Application Context Integration: Automatically handles executing jobs within the application context, ensuring that services like database connections are available to the background task.
    • Configuration: Allows configuring RQ settings directly through Flask's configuration object.
    • Scheduling: Supports Cron-style scheduling for periodic jobs via RQ.
  2. Enqueue jobs using the @RQ.job decorator

    main

    When you decorate a function with @RQ.job, it returns a JobWrapper instance. While you should not import JobWrapper directly, you can use its public methods to schedule the decorated function as a background task.

    Key methods available on the decorated function include:

    • enqueue(): Enqueue the function for immediate execution.
    • enqueue_at(datetime): Enqueue the function to run at a specific time.
    • enqueue_in(timedelta): Enqueue the function to run after a specific delay.
    • cron_register(...): Register the function as a recurring task using cron syntax.
    • __call__(): Calling the function directly executes it synchronously (not as a background job).
    from flask_rq import RQ
    
    rq = RQ()
    
    @rq.job
    def my_background_task(arg1, arg2):
        print(f"Running task with {arg1} and {arg2}")
    
    # Usage examples:
    # 1. Immediate execution
    my_background_task.enqueue(1, 2)
    
    # 2. Scheduled execution
    from datetime import datetime, timedelta
    my_background_task.enqueue_at(datetime(2025, 1, 1, 12, 0, 0), 1, 2)
    
    # 3. Delayed execution
    my_background_task.enqueue_in(timedelta(minutes=5), 1, 2)
  3. How Flask-RQ works with Flask and RQ

    main

    Flask-RQ is an extension for Flask and Quart that leverages RQ to execute functions in separate worker processes. This prevents long-running tasks from blocking the web application's response cycle.

    Key features include:

    • Configuration: Uses Flask's standard configuration system to set up RQ.
    • Application Context: Automatically executes jobs within the application context, ensuring that services like database connections are available during job execution.
  4. Difference between Future Jobs and Cron Scheduled Jobs

    main

    It is important to distinguish between the two scheduling mechanisms in RQ:

    FeatureFuture Jobs (Basic Scheduler)Cron Scheduled Jobs
    ProcessRuns as part of the workerRuns as a separate process
    MechanismDelays an already enqueued job until a specific time/intervalPeriodically enqueues new jobs based on a schedule
    RequirementWorker must run with --with-schedulerMust run flask rq cron
    Use Case"Run this specific task in 30 minutes""Run this task every day at 2 AM"
  5. Configure synchronous job execution for testing

    main

    When testing, you can configure RQ to execute jobs immediately in the local process instead of requiring a separate worker process. This is useful for simplifying test flows.

    Flask-RQ enables this mode automatically when app.testing is set to True. You can also explicitly control this behavior using the RQ_ASYNC configuration key. Note that even in async mode, a Redis server must still be running to handle RQ's bookkeeping.

    # Enable async mode via config
    app.config['RQ_ASYNC'] = True
    
    # Or rely on Flask's testing mode
    app.testing = True
  6. Use the @rq.job decorator for easier enqueuing

    main

    The @rq.job(queue="...") decorator wraps a function to provide convenient .enqueue() methods that automatically use the extension instance and the specified queue name.

    Benefits:

    • Type Safety: The wrapper retains the static type signature of the original function, allowing type checkers to validate arguments (unlike the generic queue.enqueue call).
    • Direct Execution: You can call the wrapped function directly (e.g., await func(...)) to execute it synchronously/locally instead of enqueuing it.

    Warning: Because of how RQ references jobs, you cannot pass a decorated function directly into queue.enqueue. Instead, use the wrapper's own .enqueue() method or pass the underlying function via the .func attribute.

    @rq.job(queue="email")
    async def send_password_reset(user_id: int) -> None:
        ...
    
    # Use the wrapper's method (Recommended)
    send_password_reset.enqueue(user_id=user.id)
    
    # Or use the underlying function with a queue
    # rq.queues["email"].enqueue(send_password_reset.func, user_id=user.id)
    
    # To execute the function directly (not as a job)
    await send_password_reset(user_id=user.id)
  7. How Flask-RQ workers work

    main

    Workers are separate processes that monitor RQ queues and execute enqueued jobs. At least one worker must be running for jobs to actually execute; otherwise, jobs will simply sit in the queue without error.

    Flask-RQ provides a specialized worker that ensures every job runs within a Flask application context. This allows jobs to access current_app, database connections, and other Flask extensions exactly as they would inside a view function or CLI command.

    Platform Compatibility: Unlike standard RQ workers, Flask-RQ does not rely on fork or spawn, making it compatible with Windows, macOS, and Linux. However, stopping workers on Windows can be unreliable; if you encounter issues, consider lowering the --worker-ttl or running the worker in a container or WSL.

  8. Configure which queues a worker watches

    main

    By default, a Flask-RQ worker watches all queues defined in your RQ_QUEUES configuration in the order they are listed.

    You can restrict a worker to specific queues by passing them as arguments to the CLI command or to the rq.make_worker() method. When multiple queues are specified, the worker uses the connection associated with the first queue in the list.

  9. Manage a Redis test server with Pytest

    main

    For high-fidelity testing that mimics production, run a real local RQ server. The recommended approach using pytest is to:

    1. Use a session scoped fixture to start redis-server in a subprocess using a reserved ephemeral port.
    2. Use a function scoped fixture to call .flushall() on the Redis instance after each test to ensure a clean state.

    This requires the ephemeral-port-reserve package to avoid port conflicts.

    import collections.abc as cabc
    import subprocess
    import time
    
    import ephemeral_port_reserve
    import pytest
    from redis import Redis
    from redis.exceptions import ConnectionError as RedisConnectionError
    
    @pytest.fixture(scope="session")
    def redis_port() -> int:
        return ephemeral_port_reserve.reserve()  # type: ignore[no-any-return]
    
    
    @pytest.fixture(scope="session", autouse=True)
    def _start_redis(
        tmp_path_factory: pytest.TempPathFactory, redis_port: int
    ) -> cabc.Iterator[None]:
        proc = subprocess.Popen(
            ["redis-server", "--port", str(redis_port)],
            cwd=tmp_path_factory.mktemp("redis-server"),
            stdout=subprocess.DEVNULL,
            stderr=subprocess.DEVNULL,
        )
    
        while True:
            try:
                redis = Redis(port=redis_port, single_connection_client=True)
                redis.ping()
                break
            except (ConnectionError, RedisConnectionError):  # pragma: no cover
                time.sleep(0.1)
    
        yield
        proc.terminate()
        proc.wait()
    
    
    @pytest.fixture(autouse=True)
    def _reset_redis(redis_port: int) -> cabc.Iterator[None]:
        yield
        Redis(port=redis_port, single_connection_client=True).flushall()
  10. Handle enqueuing in async functions

    main

    Flask-RQ supports both Flask and Quart. While enqueue uses a synchronous redis.Redis connection, the operation is typically fast enough that it does not block the event loop significantly.

    If you are concerned about blocking in an async def view, you can wrap the call in asyncio.to_thread to run the enqueue operation in a separate thread.

    import asyncio
    
    # Running enqueue in a thread to avoid blocking the event loop
    await asyncio.to_thread(rq.enqueue, send_password_reset, user_id=user.id)