DOCSEARCHbyAlgolia
Submit Docs

Chancy: Distributed Task Queue for Python

repository·main·Indexed 42 days ago

https://github.com/tktech/chancy

A simple and flexible distributed task queue and scheduler for Python that uses PostgreSQL as its primary backend. Chancy eliminates the need for additional infrastructure like Redis or RabbitMQ, offering transactional enqueueing, async-first design with asyncio, and built-in monitoring and worker management. Version 0.25.1 includes a core API for managing jobs and queues, a CLI for administration, and an optional SPA dashboard (chancy-ui) for cluster visualization.

Tokens
24.4K
Snippets
97
Records
121
Agent score
54%

What's inside Chancy

  1. Overview of Chancy Features

    main

    Chancy is a distributed task queue and scheduler built on top of PostgreSQL. It is designed to be a minimal-infrastructure alternative to tools like Celery or RabbitMQ.

    Key Capabilities:

    • Robust Jobs: Supports priorities, retries, timeouts, scheduling, global rate limits, memory limits, global uniqueness, error capture, and cancellation.
    • Minimal Infrastructure: Uses only PostgreSQL (requires psycopg3). No separate broker (Redis/RabbitMQ) or monitoring service (Flower) is required.
    • Flexible Concurrency: A single worker can handle multiple queues and mix threads, processes, sub-interpreters, and asyncio jobs.
    • Transactional Enqueueing: Atomically enqueue jobs and their dependent data within a single database transaction.
    • Built-in Dashboard: Includes a real-time dashboard to track jobs, queues, workers, workflows, throughput, and disk usage.

  2. What is Chancy?

    main

    Chancy is a distributed task queue and scheduler for Python built on top of PostgreSQL. It is designed to be a minimal-infrastructure alternative to tools like Celery or RabbitMQ by leveraging Postgres for all core functionality, including scheduling and monitoring.

    Key characteristics include:

    • Minimal Infrastructure: No separate services like Redis or RabbitMQ are required; everything is built on Postgres.
    • Built-in Features: Scheduling, monitoring, and worker management are built directly into the worker, eliminating the need for external tools like Celery Beat or Flower.
    • Async-first: Designed with asyncio at its core, while providing minimal synchronous APIs for legacy integration.
    • Transactional Enqueueing: Allows you to atomically enqueue jobs within the same database transaction as your application data.
    • Flexible Concurrency: A single worker can manage multiple queues and mix different concurrency models (threads, processes, sub-interpreters, and asyncio) within a single workflow.

  3. Ensure Unique Job execution

    main

    To prevent the same job from being queued or running multiple times simultaneously, assign a unique_key using with_unique_key(key).

    Note: While this prevents duplicates from being queued or running, it does not prevent multiple instances of the same job from being completed or failed.

    async with Chancy("postgresql://localhost/postgres") as chancy:
    await chancy.push(greet.job.with_unique_key("greet_alice").with_kwargs(name="Alice"))

  4. Explore Chancy plugin submodules

    main

    Chancy provides a modular plugin system organized into several specialized submodules. Depending on your requirements, you can extend or interact with the following plugin types:

    • chancy.plugins.api: API-related plugin functionality.
    • chancy.plugins.cron: Scheduled task execution via cron.
    • chancy.plugins.leadership: Leadership election or coordination logic.
    • chancy.plugins.metrics: Telemetry and performance monitoring.
    • chancy.plugins.pruner: Resource or data cleanup tasks.
    • chancy.plugins.recovery: System recovery and fault tolerance.
    • chancy.plugins.reprioritize: Task or queue priority management.
    • chancy.plugins.retry: Automatic retry logic for failed operations.
    • chancy.plugins.sentry: Error tracking and reporting via Sentry.
    • chancy.plugins.trigger: Event-driven execution triggers.
    • chancy.plugins.workflow: Complex multi-step workflow orchestration.

  5. Understand Chancy's approach to global state and job execution

    main

    Unlike Celery, Chancy does not support the job.delay() syntax. This design choice is intentional to adhere to a strict rule: Chancy has absolutely no global state.

    By avoiding global state, Chancy minimizes bugs when running multiple different concurrency models within a single application. Users must interact with the Chancy application instance explicitly rather than relying on a global registry.

  6. How Chancy handles future scheduling

    main
    Unlike Celery, which pulls future tasks into a worker's memory (potentially causing memory issues or disabling QoS with large volumes), Chancy stores the execution timestamp directly in the database. This allows any available worker to pick up the job when the timestamp is reached, making it more scalable for large numbers of scheduled tasks.

  7. Using Mixed-mode Workers with Executors

    main
    Chancy allows a single worker to handle different types of workloads by specifying different 'Executors' (pools) for different queues. This enables 'mixed-mode' operation where one worker can simultaneously run thousands of parallel asyncio tasks for I/O bound work while using other cores for CPU-bound work (like PDF generation).

  9. Chancy Dashboard

    main

    Chancy includes a built-in API plugin that provides a real-time dashboard. This eliminates the need for external monitoring tools like Flower.

    The dashboard allows you to monitor:

    • Jobs: Track real-time status, including failed jobs.
    • Queues: Monitor throughput and completion rates.
    • Workers: View active workers and their status.
    • Workflows: Track complex job sequences.
    • Infrastructure: Monitor disk space usage of Chancy tables and indexes.

  10. Use the Workflow plugin to orchestrate tasks

    main
    The chancy.plugins.workflow module provides a framework for defining and executing structured sequences of operations. You can use WorkflowPlugin to integrate these capabilities into your Chancy environment. The core abstractions include Workflow (the container for the process), WorkflowStep (individual units of work), and Sequence (a way to group or order steps).

  12. Build the chancy-ui project for production

    main

    To generate a production build of the chancy-ui application, run npm run build.

    Note: The build output is placed in the ../dist directory. This directory is automatically generated during the build process and should not be committed to version control.

    npm run build