DOCSEARCHbyAlgolia
Submit Docs

DJ Celery Panel

repository·main·Indexed 40 days ago

https://github.com/yassi/dj-celery-panel

A Django Admin panel for browsing and inspecting Celery tasks and workers. Version 0.6.0 provides tools to monitor active workers, manage queues, view periodic tasks, and search task execution history directly within the Django Admin interface. It requires Python 3.9+ and Django 4.2+.

Tokens
10.4K
Snippets
33
Records
49
Agent score
98%

What's inside DJ Celery Panel

  1. Overview of DJ Celery Panel features

    main

    DJ Celery Panel provides a suite of monitoring tools directly within the Django Admin interface:

    • Workers Monitoring: View active Celery workers, including status, pool type, and concurrency.
    • Task Management: Browse and inspect Celery tasks with details on status, arguments, results, and execution time.
    • Queue Overview: Monitor configured queues and their routing.
    • Periodic Tasks: View scheduled periodic tasks and their schedules.
    • Real-time Inspection: Access live data via Celery's inspect API.
    • Swappable Backends: Use a pluggable architecture to implement custom data sources or monitoring integrations.

  2. Overview of Django Celery Panel

    main

    Django Celery Panel is a Django admin extension designed to bring Celery monitoring directly into your existing Django Admin interface. It eliminates the need for separate monitoring services or extra infrastructure by integrating directly with your Django project.

    Key capabilities include:

    • Workers Monitoring: View active workers, status, pool types, and concurrency.
    • Task Management: Inspect task history, arguments, results, and error tracebacks.
    • Queue & Periodic Tasks: Monitor configured queues and scheduled tasks.
    • Real-time Data: Uses Celery's inspect API for live information.
    • Extensibility: Uses a swappable backend architecture to allow custom data sources.

  3. Configure Periodic Tasks Backend

    main

    You can choose how periodic tasks are displayed based on how you manage your Celery Beat schedule:

    1. CeleryPeriodicTasksConfigBackend (Default)

      • Use case: When tasks are defined in your Django settings.
      • Requirements: CELERY_BEAT_SCHEDULE must be defined in your settings.
      • Features: Displays schedules (crontab, interval, etc.) and task arguments.

    2. CeleryPeriodicTasksDjangoCeleryBeatBackend

      • Use case: When using django-celery-beat to manage schedules via the database.
      • Requirements: django-celery-beat installed and configured; database migrations applied.
      • Features: Shows task execution history (last run, total runs) and allows dynamic management via Django admin.
    # Example for django-celery-beat users
INSTALLED_APPS = [
    # ...
    'django_celery_beat',
    'dj_celery_panel',
]

DJ_CELERY_PANEL_SETTINGS = {
    "periodic_tasks_backend": "dj_celery_panel.celery_utils.CeleryPeriodicTasksDjangoCeleryBeatBackend",
}

  4. How Scopes and Permissions work in Dj Celery Panel

    main

    Dj Celery Panel uses scopes to manage access control. A scope is a named checkpoint used by the @panel_config.permission_required(scope) decorator to verify if a user can access a specific view or resource.

    Permission Inheritance

    1. Panel-wide Defaults: By default, all scopes follow the global rules defined by ALLOWED_GROUPS and REQUIRE_SUPERUSER in your settings.
    2. Scope-specific Overrides: You can define specific permission rules for individual scopes using the SCOPE_PERMISSIONS dictionary within DJ_CELERY_PANEL_SETTINGS.
    3. Fallback Behavior: If a scope is not explicitly listed in SCOPE_PERMISSIONS, it automatically falls back to the panel-wide default settings. This allows you to only define exceptions for sensitive areas.

  5. Understand Django Celery Panel integration

    main

    Django Celery Panel integrates directly into the standard Django admin interface.

    Key Integration Details:

    • Access Control: Uses Django's existing authentication and the @staff_member_required decorator.
    • Zero Impact on Models: The package does not define any Django models and requires no database migrations.
    • Read-Only: It is designed to be lightweight and only reads data; it does not modify Celery state or your database.
    • UI Consistency: Matches the Django admin's look and feel, including breadcrumbs and navigation.

  6. How swappable backends work in Django Celery Panel

    main

    Django Celery Panel is built with a pluggable architecture using swappable backends. This allows developers to customize how monitoring data is retrieved without modifying the UI components.

    You can start with the default implementations and swap them for custom implementations as your infrastructure grows. This design prevents vendor lock-in and allows you to integrate with existing custom monitoring tools or unique data sources by implementing your own backend logic.

  7. Choose a Tasks Backend

    main

    Depending on whether you need historical data or real-time monitoring, you can choose between two primary task backends:

    1. CeleryTasksDjangoCeleryResultsBackend (Default/Recommended)

      • Use case: Full task history, including completed and failed tasks.
      • Requirements: django-celery-results must be installed and CELERY_RESULT_BACKEND = 'django-db' must be configured.
      • Features: Pagination, search by name/ID, status filtering (Pending, Started, Success, Failure, Retry, Revoked), and full metadata.

    2. CeleryTasksInspectBackend

      • Use case: Real-time monitoring of currently executing (active) tasks only.
      • Requirements: At least one running Celery worker reachable via the broker.
      • Features: No database storage required; optimized for performance via a single fanout request.
    # To use the real-time inspect backend instead of history:
DJ_CELERY_PANEL_SETTINGS = {
    "tasks_backend": "dj_celery_panel.celery_utils.CeleryTasksInspectBackend",
}

  8. Include dj-celery-panel URLs

    main

    Register the panel's URL patterns in your project's main urls.py file. It is recommended to prefix them with admin/dj-celery-panel/ to integrate with the Django admin interface.

    from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/dj-celery-panel/', include('dj_celery_panel.urls')),
    path('admin/', admin.site.urls),
]

  9. Inspect and manage Celery Task history

    main

    The Tasks page provides a searchable history of executed tasks. This feature is powered by django-celery-results, which stores execution history in your Django database.

    Task Monitoring Capabilities:

    • Status Tracking: Visual indicators for PENDING, STARTED, SUCCESS, FAILURE, RETRY, and REVOKED states.
    • Search & Filter: Find tasks by name, ID, or other criteria.
    • Execution Details: View the execution timeline, duration, and which worker processed the task.

    Task Detail View:

    Clicking a task provides:

    • Formatted JSON of all arguments (args) and keyword arguments (kwargs).
    • Complete results or error messages.
    • Full exception tracebacks for failed tasks.
    • Metadata such as retries and time limits.

  10. Monitor Celery Workers in real-time

    main

    The Workers page provides real-time monitoring of all active Celery workers. It uses Celery's inspect API to show current state without relying on database storage.

    Key Information Available:

    • Worker Status: Online/offline status.
    • Identity: Worker name and hostname.
    • Configuration: Pool type (prefork, gevent, solo, eventlet) and concurrency levels.
    • Activity: Currently executing tasks.

    Worker Detail View:

    Clicking a specific worker reveals deep inspection data:

    • Process ID (PID) and resource usage (CPU, memory).
    • Prefetch count and total tasks executed.
    • Active, reserved, and scheduled tasks.
    • Registered tasks and active queues being consumed.
    • Broker connection information.

  11. Set up a local development environment

    main

    To develop locally, you can use either Docker (recommended) or a standard Python virtual environment.

    Option A: Docker (Recommended)

    Use the provided Makefile to manage services:

    • make docker_up: Starts all required services.
    • make docker_shell: Opens a shell inside the container.

    Option B: Local Environment

    1. Create and activate a virtual environment:
      python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
    2. Install the package and its dependencies:
      make install
# OR
pip install -e .
pip install -r requirements.txt

    Initialize the Example Project

    After setting up the environment, initialize the example_project to see the panel in action:

    cd example_project
python manage.py migrate
python manage.py createsuperuser
python manage.py runserver

    Visit http://127.0.0.1:8000/admin/ to access the panel.

    # Example: Local setup sequence
python -m venv venv
source venv/bin/activate
make install
cd example_project
python manage.py migrate
python manage.py runserver

  12. Monitor Celery Beat Periodic Tasks

    main

    The Periodic Tasks page displays all scheduled tasks configured in Celery Beat.

    Features:

    • Schedule Visibility: View task schedules using Interval, Crontab, or Solar types.
    • Task Status: See the task name, whether it is currently enabled, and the timestamp of its last execution.