DOCSEARCHbyAlgolia
Submit Docs

django-prometheus

repository·master·Indexed 23 days ago

https://github.com/django-commons/django-prometheus

A library that exports Django-specific monitoring metrics for Requests, Responses, Database, Cache, and Model operations in a format compatible with Prometheus.io. It supports custom metric namespaces, latency buckets, and multiple export methods including Django views, dedicated daemon threads, and WSGI multiprocess mode for Gunicorn and uWSGI.

Tokens
5.7K
Snippets
13
Records
18
Agent score
82%

What's inside django-prometheus

  1. Quickstart django-prometheus setup

    master

    To enable basic monitoring of requests and responses, follow these three steps:

    1. Add 'django_prometheus' to your INSTALLED_APPS.
    2. Add PrometheusBeforeMiddleware to the beginning of your MIDDLEWARE list and PrometheusAfterMiddleware to the end of your MIDDLEWARE list.
    3. Include django_prometheus.urls in your project's urlpatterns to expose the /metrics endpoint.
    # settings.py
INSTALLED_APPS = [
   ...
   'django_prometheus',
   ...
]

MIDDLEWARE = [
    'django_prometheus.middleware.PrometheusBeforeMiddleware',
    # All your other middlewares go here...
    'django_prometheus.middleware.PrometheusAfterMiddleware',
]

# urls.py
urlpatterns = [
    ...
    path('', include('django_prometheus.urls')),
]

  2. Export /metrics in a dedicated thread

    master

    To prevent Django application issues (like thread starvation or low-level bugs) from affecting your monitoring, you can export /metrics using an HTTPServer running in a daemon thread.

    Important: This mechanism is incompatible with Django's autoreloader because the autoreloader forks multiple processes that would compete for the same port. You must run Django with the --noreload flag.

    To enable this, set PROMETHEUS_METRICS_EXPORT_PORT and PROMETHEUS_METRICS_EXPORT_ADDRESS in your settings.py.

  3. Run a demo Prometheus instance

    master

    To run a local demo Prometheus instance to monitor your Django application, ensure your Django app is exporting metrics (the provided prometheus.yml expects them at http://127.0.0.1:8000/metrics).

    After installing Prometheus, execute the binary with the following flags to point to the configuration file and the required console templates/libraries:

    1. Ensure you have a prometheus.yml file in your current directory.
    2. Run the command below (adjusting the path ~/prometheus to your actual installation location).
    3. Access the Prometheus UI at http://localhost:9090.
    ~/prometheus/prometheus \
  --config.file=prometheus.yml \
  --web.console.templates consoles/ \
  --web.console.libraries ~/prometheus/console_libraries/

  4. Export /metrics as a Django view

    master

    The simplest way to export metrics is via a Django view. You can include the django_prometheus.urls in your urlpatterns.

    By default, this reserves the /metrics path. If you need to use a different path, you can provide a prefix in your URL configuration. If you use a prefix, ensure your Prometheus configuration is updated to scrape the new path.

    # Default: exports at /metrics
urlpatterns = [
    path('', include('django_prometheus.urls')),
]

# With prefix: exports at /monitoring/metrics
urlpatterns = [
    path('monitoring/', include('django_prometheus.urls')),
]

  5. Monitor Django databases

    master

    To monitor SQLite, MySQL, or PostgreSQL, replace the standard Django database ENGINE with the corresponding django_prometheus.db.backends engine.

    DATABASES = {
    'default': {
        'ENGINE': 'django_prometheus.db.backends.sqlite3',  # Replace django.db.backends.sqlite3
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    },
}

  6. Export /metrics globally in WSGI using multiprocess mode

    master

    For WSGI applications where workers are short-lived, use the Prometheus client's multiprocess aggregation system. This uses the PROMETHEUS_MULTIPROC_DIR environment variable to store metrics in files per process.

    Configuration

    Set the PROMETHEUS_MULTIPROC_DIR environment variable (e.g., in a uWSGI config or Kubernetes manifest) to a directory where metrics can be written. It is recommended to use a tmpfs or a directory like /run/ for performance.

    Using uWSGI Worker IDs

    By default, files are named by PID. In uWSGI, where PIDs change frequently due to worker respawns, you can switch to using uwsgi.worker_id to prevent a massive accumulation of files. This must be done in settings.py before any metrics are created.

    Warning: This uses internal prometheus_client interfaces and may change in future versions.

    # uWSGI configuration
env = PROMETHEUS_MULTIPROC_DIR=/path/to/django_metrics
    # settings.py (Must be before metrics creation)
try:
    import prometheus_client
    import uwsgi
    prometheus_client.values.ValueClass = prometheus_client.values.MultiProcessValue(
        process_identifier=uwsgi.worker_id)
except ImportError:
    pass  # not running in uwsgi

  8. Export /metrics in WSGI with multiple processes per process

    master

    When using WSGI servers like uWSGI or Gunicorn with multiple worker processes, using a single Django view or a single dedicated port will result in inconsistent metrics (each request hits a different process).

    To solve this, use PROMETHEUS_METRICS_EXPORT_PORT_RANGE to define a range of ports. Django-Prometheus will attempt to bind to the first port in the range and, if busy, move to the next. You must then configure Prometheus to scrape each port in that range as a separate target.

    Requirement: The application must be loaded into each child process.

    • For uWSGI: Set lazy-apps = true.
    • For Gunicorn: Set preload-app = false.

  9. How to implement a new database wrapper type

    master

    To add support for a new database vendor in django-prometheus, you must implement three specific classes that wrap the standard Django database engine components. This allows the library to intercept database operations and record metrics like error counts.

    Required Classes

    1. DatabaseFeatures class: Describes the features supported by the database. For django-prometheus purposes, you can simply extend the existing DatabaseFeatures class without making changes.
    2. DatabaseWrapper class: Abstracts the interface to the database. This class provides access to self.alias and self.vendor properties.
    3. CursorWrapper class: Abstracts the interface to a database cursor (the object used to execute SQL).

    Implementation Tips

    • Dynamic Cursor Generation: Because the CursorWrapper does not have direct access to the database alias or vendor, you should generate the CursorWrapper class inside a function that accepts alias and vendor as arguments.
    • Method Overloading Pattern: Most methods you overload should follow this pattern: increment a metric counter, forward all arguments to the original method, and return the result.
    • Error Tracking: When overloading execute and execute_many, wrap the call to the parent method in a try...except block to ensure the errors_total counter is incremented when an exception occurs.

  10. Import django-prometheus using only local settings

    master

    If you cannot modify the existing codebase, you can inject django-prometheus by manipulating settings and using a URL wrapper.

    1. Modify Settings: Inject the middlewares and add the app to INSTALLED_APPS using list/set concatenation.
    2. Create a URL Wrapper: Create a new file (e.g., urls_prometheus_wrapper.py) that includes django_prometheus.urls and then includes your actual project URLs.
    3. Update ROOT_URLCONF: Point ROOT_URLCONF to your new wrapper file.
    # settings.py injection
MIDDLEWARE = \
    ['django_prometheus.middleware.PrometheusBeforeMiddleware'] + \
    MIDDLEWARE + \
    ['django_prometheus.middleware.PrometheusAfterMiddleware']

INSTALLED_APPS += ['django_prometheus']

# Set the wrapper
ROOT_URLCONF = "myproject.urls_prometheus_wrapper"

# urls_prometheus_wrapper.py
from django.urls import include, path

urlpatterns = []
urlpatterns.append(path('prometheus/', include('django_prometheus.urls')))
urlpatterns.append(path('', include('myapp.urls')))

  11. Monitor Django caches

    master

    To monitor Filebased, Memcached, or Redis caches, replace the cache BACKEND with the one provided by django_prometheus.cache.backends.

    CACHES = {
    'default': {
        'BACKEND': 'django_prometheus.cache.backends.filebased.FileBasedCache', # Replace django.core.cache.backends...
        'LOCATION': '/var/tmp/django_cache',
    }
}

  12. Configure Prometheus metric namespace and latency buckets

    master

    You can customize how metrics are named and how latencies are grouped using settings.

    • PROMETHEUS_METRIC_NAMESPACE: A string that prefixes all exported metrics (e.g., project_).
    • PROMETHEUS_LATENCY_BUCKETS: A tuple of floats defining the histogram buckets for latencies. Adding more buckets increases accuracy but can decrease performance.