DOCSEARCHbyAlgolia
Submit Docs

Prometheus Pushgateway Documentation

repository·master·Indexed 25 days ago

https://github.com/prometheus/pushgateway

A metrics cache designed for ephemeral and batch jobs, allowing short-lived processes to push metrics to a central location for Prometheus to scrape. Includes documentation on the HTTP Push API (POST, PUT, DELETE), the Admin and Query APIs, persistence configuration, and strategies for alerting on failed pushes.

Tokens
3.7K
Snippets
6
Records
25
Agent score
84%

What's inside Prometheus Pushgateway

  1. Overview of Prometheus Pushgateway

    master

    The Prometheus Pushgateway is a metrics cache designed for ephemeral and batch jobs that cannot be scraped directly by Prometheus because they do not exist long enough. Instead of Prometheus scraping these jobs, the jobs push their metrics to the Pushgateway, which then exposes them for Prometheus to scrape.

    Key Characteristics:

    • Metrics Cache: It acts as a cache for metrics, not an aggregator or distributed counter. Metrics pushed are identical to those presented by a permanently running program.
    • Service-level Metrics: It is intended for service-level metrics. For machine-level metrics, the Node Exporter textfile collector is recommended.
    • No TTL/Timeout: The Pushgateway does not implement a timeout or Time-To-Live (TTL) for pushed metrics.

  2. Run Prometheus Pushgateway using Docker

    master

    You can deploy the Pushgateway using the prom/pushgateway Docker image.

    To pull the image and run it in a detached container mapping the default port 9091, use the following commands:

    docker pull prom/pushgateway

docker run -d -p 9091:9091 prom/pushgateway

  3. Configure Prometheus to scrape Pushgateway

    master

    To scrape metrics from the Pushgateway, add it as a target in your Prometheus scrape configuration.

    Important: You must set honor_labels: true in the scrape configuration. This ensures that the job and instance labels attached to the metrics by the pusher are preserved, rather than being renamed to exported_job and exported_instance by the Prometheus server.

  4. Set up a development environment

    master

    To see changes to web files immediately during development, you can instruct the binary to use local files instead of embedded ones.

    1. Add -tags dev to the flags entry in .promu.yml.
    2. Run make build.
    3. To revert to normal mode, remove the tag from .promu.yml and run make assets.

  5. Run Prometheus Pushgateway from binary

    master

    To run the Pushgateway, download the binary releases for your platform from the GitHub releases page and unpack the tarball.

    To start the service with custom configurations:

    • Use --web.listen-address to specify the address and port to listen on (e.g., 0.0.0.0:9091 or :9091).
    • Use --persistence.file to specify a file for persisting pushed metrics so they survive restarts. By default, metrics are not persisted.

  6. Push metrics via HTTP (Command Line)

    master

    You can push metrics using any HTTP tool (like curl) following the Prometheus text protocol. Each line must end with a line-feed character (\n).

    Metrics are managed in groups identified by a grouping key. The first label in the grouping key must be job.

    Note for Windows users: Use PowerShell's Invoke-WebRequest. Ensure you use double quotes for the metric text and a backtick (`) before the line ending character (n).

    ### Linux/macOS (curl)
# Push a single sample (untyped)
echo "some_metric 3.14" | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job

# Push complex metrics with type and help strings
cat <<EOF | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance
# TYPE some_metric counter
some_metric{label="val1"} 42
# TYPE another_metric gauge
# HELP another_metric Just an example.
another_metric 2398.283
EOF

### Windows (PowerShell)
Invoke-WebRequest -Uri http://localhost:9091/metrics/job/MyJob -Method POST -Body "Some_Metric 3.14`n"

  7. Run Prometheus Pushgateway locally

    master

    You can run the Pushgateway by downloading binary releases from the GitHub releases page and unpacking the tarball.

    To run the binary with custom configurations:

    • Listen Address: Use the --web.listen-address flag to change the address the Pushgateway listens on (e.g., 0.0.0.0:9091 or :9091).
    • Persistence: By default, metrics are not persisted. To ensure metrics survive restarts, use the --persistence.file flag to specify a file for storage.

  9. Alert on failed pushes

    master

    To ensure batch jobs are running correctly, use the following alerting strategies:

    1. Detecting failed or delayed pushes: Alert when push_time_seconds is significantly older than expected, or specifically when push_failure_time_seconds > push_time_seconds.
    2. Detecting malformed pushes: If a push is malformed, it won't reach a metric group and won't trigger push_failure_time_seconds. Instead, monitor the rate of pushgateway_http_requests_total{code="400",handler="push"} and inspect logs to identify the source.

  10. Configure TLS and Basic Authentication

    master

    To secure the Pushgateway with TLS and/or basic authentication, provide a configuration file using the --web.config.file flag.

    Note: These settings apply to all endpoints, including /metrics (scraping), the push API, the management API, and the web UI.

  11. Push metrics with compression

    master

    The Pushgateway HTTP server supports compressed request bodies. When pushing metrics, you can use the Content-Encoding header to specify the compression algorithm:

    • gzip: Standard gzip compression.
    • snappy: Snappy compression.

    This is useful for reducing bandwidth when pushing large volumes of metrics.

  12. Handle metric inconsistencies and performance

    master

    All metrics exposed by the Pushgateway must be consistent. Metrics with the same name must have the same type, even if they belong to different groups. Pushing inconsistent metrics (e.g., same name but different types) will result in a 400 error.

    If you have a very large number of metrics and frequent pushes, the consistency check might slow down pushes. You can use the --push.disable-consistency-check flag to skip this check, but be aware that inconsistent metrics will still be checked during a scrape, which can cause scrapes to fail.