DOCSEARCHbyAlgolia
Submit Docs

Prometheus Exporter

repository·main·Indexed 20 days ago

https://github.com/discourse/prometheus_exporter

A Ruby-based tool for aggregating and exporting custom application metrics to Prometheus. It supports single-process and multi-process architectures and provides built-in instrumentation for Rails, Sidekiq, Puma, Unicorn, Resque, and ActiveRecord connection pools. It includes capabilities for tracking HTTP request statistics, Ruby process memory (RSS), Garbage Collection, and Sidekiq job performance.

Tokens
18.9K
Snippets
54
Records
71
Agent score
70%

What's inside prometheus_exporter

  1. Overview of Prometheus Exporter

    main

    Prometheus Exporter is a framework for aggregating custom metrics from multiple processes and exporting them to Prometheus. It supports two primary operational modes:

    1. Single process mode: Metrics are collected and served from within a single process.
    2. Multi-process mode: Designed for environments with multiple processes (like web servers with multiple workers) to aggregate metrics into a single endpoint.

    It provides built-in support for various Ruby ecosystems including Rails, Sidekiq, Puma, Unicorn, Resque, and more.

  2. Use Histogram mode for metric aggregation

    main

    By default, built-in collectors report aggregations as summaries. If you need to aggregate metrics across different labels (e.g., using Prometheus's histogram_quantile function), you must run the exporter in histogram mode using the --histogram flag.

    Note: This mode sacrifices some precision but enables cross-node/cross-action aggregation.

    $ prometheus_exporter --histogram

  3. Use Prometheus Exporter in Single Process Mode

    main

    Single process mode is the simplest way to consume the exporter. You run a web server within your application process that hosts the /metrics endpoint. This is suitable for simple applications that do not use process clusters (like Puma or Unicorn).

    require 'prometheus_exporter/server'
require 'prometheus_exporter/client'
require 'prometheus_exporter/instrumentation'

# Start the web server on a specific bind address and port
server = PrometheusExporter::Server::WebServer.new bind: 'localhost', port: 12345
server.start

# Connect the default client to the server's collector
PrometheusExporter::Client.default = PrometheusExporter::LocalClient.new(collector: server.collector)

# Start basic process instrumentation (RSS, Ruby metrics, etc.)
PrometheusExporter::Instrumentation::Process.start(type: "my program", labels: {my_custom: "label for all process metrics"})

# Register and observe metrics
gauge = PrometheusExporter::Metric::Gauge.new("rss", "used RSS for process")
server.collector.register_metric(gauge)
gauge.observe(server.get_rss)

  4. Instrument GoodJob for metrics collection

    main

    To monitor GoodJob, start the instrumentation in a Rails initializer (e.g., config/initializers/good_job.rb). Metrics are generated from the database using relevant scopes.

    Metrics collected:

    • good_job_scheduled: Scheduled jobs
    • good_job_retried: Retried jobs
    • good_job_queued: Queued jobs
    • good_job_running: Running jobs
    • good_job_finished: Finished jobs
    • good_job_succeeded: Succeeded jobs
    • good_job_discarded: Discarded jobs
    # e.g. config/initializers/good_job.rb
require 'prometheus_exporter/instrumentation'
PrometheusExporter::Instrumentation::GoodJob.start

  5. Instrument Puma for metrics collection

    main

    To collect Puma metrics, you must start the instrumentation from within a Puma thread after workers have booted. This is done by adding the following to your puma.rb configuration file.

    For Puma single mode:

    # puma.rb config
require 'prometheus_exporter/instrumentation'
if !PrometheusExporter::Instrumentation::Puma.started?
  PrometheusExporter::Instrumentation::Puma.start
end

    For Puma clustered mode:

    # puma.rb config
after_worker_boot do
  require 'prometheus_exporter/instrumentation'
  if !PrometheusExporter::Instrumentation::Puma.started?
    PrometheusExporter::Instrumentation::Puma.start
  end
end

    Metrics collected:

    • puma_workers: Number of puma workers
    • puma_booted_workers: Number of puma workers booted
    • puma_old_workers: Number of old puma workers
    • puma_running_threads: Number of spawned threads (busy or waiting)
    • puma_request_backlog: Number of requests waiting for a thread
    • puma_thread_pool_capacity: Available threads at current scale
    • puma_max_threads: Available threads at max scale
    • puma_busy_threads: Running threads (waiting for work + requests waiting for a thread)

    All metrics support a phase label and custom labels via the labels option.

  6. Ensure metrics are sent before Sidekiq shutdown

    main

    To prevent losing metrics generated immediately before a Sidekiq shutdown (specifically the sidekiq_restarted_jobs_total metric), explicitly stop the default client using an at_exit block with a timeout.

    Sidekiq.configure_server do |config|
  at_exit do
    PrometheusExporter::Client.default.stop(wait_timeout_seconds: 10)
  end
end

  8. Instrument Hutch Message Processing with Prometheus Exporter

    main

    To capture Hutch metrics (job duration, totals, and failures), set the tracer in your Hutch configuration.

    ```ruby
unless Rails.env.test?
  require 'prometheus_exporter/instrumentation'
  Hutch::Config.set(:tracer, PrometheusExporter::Instrumentation::Hutch)
end

    Metrics collected:

    • hutch_job_duration_seconds (Counter)
    • hutch_jobs_total (Counter)
    • hutch_failed_jobs_total (Counter) All metrics include a job_name label.

  9. Instrument Request Queueing Time

    main

    Request Queueing is the time between a request hitting your load balancer and reaching your application. To measure this, you must inject an HTTP header as early as possible in your infrastructure (e.g., Load Balancer or Reverse Proxy).

    • AWS ALB: Natively supports the request tracing header.
    • Other Upstream Entrypoints: Configure your server/load balancer to add the header X-Request-Start: t=<MSEC>.

    Note: Request time start is reported as epoch time in seconds and lacks high precision.

  10. Run Prometheus Exporter using Docker

    main

    You can run the exporter using the official Docker image. Pull the image and run it, mapping the default port 9394.

    # Pull the latest image
docker pull discourse/prometheus_exporter:latest

# Run the container
docker run -p 9394:9394 discourse/prometheus_exporter

# Run with additional flags (e.g., verbose mode and a prefix)
docker run -p 9394:9394 discourse/prometheus_exporter --verbose --prefix=myapp

  11. Instrument Resque for metrics collection

    main

    To monitor a Resque installation, start the Resque instrumentation in a Rails initializer (e.g., config/initializers/resque.rb). This queries Redis internally via Resque.info.

    Metrics collected:

    • resque_processed_jobs: Total processed jobs
    • resque_failed_jobs: Total failed jobs
    • resque_pending_jobs: Total pending jobs
    • resque_queues: Total number of queues
    • resque_workers: Total running workers
    • resque_working: Total workers currently working
    # e.g. config/initializers/resque.rb
require 'prometheus_exporter/instrumentation'
PrometheusExporter::Instrumentation::Resque.start

  12. Collect global metrics in a custom TypeCollector

    main

    Custom type collectors are ideal for collecting global metrics (e.g., user counts). Since the exporter process is lean and does not load all Rails dependencies, you must manually load the Rails environment if you need access to your models.

    Implementation Pattern:

    # Inside your collector class
def collect(obj)
  # ...
end

def metrics
  # Ensure Rails is loaded to access models
  unless defined?(Rails)
    require File.expand_path("../../config/environment", __FILE__)
  end

  user_count_gauge = PrometheusExporter::Metric::Gauge.new('user_count', 'number of users in the app')
  user_count_gauge.observe User.count
  [user_count_gauge]
end

    Performance Tip: The /metrics endpoint is called whenever Prometheus scrapes the exporter. To avoid heavy database queries on every scrape, use a caching gem like lru_redux with LruRedux::TTL::Cache to cache metric values for a few seconds.

    unless defined?(Rails)
  require File.expand_path("../../config/environment", __FILE__)
end

user_count_gauge = PrometheusExporter::Metric::Gauge.new('user_count', 'number of users in the app')
user_count_gauge.observe User.count
[user_count_gauge]