promxy

Promxy is a proxy designed to solve scaling and redundancy challenges in Prometheus deployments. It acts as a single entry point for your metrics, addressing several common pain points encountered when scaling Prometheus horizontally:

• Eliminates Data Holes: Unlike standard load balancing between Prometheus replicas (which causes gaps in graphs during node reboots or failures), Promxy provides seamless data continuity.
• Single Grafana Datasource: Instead of managing multiple Prometheus datasources or using complex 'mixed' sources in Grafana to query across shards/regions, you can point Grafana to a single Promxy endpoint.
• Removes the Need for Aggregation Layers: Promxy replaces the need for complex Prometheus federation/aggregation shards. This reduces the load on your existing shards and eliminates the need to maintain separate alerting rules for an aggregation layer.
• Access Logging: Provides access logs for your metrics queries, which is not natively available in standard Prometheus setups.

Promxy provides two primary ways to scope or restrict queries sent to a specific server_group:

Label Filtering (label_filter)

Restricts queries to only those that match a specific set of labels known to exist in the downstream. This is not a security mechanism but an optimization.

• dynamic_labels: A list of labels (like __name__ or job) that Promxy will periodically query from the downstream to build the filter.
• sync_interval: How often to re-sync dynamic labels.
• static_labels_include: A map of labels that MUST be present in the query.
• static_labels_exclude: A map of labels that MUST NOT be present in the query.
• on_sync_error: Behavior when the filter cannot sync (e.g., target is down). Options: abort (default), open (proceed without filtering), or closed (skip target until sync succeeds).

Matcher Injection (inject_matchers)

Unlike labels (which adds labels to the result), inject_matchers modifies the query itself. It adds specific PromQL matchers to every selector sent to the group. This is ideal for multi-tenant setups where you want to ensure a user only sees data for their specific cluster/tenant.

Example: If you set inject_matchers: ['cluster="A"'], a query for up becomes up{cluster="A"}.

      label_filter:
        dynamic_labels:
          - __name__
          - job
        sync_interval: 5m
        on_sync_error: abort

      inject_matchers:
        - 'cluster="A"'

You can deploy Promxy using the provided docker-compose.yml configuration. The setup includes Promxy, VictoriaMetrics (as a storage backend), and Alertmanager.

By default, the Promxy service is configured with the following settings:

• Image: quay.io/jacksontj/promxy
• Port Mapping: 8082:8082
• Default Command Flags:
  • --config=/cmd/promxy/config.yaml: Specifies the path to the Promxy configuration file.
  • --log-level=info: Sets the logging verbosity.
  • --web.enable-lifecycle: Enables the HTTP endpoint for managing the server lifecycle (e.g., reloading configuration via POST requests).
• Volumes: Uses logvolume01 for /var/log and mounts a local directory to /cmd to provide configuration files.

services:
  promxy:
    container_name: promxy
    image: quay.io/jacksontj/promxy
    ports:
    - "8082:8082"
    command:
      - --config=/cmd/promxy/config.yaml
      - --log-level=info
      - --web.enable-lifecycle

Promxy can manage alerting settings and customize the URLs sent to an Alertmanager.

Alertmanager Configuration

Under the alerting block, you define alertmanagers targets. You can also use http_headers to inject authentication tokens (like X-Gateway-Signature) if the Alertmanager is behind a gateway.

Custom Alert Templates

The alert_templates block allows you to customize the GeneratorURL sent with alerts. You can use Go text/template syntax with access to .ExternalURL, .Expr, .AlertName, .Labels, and .Annotations.

Templates can be defined in three ways:

1. default: An inline template or a reference to a named template.
2. named: A collection of reusable templates.
3. rules: A list of rules that select a template based on match_labels (evaluated top-to-bottom).

alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "127.0.0.1:12345"

# Inside a server_group:
alert_templates:
  default: '{{.ExternalURL}}/graph?g0.expr={{.Expr | urlquery}}&g0.tab=1'
  named:
    grafana: 'https://grafana.example.com/alerting/groups?queryString=alertname%3D%22{{.AlertName | urlquery}}%22'
  rules:
    - match_labels:
        severity: critical
      template: grafana