Stackdriver Prometheus Exporter

Metrics gathered from Google Stackdriver Monitoring are transformed into Prometheus format using these rules:

Naming Convention: stackdriver_<subsystem>_<name>

• namespace: Constant prefix stackdriver.
• subsystem: Normalized monitored resource type (e.g., gce_instance).
• name: Normalized metric type (e.g., compute_googleapis_com_instance_cpu_usage_time).

Labels: Labels are an aggregation of the metric unit, metric type labels, and monitored resource labels.

Metric Mapping:

• GAUGE $\rightarrow$ Prometheus Gauge.
• CUMULATIVE $\rightarrow$ Prometheus Counter.
• DELTA $\rightarrow$ Prometheus Gauge (or an accumulating Counter if monitoring.aggregate-deltas is enabled).
• DISTRIBUTION $\rightarrow$ Prometheus Histogram (note: _sum time series is not supported).

Supported Types: BOOL, INT64, DOUBLE, and DISTRIBUTION. STRING and MONEY types are discarded.

By default, Stackdriver DELTA metrics are reported as Prometheus Gauge metrics. However, this can produce inaccurate data for rate calculations. To treat them as accumulating counters, enable the monitoring.aggregate-deltas flag.

Challenges with Aggregation

1. Start-up Delay: When the exporter starts, it has no persisted state. The first sample is treated as the starting value of a counter rather than an increment from zero. It may take several scrape cycles for rates to stabilize.
2. Slow Moving Metrics: If a metric doesn't change for a long time (default 5m), Prometheus marks the series as stale. When the metric finally changes, it appears as a new series rather than an increment.

Solutions

• monitoring.aggregate-deltas: Converts DELTA metrics into in-memory counters.
• monitoring.aggregate-deltas-ttl: Controls how long a delta metric is kept in memory after GCP stops reporting it (default 30m). Increasing this helps with slow-moving metrics but increases memory usage.

Warning: If you use honor_timestamps: true in your Prometheus scrape config, the feature that continues to export metrics not collected during a scrape may cause ingestion errors if GCP has late-arriving data.

You can apply specific filters to your metric collection using the --monitoring.filters flag. The format for a filter is <targeted_metric_prefix>:<filter_query>.

• The targeted_metric_prefix must be a prefix of one or more monitoring.metrics-prefixes.
• The filter_query is applied to the final Monitoring API call using AND logic.
• String values containing special characters (like :) must be enclosed in double quotes ".

Example: Filtering specific Pub/Sub and Compute metrics

stackdriver_exporter \
 --google.project-ids=my-test-project \
 --monitoring.metrics-prefixes='pubsub.googleapis.com/subscription' \
 --monitoring.metrics-prefixes='compute.googleapis.com/instance/cpu' \
 --monitoring.metrics-prefixes='cloudsql.googleapis.com/database' \
 --monitoring.filters='pubsub.googleapis.com/subscription:resource.labels.subscription_id=monitoring.regex.full_match("us-west4.*my-team-subs.*")' \
 --monitoring.filters='compute.googleapis.com/instance/cpu:resource.labels.instance=monitoring.regex.full_match("us-west4.*my-team-subs.*")' \
 --monitoring.filters='cloudsql.googleapis.com/database:resource.labels.database_id="my-test-project:my-database-name"'