DOCSEARCHbyAlgolia
Submit Docs

pganalyze collector

repository·main·Indexed 19 days ago

https://github.com/pganalyze/collector

A Go-based daemon that collects PostgreSQL schema, statistics, and OS-level metrics (CPU, Memory, Storage), converting them into a Protocol Buffers structure for monitoring and graphing. It supports deployment via Helm charts for Kubernetes, Docker containers for Amazon RDS and sidecar setups, and native Linux packages (deb/rpm). The collector includes features for OpenTelemetry (OTEL) log receiving, dry-run data inspection, and configurable success/error callbacks.

Tokens
6.9K
Snippets
9
Records
26
Agent score
64%

What's inside pganalyze-collector

  1. Overview of pganalyze collector packaging

    main

    The pganalyze/collector repository contains the build system for creating Linux packages (deb/rpm) for the pganalyze collector. The packaging process is divided into three stages:

    1. src/: Builds deb/rpm packages compatible with the systemd init system.
    2. test/: Validates package installation across all supported Linux distributions.
    3. repo/: Signs packages using the pganalyze signing key (available via https://keybase.io/pganalyze) and synchronizes them to S3-hosted repositories.

    All packages are built for both amd64 (64-bit X86) and arm64/aarch64 (64-bit ARMv8) architectures.

  2. Snapshot data model definitions

    main

    The snapshot format is organized into four distinct data types to optimize how the monitoring system processes and stores information. This structure allows for a two-step processing workflow: first, UPSERTing References to obtain IDs, and then COPYing the associated data using those IDs.

    TermDescription
    ReferenceEntities that can be UPSERTed into the monitoring system. Other parts of the snapshot reference these by their list index.
    InformationNon-historical data attached to a Reference (at most once). This is typically UPDATEd after the initial UPSERT.
    StatisticHistorical data attached to a Reference (at most once). Counters are normalized by the collector before snapshot creation so the recipient does not need to calculate deltas.
    EventData attached to a Reference that can occur multiple times within a single snapshot.

    To process a snapshot, a consumer should first resolve all References to get their IDs, then use those IDs to COPY the Information, Statistic, and Event data into the database.

  3. Configure OpenTelemetry (OTEL) log receiving service

    main

    If your Postgres instances are running within the Kubernetes cluster and you use log collectors (like Fluent Bit) to send logs via OpenTelemetry, you can enable a service in the collector to receive them.

    • Set service.create: true to enable the service.
    • service.name: The name used by your log sender (e.g., Fluent Bit). Defaults to pganalyze-collector-otel-service.
    • service.port: The port the log sender should target. Defaults to 4318.
    • service.targetPort: The target port on the container. This must match the port specified with db_log_otel_server in your collector configuration.
    • For multiple OTEL servers, use service.ports (a list of port/targetPort objects) instead of the individual port/targetPort settings.

  4. Configure environment variables via ConfigMap or Secret

    main

    You can pass runtime environment variables to the collector using three different methods:

    1. ConfigMap: Set configMap.create: true to have the chart generate a ConfigMap. Use configMap.values to define the data. Alternatively, point to an existing ConfigMap using configMap.name.
    2. Secret: Set secret.create: true to generate a Secret for sensitive data. Use secret.values to define the data. Alternatively, point to an existing Secret using secret.name.
    3. Direct Environment Variables: Use extraEnv (for key-value pairs) or extraEnvRaw (for advanced Kubernetes references like secretKeyRef or fieldRef) to pass variables directly to the container.

  5. Understand the pganalyze Snapshot data format

    main

    Snapshots are the communication format used between the pganalyze collector and the statistics processor. They are encoded using Protocol Buffers.

    The collector extracts statistics from a PostgreSQL database server, transforms them into this standardized format, and encodes them for transmission. This allows for efficient, structured data transfer of database metrics.

  7. Run the collector in Docker for non-RDS databases

    main

    If your database is running in a Docker environment, you can run the collector as a sidecar.

    Required Environment Variables:

    • PGA_API_KEY: Your pganalyze API key.
    • DB_NAME: The name of the database to monitor.

    Optional Environment Variables:

    • PGA_ALWAYS_COLLECT_SYSTEM_DATA: Set to true if the database is on the same host and you want to collect system metrics from inside the container.
    • DB_USERNAME and DB_PASSWORD: Database credentials.

    Tip: To collect I/O statistics, mount the database volume to the container using -v /path/to/database/volume/on/host:/var/lib/postgresql/data.

    docker pull quay.io/pganalyze/collector:stable
docker run \
  --name my-app-pga-collector \
  --link my-app-db:db \
  --env-file collector_config.env \
  quay.io/pganalyze/collector:stable

    Example collector_config.env:

    PGA_API_KEY=$YOUR_API_KEY
PGA_ALWAYS_COLLECT_SYSTEM_DATA=true
DB_NAME=your_database_name
DB_USERNAME=your_database_user
DB_PASSWORD=your_database_password

  8. Requirements for building pganalyze collector packages

    main

    To build packages or synchronize repositories using the scripts in this repository, you must have the following tools installed:

    • Docker: Required for all package building and repository synchronization tasks.
    • keybase.io client: Required for signing the deb/rpm packages.
    • AWS CLI: Required for synchronizing packages to S3-hosted repositories.

  9. Deploy pganalyze-collector via Helm

    main

    The pganalyze-collector can be deployed to Kubernetes using a Helm chart. This chart manages the deployment of the statistics collector, including optional ConfigMaps and Secrets for environment variable management, and an optional OpenTelemetry (OTEL) service for receiving logs from Postgres instances running within the cluster.

    To use the chart, you will need to provide a values.yaml file to configure your specific environment, such as image tags, resource limits, and authentication credentials.

  10. Install the pganalyze collector via APT/YUM

    main

    The collector is available as a package for Debian-based and RHEL-based systems. You can find the repositories at https://packages.pganalyze.com/.

    After installation, the configuration file is located at /etc/pganalyze-collector.conf. You must edit this file to add your pganalyze API key (one per database server) and your database connection credentials.

    If you need to monitor multiple servers, you can repeat the configuration block using a different [name] for each server.

  11. Install the collector via Helm Chart

    main

    To deploy the collector on Kubernetes, use the pganalyze Helm chart. You can provide custom configurations using a values.yml file.

    helm repo add pganalyze https://charts.pganalyze.com/
helm install my-collector pganalyze/pganalyze-collector --values=myvalues.yml

  12. Run the collector in Docker for Amazon RDS

    main

    When monitoring an Amazon RDS instance using Docker, use the quay.io/pganalyze/collector:stable image.

    Requirements:

    • Set PGA_API_KEY, DB_URL, and AWS_INSTANCE_ID environment variables.
    • The EC2 instance running the container must have an IAM role allowing CloudWatch access.
    • For better server metrics, enable "Enhanced Monitoring" in your RDS dashboard.

    Note: By default, you need one container per RDS instance. To monitor multiple databases on a single RDS instance, set the DB_ALL_NAMES=true environment variable.

    docker pull quay.io/pganalyze/collector:stable
docker run \
  --rm \
  --name pganalyze-mydb \
  -e DB_URL=postgres://username:password@my-instance-id.account.us-east-1.rds.amazonaws.com/mydb \
  -e PGA_API_KEY=YOUR_PGANALYZE_API_KEY \
  -e AWS_INSTANCE_ID=YOUR_RDS_DB_IDENTIFIER \
  quay.io/pganalyze/collector:stable