kafka_exporter

By default, consumer group metrics are collected via Kafka. If you need to collect consumer lag specifically from Zookeeper, you must enable the following flags:

1. use.consumelag.zookeeper: Set to true to enable collection from Zookeeper.
2. zookeeper.server: Provide the address of the Zookeeper server (e.g., localhost:2181).

This will enable the metric kafka_consumergroupzookeeper_lag_zookeeper.

If you are using Datadog to scrape Prometheus endpoints, ensure the Datadog collector is installed with datadog.prometheusScrape.enabled=true and datadog.prometheusScrape.serviceEndpoints=true.

helm repo add datadog https://helm.datadoghq.com
helm repo update

helm upgrade -i datadog datadog/datadog --namespace=kafka-exporter \
  --set datadog.apiKey=<key>  \
  --set targetSystem=linux \
  --set datadog.prometheusScrape.enabled=true \
  --set datadog.prometheusScrape.serviceEndpoints=true

You can run kafka_exporter as a binary, a Docker container, or via Docker Compose. It requires a Kafka server (version 0.10.1.0 or later) and exposes Prometheus metrics on port 9308 by default.

Run as a Binary

kafka_exporter --kafka.server=kafka:9092 [--kafka.server=another-server ...]

Run via Docker

Pull the latest image from Docker Hub and run it:

docker pull danielqsj/kafka-exporter:latest
docker run -ti --rm -p 9308:9308 danielqsj/kafka-exporter --kafka.server=kafka:9092

Run via Docker Compose

Create a docker-compose.yml file:

services:
  kafka-exporter:
    image: danielqsj/kafka-exporter 
    command: ["--kafka.server=kafka:9092"]
    ports:
      - 9308:9308     

Then run:

docker-compose up -d

To perform a basic installation of kafka-exporter using Helm, use the helm upgrade -i command. You must specify the Kafka brokers using the kafkaExporter.kafka.servers configuration key in the format "{host:port,host:port,...}".

helm upgrade -i kafka-exporter kafka_exporter/charts/kafka-exporter --namespace=kafka-exporter --create-namespace \
  --set kafkaExporter.kafka.servers="{kafka1:9092,kafka2:9092,.....}"

To enable Datadog support, set datadog.use_datadog=true and provide a datadog.prefix. When using Datadog for metric collection, you should disable the standard Prometheus ServiceMonitor by setting prometheus.serviceMonitor.enabled=false.

helm upgrade -i kafka-exporter kafka_exporter/charts/kafka-exporter --namespace=kafka-exporter --create-namespace \
  --set kafkaExporter.kafka.servers="{kafka1:9092,kafka2:9092,.....}" \
  --set datadog.prefix=testing-kafka-cluster \
  --set datadog.use_datadog=true \
  --set prometheus.serviceMonitor.enabled=false

To integrate with Azure Managed Prometheus, enable the ServiceMonitor and set azuremanagedprometheus.use_azuremanagedprometheus=true.

helm upgrade -i kafka-exporter kafka_exporter/charts/kafka-exporter --namespace=kafka-exporter --create-namespace \
  --set kafkaExporter.kafka.servers="{kafka1:9092,kafka2:9092,.....}" \
  --set prometheus.serviceMonitor.enabled=true \
  --set azuremanagedprometheus.use_azuremanagedprometheus=true

If you do not see any consumer group or lag information in the metrics, it is likely because there are no active consumers using a consumer group.

To test this, run a consumer with a consumer group using the Kafka CLI tool:

kafka-console-consumer.sh \
    --consumer.config /path/to/client.properties \
    --bootstrap-server localhost:9092 \
    --topic test \
    --group test-conusmer-group \
    --from-beginning

To enable TLS for the Kafka connection, set useTLS to true in kafkaOpts. You can configure the following:

• tlsServerName: Used to verify the hostname on the returned certificates.
• tlsCAFile: Path to the CA certificate file.
• tlsCertFile and tlsKeyFile: Paths to the client certificate and key for mutual authentication.
• tlsInsecureSkipTLSVerify: If true, the server's certificate validity is not checked.

The exporter supports serving metrics over HTTPS with optional Mutual TLS (mTLS) authentication.

Key configuration options via opts:

• serverUseTLS: Boolean to enable HTTPS.
• serverTlsCertFile: Path to the server's TLS certificate file.
• serverTlsKeyFile: Path to the server's TLS private key file.
• serverMutualAuthEnabled: If true, sets ClientAuth to tls.RequireAndVerifyClientCert.
• serverTlsCAFile: Path to the CA certificate file used to verify client certificates.

When serverUseTLS is enabled, the server uses a tls.Config with a minimum version of TLS 1.2 and a specific set of secure cipher suites.

The kafka_exporter supports several SASL mechanisms via the sasl.mechanism option. When configuring kafkaOpts, ensure the corresponding parameters are provided:

• SCRAM: Use scram-sha256 or scram-sha512. Requires sasl.username and sasl.password.
• GSSAPI (Kerberos): Requires sasl.service-name, sasl.kerberos-config-path, sasl.realm, and sasl.username. If using keytab authentication, set sasl.kerberos-auth-type to keytabAuth and provide sasl.keytab-path.
• AWS IAM: Use awsiam. Requires sasl.aws-region (defaults to AWS_REGION env var).
• OAuthBearer: Use oauthbearer. Requires sasl.oauthbearer-token-url (or SASL_OAUTHBEARER_TOKEN_URL env var), sasl.username, and sasl.oauthbearer-scopes (comma-separated).