kube-prometheus

Follow these steps to perform a rapid installation of the kube-prometheus stack. This process installs the Prometheus Operator, node-exporter, kube-state-metrics, Grafana, Prometheus, and Alertmanager.

Prerequisites

1. A running Kubernetes cluster deployed via kubeadm with the controller-manager and scheduler exposed (see configuration guide).
2. The kube-prometheus repository cloned locally.

Installation Steps

1. Clone the repository and enter the directory.
2. Create a dedicated namespace (e.g., monitoring).
3. Apply Prometheus Operator manifests and wait for CRDs to become available.
4. Install node-exporter and kube-state-metrics.
5. Deploy Grafana credentials and the Grafana instance.
6. Deploy the Prometheus object along with its required roles and role-bindings.
7. Install Alertmanager.

Default Access (via NodePorts)

• Prometheus UI: Port 30900
• Alertmanager UI: Port 30903
• Grafana: Port 30902 (Default credentials: admin/admin)

Note: It is highly recommended to change default Grafana credentials for production and use proper service exposure methods instead of NodePorts.

# 1. Clone repository
git clone https://github.com/prometheus-operator/kube-prometheus
cd kube-prometheus/

# 2. Create namespace
export NAMESPACE='monitoring'
kubectl create namespace "$NAMESPACE"

# 3. Install Prometheus Operator
kubectl --namespace="$NAMESPACE" apply -f manifests/prometheus-operator

# Wait for CRDs
until kubectl --namespace="$NAMESPACE" get alertmanagers.monitoring.coreos.com > /dev/null 2>&1; do sleep 1; printf "."; done

# 4. Install exporters
kubectl --namespace="$NAMESPACE" apply -f manifests/node-exporter
kubectl --namespace="$NAMESPACE" apply -f manifests/kube-state-metrics

# 5. Install Grafana
kubectl --namespace="$NAMESPACE" apply -f manifests/grafana/grafana-credentials.yaml
kubectl --namespace="$NAMESPACE" apply -f manifests/grafana

# 6. Install Prometheus
find manifests/prometheus -type f ! -name prometheus-k8s-roles.yaml ! -name prometheus-k8s-role-bindings.yaml -exec kubectl --namespace "$NAMESPACE" apply -f {} \;
kubectl apply -f manifests/prometheus/prometheus-k8s-roles.yaml
kubectl apply -f manifests/prometheus/prometheus-k8s-role-bindings.yaml

# 7. Install Alertmanager
kubectl --namespace="$NAMESPACE" apply -f manifests/alertmanager

If you have existing rules in YAML format, you can convert them to JSON and import them into your Jsonnet configuration.

1. Install gojsontoyaml:

   go get -u -v github.com/brancz/gojsontoyaml

2. Convert your YAML rule file to JSON:

   cat existingrule.yaml | gojsontoyaml -yamltojson > existingrule.json

3. Import the JSON file in your Jsonnet code using the import function to populate the spec.groups field.

local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
  exampleApplication: {
    prometheusRuleExample: {
      apiVersion: 'monitoring.coreos.com/v1',
      kind: 'PrometheusRule',
      metadata: {
        name: 'my-prometheus-rule',
        namespace: $.values.common.namespace,
      },
      spec: {
        groups: (import 'existingrule.json').groups,
      },
    },
  },
};

To generate Kubernetes manifests that point to your internal registry instead of upstream sources, use the withImageRepository mixin from kube-prometheus/addons/config-mixins.libsonnet.

This mixin allows you to override the image repository for all components (Prometheus Operator, Node Exporter, Kube-State-Metrics, Alertmanager, Prometheus, and Grafana) by passing your internal registry URL as an argument.

local mixin = import 'kube-prometheus/addons/config-mixins.libsonnet';
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
} + mixin.withImageRepository('internal-registry.com/organization');

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

You can run Prometheus in Agent mode using kube-prometheus, but this requires using strategic merge patches. This configuration is not officially recommended and is provided without support.

To enable Agent mode, you must:

1. Set enableFeatures: ['agent'] in the prometheus values.
2. Configure remoteWrite with a valid URL.
3. Disable alerting and rule selection by setting alerting and ruleSelector to empty objects.
4. Update the Prometheus container arguments to include --enable-feature=agent and --storage.agent.path=/prometheus.

local kp =
  (import 'kube-prometheus/main.libsonnet') +
  {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },
      prometheus+: {
        resources: {
          requests: { memory: '100Mi' },
        },
        enableFeatures: ['agent'],
      },
    },
    prometheus+: {
      prometheus+: {
        spec+: {
          replicas: 1,
          alerting:: {},
          ruleSelector:: {},
          remoteWrite: [{
            url: 'http://remote-write-url.com',
          }],
          containers+: [
            {
              name: 'prometheus',
              args+: [
                '--config.file=/etc/prometheus/config_out/prometheus.env.yaml',
                '--storage.agent.path=/prometheus',
                '--enable-feature=agent',
                '--web.enable-lifecycle',
              ],
            },
          ],
        },
      },
    },
  };

{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
{ 
  ['prometheus-' + name]: kp.prometheus[name] 
  for name in std.objectFields(kp.prometheus) 
}

To provide a custom Alertmanager configuration, you must set the values.alertmanager.config field within your Jsonnet configuration. This field accepts the Alertmanager configuration content as a string.

You can either inline the configuration directly as a multi-line string or import an external YAML file using the importstr function.

// Example 1: Inlined configuration
((import 'kube-prometheus/main.libsonnet') + {
   values+:: {
     alertmanager+: {
       config: |||
         global:
           resolve_timeout: 10m
         route:
           group_by: ['job']
           group_wait: 30s
           group_interval: 5m
           repeat_interval: 12h
           receiver: 'null'
           routes:
           - match:
               alertname: Watchdog
             receiver: 'null'
         receivers:
         - name: 'null'
       |||,
     },
   },
 }).alertmanager.secret

// Example 2: Importing from an external YAML file
((import 'kube-prometheus/main.libsonnet') + {
   values+:: {
     alertmanager+: {
       config: importstr 'alertmanager-config.yaml',
     },
   },
 }).alertmanager.secret

To improve the development and review experience, kube-prometheus supports ephemeral developer workspaces using GitHub Codespaces. These workspaces provide a pre-configured environment with a Kubernetes cluster where you can deploy and test kube-prometheus changes.

Workflow for Reviewing Pull Requests

When reviewing a PR, the workspace provides a fully-functional Kubernetes cluster that generates real monitoring data. This allows you to verify if the proposed changes work as expected by observing the actual impact on the cluster.

Workflow for Developing Features or Bug Fixes

If you are actively modifying the codebase to implement new features or fix bugs, follow these steps:

1. Regenerate manifests: Run make generate to update the YAML manifests based on your changes.
2. Deploy: Run make deploy to apply the updated manifests to the cluster.

You can add new Grafana dashboards by extending the grafanaDashboards key within the grafana object in your Jsonnet configuration. It is recommended to use the grafonnet library to define dashboards using a DSL.

To add a dashboard, include it in the values.grafana.dashboards path of your top-level configuration object.

local grafana = import 'grafonnet/grafana.libsonnet';
local dashboard = grafana.dashboard;
// ... other imports

local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+:: {
      namespace: 'monitoring',
    },
    grafana+: {
      dashboards+:: {
        'my-dashboard.json':
          dashboard.new('My Dashboard')
          .addTemplate({
              current: { text: 'Prometheus', value: 'Prometheus' },
              name: 'datasource',
              query: 'prometheus',
              type: 'datasource',
              // ... other template options
          })
          .addRow(
            row.new()
            .addPanel(graphPanel.new('My Panel', span=6, datasource='$datasource')
                      .addTarget(prometheus.target('vector(1)')))
          ),
      },
    },
  },
};

To monitor an etcd cluster that is hosted outside of Kubernetes, you must ensure network connectivity and verify the Prometheus scrape configuration. This approach is compatible with kube-aws and other similar tools.

Prerequisites and Configuration Patterns

When using etcd.jsonnet or static-etcd.libsonnet, the following automation is performed:

1. TLS Configuration: The three etcd TLS client files (CA, cert, and key) are placed into a Kubernetes Secret in the namespace, which the Prometheus Operator then loads.
2. Resource Creation: A Service, Endpoint, and ServiceMonitor are created to expose etcd metrics on port 2379.

Step 1: Open the Port

You must allow the nodes running Prometheus to communicate with the etcd cluster on port 2379 (or whichever port etcd uses to expose metrics).

• For kube-aws: Edit the etcd security group inbound rules to specify the security group of your Kubernetes worker nodes as the source.
• Note on IP addresses in kube-aws: etcd nodes may respond to :2379/metrics on either the EC2 instance IP (eth0) or the EIP/ENI address (eth1). If you specify the Instance IP in the Prometheus Operator ServiceMonitor and the EC2 instance goes down, you will need to update the ServiceMonitor. Using EIP/ENI is preferred as they are persistent.

Step 2: Verify Scrape Configuration

1. Check Prometheus Config: Navigate to the Prometheus UI at :9090/config and verify that an etcd job entry exists, for example:

   - job_name: monitoring/etcd-k8s/0
     scrape_interval: 30s
     scrape_timeout: 10s

2. Check Targets: Navigate to the :9090/targets page:
   • If you see etcd with an UP state, monitoring is working.
   • If the state is not UP, check the Error column for details.
   • If no etcd targets appear, Prometheus is not attempting to scrape the service.