Vector Helm Charts

repository·develop·Indexed 41 days ago

https://github.com/vectordotdev/helm-charts

Official Helm charts for deploying Vector, a high-performance observability data pipeline. Supports three deployment roles: Agent (DaemonSet), Aggregator (StatefulSet), and Stateless-Aggregator (Deployment). Includes documentation on customConfig, autoscaling via HPA or KEDA, persistence, HAProxy load balancer integration, and migration guides from legacy vector-agent and vector-aggregator charts.

Tokens
6.1K
Snippets
17
Records
26
Agent score
51%

What's inside Vector

  1. Use template syntax in `customConfig`

    develop

    Vector's internal template syntax uses {{ ... }} which conflicts with Helm's templating engine. If you are providing configuration via the customConfig option, you must escape Vector's templates using Helm's print function to prevent Helm from attempting to evaluate them during installation.

    customConfig:
      sinks:
        loki:
          labels:
            host: |- 
              {{ print "{{ host }}" }}
            source: |- 
              {{ print "{{ source_type }}" }}
  2. Configure Vector deployment roles

    develop

    The Vector Helm chart supports three distinct deployment roles, which determine the underlying Kubernetes workload type and behavior:

    • Agent: Typically deployed as a DaemonSet to collect logs/metrics from every node. Validates with rbac.create: true for cluster-wide permissions.
    • Aggregator: Deployed as a StatefulSet. Used for centralized processing and buffering. Supports persistence for data storage and autoscaling.
    • Stateless-Aggregator: Deployed as a Deployment. Used for centralized processing where stateful persistence is not required. Supports autoscaling and topologySpreadConstraints.
  3. Install Vector with Datadog integration

    develop

    To set up Vector as an aggregator for Datadog, follow these steps to install the Vector Helm chart and configure it with your Datadog API key.

    1. Prerequisites

    • Helm v3.0+
    • kubectl installed
    • Datadog and Vector Helm repositories added:
    helm repo add datadog https://helm.datadoghq.com
    helm repo add vector https://helm.vector.dev
    helm repo update

    2. Install Vector

    Export your Datadog API key and install Vector into its own namespace using the provided Datadog values file:

    export DATADOG_API_KEY="<REPLACE_ME>"
    
    helm install vector vector/vector --namespace vector --create-namespace \
    	--values examples/datadog-values.yaml --set secrets.generic.datadog_api_key="${DATADOG_API_KEY}"

    3. Verify Installation

    Tail the logs of the Vector StatefulSet to ensure the API key is correct and the healthcheck passes. You should see INFO level messages indicating Healthcheck: Passed. for each sink.

  4. Migrate from vector-agent to the vector chart

    develop

    To migrate from the legacy vector-agent chart to the current vector chart, you must update your values.yaml to account for changes in deployment roles, tolerations, and default configurations.

    Deploying as an Agent

    To deploy the chart as a DaemonSet (the equivalent of the old agent behavior), set the role to "Agent":

    role: "Agent"

    Configuring Tolerations

    The tolerations option no longer includes default values. If you need to match previous behavior (e.g., allowing the agent to run on master nodes), you must define them explicitly:

    tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule

    Reverting to Legacy Default Configurations

    The new chart enables the Vector API and a console sink by default, and renames the prometheus_exporter sink from prometheus_sink to prom_exporter. To maintain the exact configuration used in older versions, use the following customConfig block:

    customConfig:
      data_dir: /vector-data-dir
      api:
        enabled: false
      sources:
        kubernetes_logs:
          type: kubernetes_logs
        host_metrics:
          filesystem:
            devices:
              excludes: [binfmt_misc]
            filesystems:
              excludes: [binfmt_misc]
            mountpoints:
              excludes: ["*/proc/sys/fs/binfmt_misc"]
          type: host_metrics
        internal_metrics:
          type: internal_metrics
      sinks:
        prometheus_sink:
          type: prometheus_exporter
          inputs: [host_metrics, internal_metrics]
          address: 0.0.0.0:9090
  5. Run an automated release

    develop

    To initiate the automated release process, trigger the release-prepare.yml workflow. This will open a release PR where you can review changes and edit CHANGELOG.md. Once you squash-merge that PR into develop, the Post Release workflow will automatically merge develop into master (triggering the chart release via cr) and open a version bump PR for the next cycle.

    You can trigger the workflow via the GitHub Actions UI or the GitHub CLI.

    gh workflow run release-prepare.yml
  6. Release charts manually

    develop

    If the automated workflows fail, follow these steps to perform a manual release:

    1. Update Vector version: Run .github/release-vector-version.sh to update the Vector image version, then run helm-docs.
      • Commit message: feat(vector): Bump Vector to <version> and update Helm docs
    2. Regenerate Changelog: Run .github/release-changelog.sh to regenerate the CHANGELOG.md.
      • Commit message: feat(vector): Regenerate CHANGELOG for <version>
    3. Merge to develop: Submit both commits as a single PR to develop and merge it.
    4. Trigger release workflow: Merge develop into master:
      git switch master && git pull
      git merge develop
      git push
    5. **Bump chart version**: Bump the chart minor version in `charts/vector/Chart.yaml`, run `helm-docs`, and open a PR to `develop`.
    
  7. Migrate from `vector-aggregator` to the new Vector chart

    develop

    The Vector chart has undergone significant changes in default configurations. By default, Vector is deployed as a StatefulSet and the role option should remain set to "Aggregator".

    Default Configuration Changes

    New default behaviors include:

    • Vector's API is enabled by default.
    • Several sources are now included by default: datadog_agent, fluent, logstash, splunk_hec, statsd, and syslog.
    • A console sink is included by default.
    • The prometheus_exporter sink has replaced the old prometheus_sink.

    Reverting to Original Defaults

    To maintain the original vector-aggregator behavior, use the following customConfig in your values.yaml:

    customConfig:
      data_dir: /vector-data-dir
      api:
        enabled: false
      sources:
        vector:
          address: 0.0.0.0:9000
          type: vector
          version: "2"
        internal_metrics:
          type: internal_metrics
      sinks:
        prometheus_sink:
          type: prometheus_exporter
          inputs: [internal_metrics]
          address: 0.0.0.0:9090
  8. Configure Datadog Agents to forward to Vector

    develop

    By default, Datadog Agents forward data directly to Datadog. To route logs and metrics through your Vector Aggregators, you must create a dd-agent.yaml values file and use it during the Datadog Helm installation.

    For Datadog Agent version 7.35/6.35 or greater

    Use the customAgentConfig.vector block. Note that if using minikube, you may need to uncomment kubelet_tls_verify: false.

    ---
    datadog:
      logs:
        enabled: true
        containerCollectAll: true
    agents:
      useConfigMap: true
      customAgentConfig:
        # kubelet_tls_verify: false
        vector:
          logs:
            enabled: true
            url: http://vector-haproxy.vector:8282
          metrics:
            enabled: true
            url: http://vector-haproxy.vector:8282

    For Datadog Agent version 7.34/6.34 or older

    Metrics will not be routed through Vector for these versions. Use the logs_config block:

    ---
    datadog:
      logs:
        enabled: true
        containerCollectAll: true
    agents:
      useConfigMap: true
      customAgentConfig:
        # kubelet_tls_verify: false
        logs_config:
          logs_dd_url: vector-haproxy.vector:8282
          logs_no_ssl: true
          use_http: true

    Install Datadog Agent

    Install the Datadog chart using your generated values file:

    helm install datadog datadog/datadog --namespace datadog --create-namespace \
    	--values dd-agent.yaml --set datadog.apiKey="${DATADOG_API_KEY}"