DOCSEARCHbyAlgolia
Submit Docs

Datadog Operator

repository·main·Indexed 18 days ago

https://github.com/datadog/datadog-operator

A Kubernetes-native operator for deploying, configuring, and managing the Datadog Agent via Custom Resources. Includes CLI tools such as operator-render for simulating reconciliation loops and generating manifests, check-operator for validating Agent rolling updates, and yaml-mapper for converting Helm values to DatadogAgent CRD specifications.

Tokens
93K
Snippets
211
Records
357
Agent score
62%

What's inside datadog-operator

  1. Overview of Datadog Operator

    main

    The Datadog Operator is a Kubernetes-native way to deploy and manage the Datadog Agent. Unlike the Helm chart, the Operator is integrated into the Kubernetes reconciliation loop and treats Datadog Agent configurations as first-class Kubernetes resources.

    Key Capabilities:

    • Configuration Validation: Prevents common configuration errors during deployment.
    • Resource Orchestration: Automates the creation and updating of Datadog Agent resources.
    • Status Reporting: Reports the Agent's configuration status directly within its Kubernetes Custom Resource Definition (CRD).
    • Advanced Deployment: Supports advanced DaemonSet deployments via the ExtendedDaemonSet.

    Advantages over Helm charts:

    • Built-in defaults based on Datadog best practices.
    • More flexible configuration for future enhancements.
    • Managed via the Kubernetes reconciliation loop.

  2. What is operator-render and how does it work?

    main

    Overview

    operator-render is a CLI tool designed to simulate the Datadog Operator's reconciliation loop offline. It allows you to take a DatadogAgent (DDA) and optional DatadogAgentProfile (DAP) manifest and generate the complete set of Kubernetes resources (DaemonSets, Deployments, RBAC, Services, etc.) that the operator would normally create in a live cluster.

    How it works

    The tool uses the operator's actual reconciler code via a fake Kubernetes client:

    1. Fake Client Setup: It builds a client pre-populated with your DDA, DAPs, and the DatadogAgentInternal (DDAI) CRD.
    2. Multi-pass Reconciliation: It runs the DDA reconciler multiple times to handle finalizers and create DDAI objects. It then runs the DDAI reconciler repeatedly until the rendered resources stabilize.
    3. Resource Assembly: Shared resources (like the local Agent Service) are assembled from contributions across multiple DDAIs.
    4. Serialization: It collects all resources, strips non-deterministic metadata (like uid and resourceVersion), sorts them by kind and name, and outputs them as YAML or JSON.

    Note: The binary loads the DDAI CRD from config/crd/bases/v1/ at runtime. Because this path is baked in at compile time, the binary must be run from within the source tree it was built from.

  3. What is a DatadogGenericResource (DDGR)?

    main

    The DatadogGenericResource (DDGR) is a Custom Resource Definition (CRD) that allows you to manage various Datadog resources (like Monitors, Dashboards, and Notebooks) as Kubernetes resources.

    It is the preferred method for managing resources that are also supported by older, specific CRDs (e.g., DatadogMonitor). DDGR is more flexible because its jsonSpec field accepts the exact JSON payload required by the Datadog API, making it easier to consume new Datadog API features without waiting for new CRD updates.

    A DatadogGenericResource requires two main fields:

    1. type: The specific Datadog resource type (e.g., synthetics_browser_test).
    2. jsonSpec: A JSON string representing the resource configuration as defined in the Datadog API.
    apiVersion: datadoghq.com/v1alpha1
kind: DatadogGenericResource
metadata:
  name: browser-test-example
spec:
  type: synthetics_browser_test
  jsonSpec: |- 
    {
      "config": { ... },
      "name": "Example Browser test"
    }

  4. Understand Datadog Operator Kubernetes permissions

    main
    The Datadog Operator requires RBAC (Role-Based Access Control) permissions to manage its own resources and the Datadog Agent components it deploys. Because the Operator manages the lifecycle of native Kubernetes resources (like DaemonSets or Deployments) and creates specific roles for Agent components, it requires a broad ClusterRole rather than a namespace-scoped Role.

  5. Identify minimal permissions for the Datadog Operator

    main

    To run without errors, the Operator requires a baseline set of permissions. This includes:

    • Leader Election: Managing leases in the coordination.k8s.io API group.
    • Custom Resource Lifecycle: Managing (create, get, list, watch, update, delete, patch) Datadog custom resources in the datadoghq.com API group. The specific resources required depend on which controllers are enabled (e.g., DatadogDashboard, DatadogMonitor, DatadogSLO, or DatadogGenericResource).
    • Managed Resources: If a custom resource manages native Kubernetes resources (e.g., the DatadogAgent controller managing a DaemonSet), the Operator must also have permissions for those native resources.

  6. GKE AllowlistSynchronizer integration

    main

    The operator manages a GKE AllowlistSynchronizer to handle workload exemptions. It creates or updates a resource named datadog-synchronizer of type auto.gke.io/v1.

    The synchronizer resolves a Workload Allowlist version based on the experimental.agent.datadoghq.com/autopilot-allowlist-version annotation. If this annotation is missing or invalid, it falls back to the DefaultWorkloadAllowlistVersion. The synchronizer points to a specific YAML definition: Datadog/datadog/datadog-datadog-daemonset-exemption-<version>.yaml.

  7. Distinguish between direct and indirect permissions

    main

    When configuring or troubleshooting permissions, distinguish between what the Operator needs to perform its job and what the Agent needs to function:

    Permission TypeResource/VerbsNeeded ByPurpose
    Directdaemonsets (create, update, delete)OperatorManaging Agent deployment lifecycle
    Directserviceaccounts (create, update)OperatorCreating Agent ServiceAccounts with appropriate permissions
    Indirectnodes (get, list, watch)Agent DaemonSetAccessing kubelet endpoints for metrics collection
    Direct & Indirectsecrets (create, get, list)Operator & AgentStoring/managing credentials

    Note: The Operator's broad permissions are used to create more restricted roles for Agent components, following the principle of least privilege.

  8. DDAI behavior with and without DatadogAgentProfiles

    main

    The number and purpose of DatadogAgentInternal (DDAI) resources created depend on whether you are using DatadogAgentProfiles.

    Without Profiles

    A single DatadogAgent creates exactly one DatadogAgentInternal with the same name and namespace. This DDAI mirrors the DatadogAgent configuration and manages all workloads.

    With Profiles

    When using DatadogAgentProfiles, the Operator creates:

    1. One default DatadogAgentInternal (for nodes not matched by any profile).
    2. One additional DatadogAgentInternal per profile.

    Important Note on Workload Distribution:

    • Profile-specific DDAIs: Run only the node Agent.
    • Default DDAI: Runs the Cluster Agent and Cluster Checks Runner.
    ### Example: Behavior With Profiles

$ kubectl get datadogagent -n datadog
NAME      AGE
datadog   10m

$ kubectl get datadogagentprofile -n datadog
NAME                AGE
high-memory-nodes   2m

$ kubectl get ddai -n datadog
NAME                AGENT     CLUSTER-AGENT   CLUSTER-CHECKS-RUNNER   AGE
datadog             Running   Running         Running                 10m
high-memory-nodes   Running                                           2m

  9. Prefer DatadogGenericResource for new SLOs

    main

    Deprecation Notice

    For new Service Level Objective (SLO) resources, it is recommended to use the DatadogGenericResource CRD with type: slo instead of the DatadogSLO CRD. DatadogSLO is being soft-deprecated and remains supported for existing users, but DatadogGenericResource (DDGR) is the preferred path for accessing new Datadog API capabilities.

    To migrate existing SLOs from DatadogSLO to DatadogGenericResource, refer to the DDGR migration guide.

  10. Compare Datadog Operator and Helm chart

    main
    While the Datadog Agent can be installed via the official Datadog Helm chart or a manual DaemonSet, the Datadog Operator is recommended for Kubernetes environments because it provides built-in best practices, better flexibility, and is treated as a first-class resource by the Kubernetes API through the reconciliation loop.

  11. Understand the Operator EKS Add-on structure

    main

    The operator-eks-addon is a wrapper Helm chart designed for installing the Datadog Operator as an EKS add-on. It includes the datadog-operator chart as a dependency.

    When using this add-on, it is critical to ensure version compatibility between the add-on chart, the operator, the CRDs, and the Datadog Agent/Cluster Agent components. Use the version mapping table to verify that your chosen operator-addon-chart version aligns with the expected versions of the underlying components.

  12. How Kubernetes State Metrics Core works

    main

    The v2 Kubernetes State Metrics check is embedded as a "core check" within the Datadog Agent. When a Cluster Check Runner is scheduled to run this check, the configured collectors are started as separate routines within the container itself.

    Key behaviors:

    • No external dependency: The agent does not need to monitor independent instances of kube-state-metrics.
    • Autodiscovery: The Datadog Node Agent is instructed to ignore independent Kubernetes State Metrics instances (identified by image name) to prevent duplicate data.
    • Migration Tip: If you previously used v1 of the check via Kubernetes annotations, you should remove them to avoid confusion.