DOCSEARCHbyAlgolia
Submit Docs

Elastic Agent

repository·main·Indexed 42 days ago

https://github.com/elastic/elastic-agent

A single, unified agent used to collect data and provide security protection across various environments. It can be managed via Fleet or run in standalone mode. Documentation includes guides for Kubernetes deployment via Helm, including the Kube-stack for OpenTelemetry observability, kube-state-metrics, Elastic AutoOps Agent, and Cloud Defend (Defend for Containers).

Tokens
121.2K
Snippets
293
Records
494
Agent score
54%

What's inside Elastic Agent

  1. Overview of Elastic Distribution for OpenTelemetry Collector components

    main
    The Elastic Distribution for OpenTelemetry Collector includes a variety of components categorized into Receivers, Exporters, Processors, Extensions, and Connectors. These components allow for the ingestion, transformation, and exportation of telemetry data (logs, metrics, and traces) specifically optimized for Elastic observability stacks.

  2. Overview of the Verifier Receiver

    main

    The Verifier Receiver is a custom EDOT (Elastic Distribution of OpenTelemetry) collector component. Its purpose is to verify permissions for integrations based on Identity Federation and report the results as OTEL logs to Elasticsearch.

    Key Features:

    • Multi-Provider Support: Extensible architecture for AWS, Azure, GCP, and Okta.
    • Active & On-demand Verification: Performs actual API calls (using api_call or dry_run methods) to check if permissions are granted or denied.
    • Structured Reporting: Outputs OTEL logs containing full policy and integration context, grouped by Identity Federation and policy for easier remediation.
    • Permission Registry: Uses an internal mapping to match integration types to required permissions.

  3. Use the built-in OpenTelemetry Collector in Elastic Agent

    main

    Starting with version 9.5+, Elastic Agent includes a built-in OpenTelemetry (OTel) Collector. This allows you to collect and forward traces, metrics, and logs directly to Elastic Observability using an open-source distribution of the OpenTelemetry Collector.

    To begin using this capability, follow the official quickstart guides for your specific environment.

  4. Explore Elastic Agent Helm chart configuration examples

    main

    The eck-integrations Helm chart provides several pre-configured examples for different deployment scenarios. You can use these examples as templates for your own values.yaml configuration.

    Available configuration patterns include:

    Kubernetes Deployments

    • Defaults: Standard Kubernetes deployment configuration.
    • KSM Autosharding: Configuration optimized for KSM autosharding.
    • Only Container Logs: Configured to collect only container logs.
    • Hints Autodiscover: Uses Kubernetes hints for autodiscovery.

    Fleet Managed Agents

    • Managed Agent: Standard Fleet-managed deployment.
    • Managed Agent with KSM Autosharding: Fleet-managed deployment optimized for KSM autosharding.

    Specialized Integrations & Security

    • Nginx: Custom integration setup for Nginx.
    • Multiple Integrations: How to configure the agent to run multiple integrations simultaneously.
    • Cloud Defend: Standalone deployment for Cloud Defend.
    • Custom RBAC: Examples for using user-created ClusterRoles and ServiceAccounts instead of defaults.

    System & Other

    • System: Example for custom authlogs paths.

  5. Configure the Kubernetes Autodiscover provider

    main

    The Kubernetes provider allows Elastic Agent to automatically discover and monitor Kubernetes resources. It can be used in two modes:

    1. Standalone Elastic Agent: Configured via the elastic-agent-standalone-kubernetes.yaml manifest. Both Conditions-based and Hints-based autodiscover are supported.
    2. Fleet Managed Elastic Agent: Enabled by default with default values. Conditions-based autodiscover is supported, but Hints-based autodiscover is not supported. Configuration follows the Fleet management documentation.

  6. What is the Elastic APM processor and how does it work?

    main

    The Elastic APM processor is an OpenTelemetry Collector component designed to enrich OpenTelemetry trace data with Elastic-specific attributes and metadata. This enrichment ensures that traces are compatible with Elastic APM UIs and features like service maps, transaction groups, and enhanced trace visualization.

    Key Concepts

    • Relationship with Connector: The processor works in tandem with the Elastic APM connector. While the processor enriches trace data, the connector generates pre-aggregated APM metrics from that trace data.
    • Deployment Modes:
      • Agent mode: Processes trace data received from OpenTelemetry SDKs before exporting to Elasticsearch.
      • Gateway mode: Enriches traces received from other collectors (running in agent mode) before ingestion.
    • Replacement for elastictrace: The elasticapm processor replaces the deprecated elastictrace processor. If upgrading from older versions, you must update your configuration to use elasticapm.
    • When NOT to use it: If you are using the Managed OTLP (MOTLP) endpoint, you do not need this processor because enrichment happens server-side. Using it with MOTLP can cause conflicts or duplicate processing.

  7. What is the APM Config extension?

    main

    The APM Config extension (apmconfigextension) is an OpenTelemetry Collector component designed for central configuration delivery to EDOT SDKs. It uses the Open Agent Management Protocol (OpAMP) to establish a bidirectional control channel between the Collector and an OpAMP-enabled APM Server endpoint.

    This allows configuration updates (such as sampling, attribute collection, security settings, and custom SDK options) to be retrieved and delivered dynamically to connected SDKs without requiring application restarts.

    Note: This extension is specifically for configuration synchronization; it does not modify telemetry or manage pipelines. It is required when using the Central Configuration feature for EDOT SDKs.

  8. What is the Kubelet stats receiver?

    main

    The Kubelet stats receiver is an OpenTelemetry Collector component used within Elastic Agent (and the edot-collector) to collect Kubernetes resource metrics. It queries the Kubelet's /stats/summary endpoint to retrieve and convert usage statistics into OpenTelemetry metrics.

    Metrics collected include:

    • Node metrics: CPU, memory, network, and filesystem usage.
    • Pod metrics: Resource usage and pod throttling.
    • Container metrics: Fine-grained CPU/memory usage and ephemeral storage.
    • Volume metrics: Filesystem and storage usage.

  9. What is the Kubernetes objects receiver?

    main

    The Kubernetes objects receiver (k8sobjects) is an OpenTelemetry Collector component used by Elastic Agent to collect Kubernetes API objects (such as pods, namespaces, deployments, and events) and emit them as log signals for Elastic Observability.

    It supports two collection modes:

    • pull: Periodically lists Kubernetes objects at fixed intervals.
    • watch: Streams real-time updates from the API server as objects are created, modified, or deleted.

    Common use cases include audit tracking, configuration drift detection, and correlating Kubernetes resource changes with application telemetry.

  10. Understand Elasticsearch exporter document routing

    main

    Documents are routed to indices or data streams using a hierarchy of modes. The first applicable mode is used:

    1. Static mode: Routes to logs_index, metrics_index, or traces_index if they are explicitly configured.
    2. Dynamic mode (Index attribute): Uses the elasticsearch.index attribute. Precedence: log/metric/span attribute $\rightarrow$ scope attribute $\rightarrow$ resource attribute.
    3. Dynamic mode (Data stream routing): Constructs a data stream name: ${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}.
      • data_stream.type is static (logs, metrics, or traces).
      • data_stream.dataset and data_stream.namespace are derived from attributes (with precedence: attribute $\rightarrow$ scope $\rightarrow$ resource).
      • Fallbacks: dataset $\rightarrow$ generic, namespace $\rightarrow$ default.
      • If the scope name matches /receiver/(\w*receiver), the capture group is used as the dataset.

    Note for otel mapping mode:

    • For Data Stream routing, data_stream.dataset always ends with .otel.
    • Span events are treated as separate documents and routed as logs (with data_stream.type: logs).
    • The elasticsearch.index attribute is removed from the final document.

  11. Understand Agent vs. Gateway modes in Elastic Agent

    main

    The Elastic Agent (edot-collector) operates in two primary deployment modes depending on your architecture:

    • Agent mode: The agent ingests data directly from local infrastructure (logs, metrics) and OpenTelemetry SDKs. It then forwards this data to Elastic or to another collector running in Gateway mode.
    • Gateway mode: The agent acts as a central aggregator. It ingests data from multiple collectors running in Agent mode and forwards the aggregated data to Elastic.

  12. How the Gateway Collector pipeline works

    main

    The Gateway Collector acts as the central ingestion point. Its configuration depends on your deployment model:

    Scenario A: Direct ingestion into Elasticsearch

    Used in self-managed or Elastic Cloud (ECH) environments. The Gateway performs central enrichment before exporting data directly to Elasticsearch.

    • Enrichment: Uses pipelines for application telemetry and host metrics.
    • Routing: Uses a routing connector to separate infrastructure metrics from other metrics. Infrastructure metrics are routed to an ECS-based pipeline using ECS-compatibility exporter mode. Other metrics are exported in OTel-native format to Elasticsearch using the elasticsearch exporter.

    Scenario B: Managed OTLP Endpoint

    Used when sending data to a managed Elastic OTLP endpoint.

    • Workflow: The Gateway receives data via the OTLP receiver, processes it through a batch processor, and exports it via OTLP to the managed endpoint.
    • Benefit: No Elastic-specific enrichment is required within the Kubernetes cluster, as enrichment is handled by the managed endpoint.