DOCSEARCHbyAlgolia
Submit Docs

Elastic Common Schema (ECS)

repository·main·Indexed 22 days ago

https://github.com/elastic/ecs

Definitions and tooling for a standardized field schema used to ingest and correlate logs, metrics, and security data in Elasticsearch. Includes documentation on defining field sets and individual fields via YAML, the RFC process for schema changes, and a generation pipeline using scripts/generator.py to produce artifacts such as Markdown documentation and Elasticsearch templates. Aligned with OpenTelemetry Semantic Conventions.

Tokens
187K
Snippets
274
Records
588
Agent score
73%

What's inside elastic-ecs

  1. Overview of the Markdown Documentation Generator

    main

    The Markdown Generator (generators/markdown_fields.py) is the final stage in the ECS documentation pipeline. It transforms structured YAML field schemas into human-readable markdown files used for official Elastic documentation.

    It generates several types of documentation:

    • Field Reference Pages: A complete catalog of all ECS fields.
    • Fieldset Pages: Detailed documentation for specific groups like HTTP, User, or Process.
    • OTel Alignment Documentation: Statistics and details showing how ECS converges with OpenTelemetry.
    • Index and Navigation: Landing pages and cross-references.

  2. Overview of ECS generator modules

    main

    The ECS toolchain is composed of several specialized generator modules located in scripts/generators/. Each module handles a specific output format or integration:

    ModulePurpose
    otel.pyOpenTelemetry Semantic Conventions integration and validation
    markdown_fields.pyMarkdown documentation generation using Jinja2 templates
    intermediate_files.pyGenerates flat and nested intermediate representations used as a bridge between schema processing and final artifacts
    es_template.pyGenerates Elasticsearch templates (supporting both Composable and legacy formats)
    csv_generator.pyExports field references to CSV for use in spreadsheets (Excel, Google Sheets)
    beats.pyGenerates YAML field definitions for Elastic Beats
    ecs_helpers.pyShared utility functions used across the toolchain

  3. Overview of the Elasticsearch Template Generator

    main

    The Elasticsearch Template Generator (generators/es_template.py) is a tool that converts Elastic Common Schema (ECS) field schemas into Elasticsearch index templates. These templates define the mappings (field types and properties) required for indices storing ECS-structured data.

    It supports two primary formats:

    1. Composable Templates (Modern): Uses modular component templates, recommended for Elasticsearch 7.8+.
    2. Legacy Templates (Deprecated): Produces a single monolithic template, used for older Elasticsearch versions or backwards compatibility.

  4. Overview of the ECS Schema Processing Pipeline

    main

    The ECS schema processing pipeline is a multi-stage system that transforms raw YAML schema definitions (located in schemas/*.yml) into various consumer-facing formats such as Elasticsearch templates, Beats configurations, CSV exports, and Markdown documentation.

    Pipeline Stages:

    1. loader.py: Loads YAML files and creates a deeply nested dictionary structure.
    2. cleaner.py: Validates the schema, normalizes data, and applies default values.
    3. finalizer.py: Handles field reuse logic and calculates final field names.
    4. Filters (Optional): subset_filter.py and exclude_filter.py can be applied to refine the schema.
    5. intermediate_files.py: Generates flat and nested YAML files.
    6. Generators: The final stage where specific outputs are produced via:
      • es_template.py (Elasticsearch templates)
      • beats.py (Beats field definitions)
      • csv_generator.py (CSV field export)
      • markdown_fields (Markdown documentation)
    ┌─────────────┐
│ YAML Schema │  Raw schema files in schemas/*.yml
│   Files     │
└──────┬──────┘
       │
       v
┌─────────────┐
│   loader.py │  Load & nest: YAML → deeply nested dict
│   │
       v
┌─────────────┐
│  cleaner.py │  Validate, normalize, apply defaults
│   │
       v
┌─────────────┐
│finalizer.py │  Perform field reuse, calculate names
│   │
       v          (Optional filters)
┌─────────────┐  ┌────────────────┐
│subset_filter│─>│exclude_filter  │
│    .py      │  │      .py       │
└──────┬──────┘  └──────┬─────────┘
       │                  │
       v                  v
┌─────────────────────────────┐
│   intermediate_files.py     │  Generate flat & nested YAML
│              │
               v
     ┌────────────────────┐
     │    Generators      │
     ├────────────────────┤
     │ • es_template.py   │  Elasticsearch templates
     │ • beats.py         │  Beats field definitions
     │ • csv_generator.py │  CSV field export
     │ • markdown_fields  │  Markdown documentation
     └────────────────────┘

  5. Identify Elastic products and solutions that support ECS

    main

    As of version 7.0, the following Elastic products and solutions provide out-of-the-box support for the Elastic Common Schema (ECS):

    • Beats: Data shippers that use ECS for field mapping.
    • APM (Application Performance Monitoring): Observability solution for application traces and metrics.
    • Elastic Security: Security solution for SIEM and endpoint protection. For specific field mappings used within the Security app, refer to the Elastic Security Field Reference.
    • Elastic Endpoint Security Server: Endpoint protection services.
    • Log Monitoring: Observability solution for log analysis.
    • Metrics Monitoring: Observability solution for infrastructure and host metrics.

    Additionally, various language-specific log formatters support ECS natively, and Logstash outputs for Elasticsearch include an ECS compatibility mode.

  6. Understand the scope of impact for ECS Asset Integration

    main

    The proposed ECS fields for asset integration primarily impact the following areas:

    • Ingestion mechanisms: The primary mechanism for ingesting this dataset is via Entity Analytics fleet integrations.
    • Usage mechanisms: The primary users of these proposed ECS fields and values will be the Elastic Security solution, specifically within Entity Analytics & Threat Hunting workflows.

  7. Status of Host Metrics RFC 0037

    main

    RFC 0037 regarding host metrics (CPU, Memory, Disk, Load) is currently inactive.

    • GA Fields: Only host.network.ingress.bytes and host.network.egress.bytes were merged into the ECS schema via OTel alignment.
    • Non-merged Fields: Proposed fields for cpu, memory, load, and filesystem were never merged into the official schema.
    • Future Work: If there is interest in the remaining fields, a new proposal must be submitted under the current ECS proposal process.

  8. What is Elastic Common Schema (ECS)?

    main

    The Elastic Common Schema (ECS) is an open-source specification that defines a common set of field names and Elasticsearch datatypes for storing event data (such as logs and metrics) in Elasticsearch.

    By normalizing event data according to ECS, you can better analyze, visualize, and correlate data across different sources. ECS is designed to be compatible with:

    • Diverse Event Sources: Elastic products, third-party products, or custom applications.
    • Various Ingestion Architectures: Beats processors, Logstash, Elasticsearch ingest nodes, or direct ingestion.
    • Multiple Consumers: APIs, Kibana queries, dashboards, and custom applications.

  9. Understand the concept of a Client in ECS

    main

    In the Elastic Common Schema (ECS), a client is defined as the initiator of a network connection for events regarding sessions, connections, or bidirectional flow records.

    Key Characteristics:

    • TCP Events: The client is the initiator of the TCP connection that sends the SYN packet(s).
    • Other Protocols: The client is generally the initiator or requestor in the network transaction.
    • Terminology: Some systems may use the term "originator" to refer to the client in TCP connections.
    • Context: Client fields describe the system acting as the client and are usually populated alongside server fields. They are generally not populated for packet-level events.
    • Best Practice: While client/server representations add semantic context helpful for visualization, you should always ensure that source and destination fields are filled appropriately regardless of the client/server labels.

  10. Distinguish between Host, Agent, and Observer

    main

    ECS uses distinct field sets to identify where an event happened and who captured it:

    • Host (host.*): The computing instance where the event occurred (e.g., physical device, VM, container, or cloud instance). Use cloud.* or container.* for specific details.
    • Agent (agent.*): The software that collected or observed the event (e.g., Beats, Elastic Agent). Includes details like the agent's version.
    • Observer (observer.*): An external monitoring or intermediary device (e.g., firewall, APM server, web proxy) that detects network, security, or application events.

  11. Implement ECS Base Fields

    main

    Base fields are top-level fields residing outside any specific field set. To ensure data consistency and discoverability, follow these conventions:

    • @timestamp: Required. Must be populated with the event's original timestamp.
    • message: Most events should populate this field with the raw or primary event message.
    • ecs.version: Populate this with the version of ECS used to develop the mapping or ingest pipeline. This helps detect when mappings are outdated or explain discrepancies between data sources.
    • tags and labels: Use these for simple metadata, stored as keyword values.

  12. Understand Destination fields [ecs-destination]

    main

    Destination fields capture details about the receiver of a network exchange or packet. These fields are typically populated from network events, packets, or other transactions containing network details.

    Usage Guidelines

    • Pairing with Source: Destination fields are usually populated in conjunction with source fields. Both source and destination should be considered the baseline and must be filled if the event contains details for both.
    • Role Identification: If the event identifies specific client and server roles, the client and server fields should also be populated alongside destination fields.