DOCSEARCHbyAlgolia
Submit Docs

Elastic Beats

repository·main·Indexed 11 days ago

https://github.com/elastic/beats

A collection of lightweight, Go-based data shippers used to collect operational data—such as logs, metrics, and network packets—and ship them to Elasticsearch or Logstash. Includes specific shippers such as Filebeat, Metricbeat, Auditbeat, and Heartbeat, with deployment guides for Kubernetes (via DaemonSets), Cloud Foundry, and local development using Kind and Tilt.

Tokens
301K
Snippets
904
Records
1.5K
Agent score
94%

What's inside Beats

  1. Overview of Beats and libbeat

    main

    Beats are lightweight, Go-based data shippers designed to capture operational data (logs, metrics, network packets, etc.) and send it to Elasticsearch or Logstash for visualization in Kibana. They are characterized by a small installation footprint, limited system resource usage, and no runtime dependencies.

    This repository contains libbeat, the Go framework used to create Beats, along with all officially supported Beats.

  2. What is Winlogbeat

    main
    Winlogbeat is an open-source log collector designed specifically for Windows Event Logs. It collects these logs and ships them to either Elasticsearch or Logstash. On Windows systems, it is designed to install and run as a Windows service.

  4. Overview of Filebeat

    main
    Filebeat is an open source file harvester designed to fetch log files and feed them into Logstash or other destinations. It is primarily used as a lightweight replacement for logstash-forwarder. It utilizes the libbeat lumberjack output to ship data efficiently.

  5. Overview of the Metricbeat Elasticsearch Module

    main

    The Elasticsearch module for Metricbeat collects various metrics from an Elasticsearch cluster by calling specific Elasticsearch API endpoints.

    Key characteristics:

    • Namespacing: Exported fields are namespaced using the pattern {module}.{metricset} (e.g., elasticsearch.cluster_stats.xxx).
    • Data Mapping: While exported fields generally follow the paths of the original Elasticsearch API responses, they may differ slightly based on the internal schema defined for each metricset.
    • Metricsets: The module is organized into 'metricsets', each targeting a specific functional area of Elasticsearch (such as nodes, indices, or cluster stats).

  6. Overview of the Filebeat threatintel module

    main

    The threatintel module ingests data from various threat intelligence sources. This data is designed for use with Indicator Match rules and is compatible with Enrich Processors.

    Ingested threat intelligence attributes are stored under the threat.indicator.* fields.

    Available filesets include:

    • abuseurl: URL entities from Abuse.ch.
    • abusemalware: Malware/Payload entities from Abuse.ch.
    • misp: Threat intel attributes from MISP (replaces the legacy MISP module).
    • malwarebazaar: Malware/Payload entities from Malware Bazaar.
    • otx: Threat intel attributes from AlienVault OTX.
    • anomali: Threat intel attributes from Anomali Limo.
    • anomalithreatstream: Threat intel attributes from Anomali ThreatStream.
    • threatq: Threat intel attributes from ThreatQuotient.

  7. What is Packetbeat

    main

    Packetbeat is an open source network packet analyzer that functions as a distributed real-time protocol analyzer. It sniffs network traffic between application processes and parses protocols on the fly, including:

    • HTTP
    • MySQL
    • PostgreSQL
    • Redis
    • Thrift

    Packetbeat correlates messages into transactions and ships each transaction as a JSON document to Elasticsearch. Once indexed, you can use Kibana to perform ad-hoc queries and visualize key metrics.

  8. Overview of the Threat Intel module

    main

    The Filebeat threatintel module ingests data from various threat intelligence sources. This data is designed to be used with Indicator Match rules or Enrich Processors.

    Key characteristics:

    • Ingested data is stored under threat.indicator.* fields.
    • It supports multiple filesets for different providers.
    • Note: For many of these use cases, Elastic recommends using Elastic Agent via Elastic Integrations instead of Filebeat.

  10. Overview of the Salesforce Filebeat Module

    main

    The Salesforce module collects logs from a Salesforce instance using the Salesforce REST API. It supports both real-time and historical data collection for various log types.

    Supported log types include:

    • Login: Login events (via EventLogFile or real-time Objects).
    • Logout: Logout events (via EventLogFile or real-time Objects).
    • APEX: APEX execution logs (via EventLogFile).
    • Setup Audit Trail: Configuration change events (via real-time Objects).

  11. What is Metricbeat

    main
    Metricbeat is a lightweight shipper that fetches a set of metrics at predefined intervals from the operating system and various services (such as Apache web server, Redis, and more). It then ships these metrics to Elasticsearch or Logstash for analysis and storage.

  12. Overview of the Filebeat AWS module

    main

    The AWS module for Filebeat is designed to ingest AWS logs from S3 buckets. It utilizes the Filebeat S3 input, which can retrieve log files via two methods:

    1. SQS Notification (Preferred): Uses an Amazon SQS queue to receive notifications when new files arrive in S3. This method is highly scalable and cost-effective.
    2. S3 Polling: Directly polls a list of S3 objects in a bucket. This is more expensive in terms of performance and cost and does not scale horizontally without risking ingestion duplication. Use this only if SQS notifications cannot be attached to your S3 buckets.

    Supported filesets include:

    • s3access: S3 server access logs.
    • elb: Elastic Load Balancing (ELB) access logs.
    • vpcflow: VPC flow logs.
    • cloudtrail: CloudTrail logs (events representing actions taken by users, roles, or services).
    • cloudwatch: General CloudWatch logs exported to S3.
    • ec2: EC2 logs stored in CloudWatch and exported to S3.