Fluent Bit Documentation

Fluent Bit is an open-source telemetry agent designed to process logs, metrics, traces, and profiles. It is optimized for high performance and low resource consumption, making it suitable for environments ranging from constrained edge systems to complex cloud infrastructures.

Key Capabilities

• Telemetry Processing: Handles logs, metrics, traces, and profiles.
• Vendor-Neutrality: Supports native OpenTelemetry (OTLP) for both ingestion and delivery, and integrates seamlessly with ecosystems like Prometheus.
• Deployment Models:
  • Edge Agent: Deployed locally for localized telemetry data handling.
  • Central Aggregator/Collector: Utilized to manage telemetry data across multiple sources and environments.
• Observability Enhancement: Rather than replacing existing layers, it adapts and optimizes logging layers while adding metrics and traces processing.

The Splunk output plugin allows you to ingest logs and metrics into a Splunk Enterprise service using the HTTP Event Collector (HEC) interface.

To use this plugin, you must configure the connection to your Splunk HEC endpoint, typically requiring a splunk_token or basic authentication credentials (http_user and http_passwd).

Supported event types: logs, metrics

The disk input plugin gathers disk throughput information from the running system at a specified interval. It produces log-based metrics (e.g., JSON payloads).

Note: If you require Prometheus-based metrics instead of log-based metrics, use the Node Exporter Metrics input plugin instead.

fluent-bit -i disk -o stdout

The Memory (mem) input plugin gathers memory and swap usage on Linux at a fixed interval. It reports totals and free space as log-based metrics.

Note: This plugin emits logs event types. If you require Prometheus-format metrics, use the Node Exporter metrics input plugin instead.

The azure_kusto output plugin allows you to ingest logs into an Azure Data Explorer cluster or a Microsoft Fabric Eventhouse cluster using the Queued Ingestion mechanism.

To ingest data successfully, you must ensure a table exists in your Kusto database with a schema that matches the incoming JSON payload. By default, Fluent Bit expects a table with the following three properties:

• log: the actual event payload (type dynamic)
• tag: the event tag (type string)
• timestamp: the event timestamp (type datetime)

You can also use a custom JSON ingestion mapping by configuring the ingestion_mapping_reference key.

.create table FluentBit (log:dynamic, tag:string, timestamp:datetime)

The kinesis_streams output plugin allows you to ingest records into Amazon Kinesis Data Streams. This plugin is a high-performance C implementation that uses a random partition key when uploading records via the PutRecords API.

Note: This plugin is named kinesis_streams to avoid conflict with the older Golang-based kinesis plugin.

To use this plugin, you must ensure your AWS environment has the necessary permissions and that you provide the required configuration such as region and stream name.

pipeline:
  outputs:
    - name: kinesis_streams
      match: '*'
      region: us-east-1
      stream: my-stream

The process_exporter_metrics plugin collects process-level system metrics (CPU, memory, I/O, etc.) by reading from the Linux /proc filesystem.

Key constraints:

• Linux Only: This plugin requires access to the proc filesystem and is not supported on macOS.
• Metrics Pipeline: Metrics collected by this plugin flow through a separate pipeline from logs; existing log filters do not operate on metrics.
• Threading: This input always runs in its own dedicated thread.

The cpu input plugin measures CPU usage for either a specific process or the entire system (per CPU core) on Linux systems. It generates log-based metrics (e.g., JSON payloads) reporting usage in percentages (0-100%).

Note: This plugin is for log-based metrics. If you require Prometheus-based metrics, use the Node Exporter Metrics input plugin instead.

The forward output plugin enables interoperability between Fluent Bit and Fluentd by using the Forward protocol. It supports routing logs, metrics, and traces to a local or remote Fluentd instance.

It supports two transport modes:

• Forward (TCP): Uses a plain TCP connection.
• Secure Forward (TLS): Uses TLS encryption when enabled.

fluent-bit -i INPUT -o forward://HOST:PORT