LoongCollector Documentation

The open-source version of LoongCollector primarily uses a local configuration management mode. For scenarios with many instances, managing configuration changes individually is complex, and monitoring version information and running status across multiple instances is difficult.

ConfigServer is a centralized management tool designed to address these challenges. It provides a global control plane for LoongCollector instances with the following capabilities:

• Registration: Allows LoongCollector instances to register themselves with the ConfigServer.
• Agent Grouping: Enables unified management of LoongCollector instances organized into Agent groups.
• Remote Batch Configuration: Supports pushing and updating collection configurations to LoongCollector instances remotely in batches.
• Status Monitoring: Provides centralized monitoring of the running status of LoongCollector instances.

Version 1.6.0 introduced several key features and improvements:

Protocol & Decoder Support

• OpenTelemetry Metrics: Added support for decoding OpenTelemetry metrics to the SLS Log Protocol. Both service_otlp and service_http_server v1 pipelines now support OTLP metrics.
• InfluxDB: Implemented the DecodeV2 method for InfluxDB.
• OTLP Input: service-otlp-input now supports gzip decompression.

Plugin & Metric Improvements

• Kubernetes: metric_meta_kubernetes now supports collecting kruise CRD metadata.
• Kafka/Pulsar: Optimized flusher pulsar and flusher kafka v2 for better performance in static topic scenarios. The service_kafka input plugin was rewritten.
• Prometheus: Added more Prometheus service discovery support and optimized the Prometheus decoder by 20%.

Performance & Configuration

• Processor Performance: Improved processor desensitize performance by 20-80%.
• Config Server: Changed the Config Server's GET method to POST to allow carrying payloads for browser-based interactions.

LoongCollector is a high-performance observability data collector optimized for log scenarios, with expanding capabilities in metrics and tracing. It differentiates itself from competitors like Fluent Bit, OpenTelemetry Collector, and Vector through the following strengths:

Key Advantages

• Log Collection: Strong capabilities with rich collection and processing plugins. It is highly Kubernetes-friendly, excelling in Stdout collection and AutoTagging.
• Metrics & Tracing: Provides native support for host scenarios and Prometheus scraping, with ongoing enhancements via eBPF. It can act as a proxy for tracing scenarios.
• Performance & Reliability:
  • High Throughput: Capable of processing logs at speeds up to 200MB/s.
  • Low Resource Overhead: Optimized for minimal footprint.
  • Reliability Mechanisms: Features a complete checkpoint mechanism, multi-level high/low watermark feedback queues, multi-tenant isolation, and overall resource control.
• Programmability:
  • Plugin Languages: Supports development in C++ and Go.
  • Advanced Processing: Features strong SPL (Stream Processing Language) for high-performance processing and orchestration.
  • Pipeline Flexibility: Offers highly composable multi-language collection configuration pipelines.
• Management & Control:
  • Global Management: Supports an open management protocol, machine group management, heartbeat management, and configuration hot-reloading via ConfigServer.
  • Kubernetes Integration: Commercial versions support K8s Operator and CRDs.

LoongCollector, originally known as iLogtail, is a high-performance observability data collector that has evolved through several key development stages:

1. iLogtail Stage: Focused on solving monitoring and observability challenges for large-scale clusters (e.g., Alibaba's Feitian 5K project), providing millisecond-level log tailing latency.
2. Alibaba Group Stage: Scaled to support millions of nodes across Alibaba and Ant Group, introducing multi-tenancy, advanced log processing (Regex, JSON, delimiters), and high reliability for critical data like billing and transactions.
3. Cloud Native Stage: Transitioned to support containerized environments (K8s), introduced a plugin system (Input, Processor, Aggregator, Flusher), and enabled configuration via Kubernetes Operators (using AliyunLogConfig CRDs).
4. Open Source & LoongCollector Stage: The project rebranded to LoongCollector to reflect its mission as a unified, high-performance, and programmable observability agent designed for next-generation observability pipelines.

LoongCollector is available in two editions: Community Edition and Commercial Edition. Use the following comparison to determine which edition meets your deployment and management requirements.

Key Differences

Shared Capabilities

Both editions support:

• Deployment Modes: Linux, Kubernetes DaemonSet, and Kubernetes Sidecar.
• Data Types: Logs, Traces, and Metrics.
• Performance: Single-core performance of ~100MB/s (simple) or ~20MB/s (regex); supports 2000+ configurations.
• Reliability: CheckPoint support.
• Advanced Control: Historical data import, multi-tenant isolation, and collection degradation/recovery.

Version 2.0.0 introduced several significant architectural and functional updates:

Core Architecture

• Input and Flusher Interfaces: Added new public interfaces for inputs and flushers.
• Pipeline Definition: Support for complete pipeline definitions.
• V2 Configuration: Support for the new V2 configuration format.
• Environment Configuration: Upgraded environment variable logic using the new Pipeline API for logtail environment management.

Data Processing & Protocols

• JSONLine Protocol: Added support for the jsonline protocol.
• SPL Processor: Added support for the processor_spl (Structured Processing Language) in ilogtail for enhanced data processing. Note: This is unloaded by default; a switch is required to control its loading.
• Loki Flusher: Added support for contents only mode for the Loki flusher.

Configuration & Metadata

• Tagging Behavior: Added UsingOldContentTag. When set to false, Tags are placed in Meta instead of Logs during cgo operations.
• Kafka Configuration: Added MaxOpenRequests to Kafka network configuration.
• Self-Metrics: Added support for labels in self_metric.
• Flusher Settings: Added flusher_http queue buffer and async interceptor settings.