DOCSEARCHbyAlgolia
Submit Docs

DeepFlow Observability Platform Documentation

repository·main·Indexed 26 days ago

https://github.com/deepflowio/deepflow

DeepFlow is an observability platform providing zero-code visibility into cloud-native and AI applications using eBPF and Wasm. It enables full-stack correlation across metrics, tracing, and profiling. The platform includes deepflow-agent and libtrace for kernel and userspace probing of protocols such as HTTP, gRPC, MySQL, and Redis, and supports deployment via Helm or docker-compose.

Tokens
93.5K
Snippets
243
Records
554
Agent score
88%

What's inside DeepFlow

  2. Overview of FlowMetrics Data Storage

    main

    FlowMetrics is a daemon used to receive trident packet header statistics. It stores various types of flow and packet metrics in a time-series database.

    Important Note: For all 'edge' databases (which represent bidirectional flows between two endpoints), both tx (transmitted) and rx (received) statistics are calculated from the perspective of the client.

  3. Overview of DeepFlow Observability

    main

    DeepFlow is an observability platform designed for cloud-native and AI applications. It provides Zero Code data collection using eBPF for metrics, distributed tracing, request logs, and function profiling. Key capabilities include:

    • Universal Map: Provides service maps for application, AI, and infrastructure services using eBPF and Wasm plugins for private protocols.
    • Distributed Tracing: Zero-instrumentation tracing for any language or infrastructure (gateways, service meshes, databases, etc.).
    • Continuous Profiling: Low-overhead (<1%) profiling for OnCPU/OffCPU/GPU/Memory/Network function call stacks.
    • SmartEncoding: Reduces storage overhead by up to 10x compared to ClickHouse by using pre-encoded meta tags.
    • Seamless Integration: Supports Prometheus, OpenTelemetry, SkyWalking, and Pyroscope. Provides SQL, PromQL, and OLTP APIs for data querying.

  4. Overview of DeepFlow

    main
    DeepFlow is an open-source observability platform designed for complex cloud-native and AI applications. It uses eBPF to achieve zero-code data collection for metrics, distributed tracing, request logs, and function profiling. By combining this with SmartEncoding, it provides full-stack correlation and efficient access to observability data across the entire stack, from code to infrastructure.

  5. Overview of DeepFlow L7 Protocol Processing

    main

    In deepflow-agent, data packets transition from raw bytes to application-layer structures through several key components and interfaces:

    • L7Protocol: Located in agent/crates/public/src/l7_protocol.rs, used to identify protocol constants.
    • L7ProtocolParser: Located in agent/src/common/l7_protocol_log.rs, a trait used for protocol identification and parsing into L7ProtocolInfo.
    • L7ProtocolInfo: Located in agent/src/common/l7_protocol_info.rs, the structure parsed by the parser used for session aggregation.
    • L7ProtocolInfoInterface: Located in agent/src/common/l7_protocol_info.rs, an interface that all L7ProtocolInfo implementations must satisfy.
    • L7ProtocolSendLog: Located in agent/src/flow_generator/protocol_logs/pb_adapter.rs, the unified structure sent to deepflow-server.

    Processing Flow:

    1. Identification: For unmarked packets, the agent iterates through parsers calling L7ProtocolParser::check_payload(). If it returns true, the flow is marked with the protocol.
    2. Parsing: The agent calls L7ProtocolParser::parse_payload() to generate L7ProtocolInfo.
    3. Aggregation: L7ProtocolInfo is added to a session for request/response matching. When a match is found, L7ProtocolInfo::merge_log() is called.
    4. Reporting: The info is converted via L7ProtocolSendLog::from(L7ProtocolInfo) and sent to the server.

  7. Understand the Data Receiving mechanism

    main

    The data receiving pipeline in libtrace uses a multi-threaded approach to move data from kernel eBPF events to userspace callbacks:

    1. Polling: The perf_reader_poll function continuously monitors the perf buffer.
    2. Dispatching: When data is detected, reader_raw_cb is triggered, which uses dispatch_queue_index to enqueue data into specific ring buffers (e.g., ring 0, ring 1, ..., ring n).
    3. Worker Threads: Dedicated worker threads perform process_data by dequeuing data from the ring buffers using ring_sc_dequeue_burst.
    4. Callback Execution: The worker thread executes prefetch_and_process_data, which ultimately invokes a Rust callback function. After the callback completes, the data is freed.

  9. Understand DeepFlow Meta Collection and Synchronization

    main

    DeepFlow collects infrastructure metadata to provide context to network flows.

    Metadata Sources:

    • Kubernetes: The deepflow-agent.k8s uses a ResourceWatcher (via ApiWatcher) to listen to the K8s API Server (list, watch). It uses ActivePoller (requires SYS_ADMIN for setns) or PassivePoller (using AF_PACKET) to sync Pod and Node MAC addresses.
    • Hosts: PlatformSynchronizer collects host information (hostname, IP) from the host.
    • Cloud: controller.cloud collects cloud-specific metadata from Cloud APIs.

    Metadata Storage and Sync:

    1. Ingestion: Metadata is sent via gRPC (e.g., GenesisSync, KubernetesAPISync) to controller.genesis in the deepflow-server.
    2. Processing: controller.genesis and controller.cloud pass data to controller.recorder.
    3. Persistence: controller.recorder writes metadata to MySQL Meta Tables.
    4. Tagging: controller.tagrecorder uses metadata from MySQL to generate tags in MySQL Tag Tables.
    5. Downstream Sync: MySQL Tag Tables are synchronized to ClickHouse to enable high-performance querying of tagged flows.

  10. Understand DeepFlow Server Architecture

    main
    The DeepFlow server architecture is composed of three primary components: controller, querier, and ingester. These components work together to manage cloud/container resources, provide a unified query language, and handle high-performance data ingestion and tagging.

  11. Understand the Labeler's role in DeepFlow

    main

    The Labeler is responsible for enriching flows with metadata. It attaches two primary types of information to flows:

    1. Cloud Platform Information: Metadata related to the underlying infrastructure, such as the project the device belongs to, device ID, device type, and subnet information.
    2. DeepFlow Control Information: Metadata related to DeepFlow's internal management, including the resource group, labels, and whitelist status associated with the flow.

  12. DeepFlow Architecture and Components

    main

    The DeepFlow Community Edition architecture consists of two primary components:

    1. Agent: Runs on each K8s node, legacy host, or cloud host. It is responsible for AutoMetrics and AutoTracing data collection for all application processes on that host.
    2. Server: Runs in a K8s cluster. It provides Agent management, tag injection, data ingestion, and query services.