DOCSEARCHbyAlgolia
Submit Docs

GoFlow2 Documentation

repository·main·Indexed 21 days ago

https://github.com/netsampler/goflow2

A high-performance NetFlow, IPFIX, and sFlow collector written in Go. GoFlow2 converts diverse flow protocols into consistent serialized formats like Protobuf or JSON for downstream processing. It features a modular pipeline consisting of decoders, producers, formats, and transports (such as Kafka, File, or Stdout). The project includes an enricher component for adding GeoIP metadata (ASN and Country) using MaxMind GeoIP2 databases and provides deployment configurations for ELK and KCG (Kafka, Clickhouse, Grafana) stacks.

Tokens
17.3K
Snippets
42
Records
61
Agent score
71%

What's inside GoFlow2

  1. How GoFlow2 modularity works

    main

    GoFlow2 is designed as a modular pipeline consisting of several components:

    • Decoder: Converts the raw payload of a flow packet (NetFlow, IPFIX, sFlow) into a structured format.
    • Producer: Converts the decoded samples into a target format (e.g., Protobuf or JSON).
    • Format: Utilities to marshal messages (e.g., JSON or text).
    • Transport: Handles the delivery of the message (e.g., Kafka, File, or Stdout).

    This modularity allows developers to build custom collectors by replacing specific parts, such as using RabbitMQ instead of Kafka, or implementing custom decoders for protocols like MPLS.

  2. Understand the impact of queue_size on RAM usage

    main

    In buffered mode (blocking=false), RAM usage is directly dependent on the queue_size setting. Because UDP packets can be up to 9000 bytes, a large queue_size can lead to Out-Of-Memory (OoM) crashes if the packet rate exceeds decoding capacity for an extended period.

    Estimation Example: On a machine with 2GB of RAM, you can buffer approximately 222,222 packets (assuming no overhead and maximum packet size).

  3. Understand the NetFlow TemplateStore interfaces

    main

    The NetFlow decoder uses a specialized storage system for templates. The API is defined in decoders/netflow/template_store.go through three primary components:

    1. netflow.FlowContext: Carries routing metadata. It currently uses RouterKey to provide a per-router namespace for template operations.
    2. netflow.TemplateStore: The interface used directly by the decoder. The decoder calls AddTemplate for template sets and GetTemplate for data sets.
    3. netflow.ManagedTemplateStore: An extension of TemplateStore that includes lifecycle and operational methods. This interface is used by the application wiring to manage the store's lifecycle.

    Lifecycle methods in ManagedTemplateStore include:

    • RemoveTemplate
    • GetAll
    • Start
    • Close
    • Errors

  4. Understand the FlowStore storage model

    main

    FlowStore (found in pkg/flowstore/store.go) is a generic, protocol-agnostic, in-memory key/value storage engine. It is designed to be reusable for different types of data, such as NetFlow templates or flow counters.

    Core Capabilities:

    • Generic Keys/Values: Works with any key and value types.
    • Set vs. Add Semantics:
      • Set: Replaces the existing value with a new one (used for template access).
      • Add: Merges a delta into an existing entry (used for aggregated counters like packet/byte counts).
    • TTL Expiry: Supports manual and automatic expiration with various refresh modes.
    • Eviction: Supports FIFO (First-In-First-Out) eviction when a maximum size is reached.
    • Hooks: Provides lifecycle hooks for set, get, and delete events.
    • Lifecycle Management: Includes helpers like Start(), Close(), and periodic sweepers.

  5. Compare Get and GetQuiet operations

    main

    When retrieving values from a FlowStore, choose between Get and GetQuiet based on whether you need side effects like TTL extension or monitoring.

    FeatureGetGetQuiet
    TTL ExtensionCan refresh TTL if WithRefreshTTLOnRead() is enabled.Does not refresh TTL on read.
    HooksFires the OnGet hook.Does not fire OnGet.
    Use CaseStandard retrieval where activity should keep the entry alive.High-performance or non-intrusive lookups where you don't want to affect the entry's lifecycle.

  6. Understand the data flow in the KCG stack

    main

    The stack follows this data pipeline:

    1. GoFlow2: Collects NetFlow v9/IPFIX and sFlow packets and sends them as protobuf messages into Apache Kafka.
    2. Prometheus: Scrapes metrics from the GoFlow2 collector.
    3. Clickhouse: Consumes data from Kafka, stores raw data, and performs aggregations over specific columns using MATERIALIZED TABLES and VIEWS (defined in the Clickhouse schema script).
    4. Grafana: Visualizes the processed data via pre-made dashboards.

  7. Quickstart: Run GoFlow2 as a collector

    main

    GoFlow2 is a NetFlow/IPFIX/sFlow collector. To start a basic collector that prints JSON samples to stdout, download the latest release and run the binary. By default, it listens for sFlow on port :6343 and NetFlowV9/IPFIX on port :2055 via UDP.

    $ ./goflow2
    #!/bin/bash
./goflow2

  8. Requirements for compiling Protobuf for Golang

    main

    To compile the GoFlow2 protobuf definitions, ensure the following tools are installed and present in your system's PATH:

    1. protoc: The official protobuf compiler.
    2. protoc-gen-go: The Go plugin for protoc used to generate Go code from protobuf definitions.

    On macOS, a common practice is to place these binaries in /usr/local/bin.

  9. Deploy the Flows + Kafka + Clickhouse + Grafana + Prometheus stack

    main

    You can deploy a complete observability stack using the provided docker-compose.yml file. This stack includes Apache Kafka, GoFlow2, Prometheus, Clickhouse, and Grafana.

    Running the compose command will automatically build the GoFlow2 and Grafana containers.

    Note for Colima users: Colima does not support UDP port forwarding, which means flows will not be collected. To work around this, you can run GoFlow2 locally and point it to Kafka by adding 127.0.0.1 kafka to your /etc/hosts file.

    $ docker-compose up

  10. Configure FlowStore expiry and TTL modes

    main

    FlowStore allows you to control how long entries stay in memory using TTL (Time-To-Live) settings.

    Manual Control

    • WithTTL(ttl): Sets a specific TTL for a single write.
    • WithoutExpiration(): Disables expiration for a specific entry.
    • ExpireStale(): Immediately removes all expired entries.

    Automatic Control

    • WithDefaultTTL(ttl): Sets a default TTL for all new entries.
    • StartSweeper(interval) or Start(interval): Enables a background process that periodically checks for and removes expired entries.

    TTL Extension (Refresh) Modes

    Depending on your workload, you can choose how to extend the life of an entry:

    • none: Entries expire strictly when their TTL elapses.
    • on write: Use WithRefreshTTLOnWrite() to refresh the TTL every time a Set or Add operation occurs. (Common for active counters).
    • on read and write: Use WithRefreshTTLOnWrite() combined with WithRefreshTTLOnRead() to refresh the TTL on both updates and Get operations. (Common for templates to keep them alive as long as they are being actively read).

  11. Requirements for implementing a custom ManagedTemplateStore

    main

    If you need to provide a replacement for the default template store, your implementation must satisfy the netflow.ManagedTemplateStore interface and adhere to these requirements:

    1. Keying: Templates must be keyed by router, version, observation domain, and template ID.
    2. Snapshots: Must support GetAll() to provide snapshots of the store.
    3. Preloading: Must allow templates to be preloaded before Start() is called.
    4. Lifecycle Semantics: Must define clear behaviors for Start() and Close().
    5. Error Reporting: Must expose an Errors() method if the implementation performs any background persistence or asynchronous work.

  12. Configure GoFlow2 transport and format

    main

    GoFlow2 supports different transport mechanisms and output formats.

    Send to a file

    For log integrations (e.g., Loki, Splunk, Fluentd), redirect output to a file:

    $ ./goflow2 -transport.file /var/logs/goflow2.log

    Send to Kafka as Protobuf

    To send binary protobuf data to a Kafka cluster, use the following flags:

    $ ./goflow2 -transport=kafka \
  -transport.kafka.brokers=localhost:9092 \
  -transport.kafka.topic=flows \
  -format=bin

    Kafka Compression

    You can configure the Kafka producer compression type using -transport.kafka.compression. Supported codecs are defined in the Sarama documentation.

    -transport.kafka.compression=gzip
    # Example: Kafka transport with gzip compression
./goflow2 -transport=kafka -transport.kafka.brokers=localhost:9092 -transport.kafka.topic=flows -format=bin -transport.kafka.compression=gzip