IPFIXcol2

pcap2flow is a utility designed to extract NetFlow v5/v9 and IPFIX packets from a PCAP file and replay them to a collector.

Key behaviors:

• Session Preservation: To avoid mixing packets from different original Transport Sessions (TS), the tool creates a new independent session to the target collector for each original TS found in the PCAP.
• Packet Filtering: All non-flow packets within the PCAP file are automatically ignored.
• Intended Use: This tool is strictly for testing collectors and their plugins using previously captured flow packets. It is not intended for production environments due to non-ideal performance.

The fdsdump tool reads flow data stored in the FDS format and allows you to display, filter, sort, and aggregate the records. It supports multiple output formats including formatted tables and JSON. You can specify input files using the -r flag, which also supports glob patterns and can be used multiple times to include multiple file sets.

fdsdump [options] -r <FDS_FILES>

The TCP input plugin receives IPFIX messages over the TCP transport protocol from one or more exporters and passes them into the collector. Because TCP is a reliable transport protocol, exporters can detect connection or disconnection, which avoids common issues with template retransmission and initial periods of inability to interpret flow records found in UDP-based collectors.

Key features:

• Reliability: Detects connection/disconnection.
• Compression Support: Automatically detects and handles IPFIX messages compressed using LZ4 stream compression (compatible with ipfixprobe).
• Concurrency: Multiple instances can run concurrently if they listen on different ports or local IP addresses.

<input>
    <name>TCP input</name>
    <plugin>tcp</plugin>
    <params>
        <localPort>4739</localPort>
        <localIPAddress></localIPAddress>
    </params>
</input>

IPFIXcol2 is a high-performance flow data collector that uses a plugin-based architecture. You can compose a pipeline by selecting combinations of the following plugin types:

• Input plugins: Receive NetFlow/IPFIX data (e.g., UDP, TCP, FDS File, IPFIX File). Multiple instances can run concurrently on different interfaces/ports.
• Intermediate plugins: Modify, enrich, or filter records (e.g., Anonymization using Crypto-PAn).
• Output plugins: Store or forward flows (e.g., JSON, ClickHouse, Kafka, FDS File, Viewer).

Note: Some plugins (like ClickHouse, UniRec, or lnfstore) require individual installation due to extra dependencies.

Statistics mode

Activated with the -I flag. It prints summary information about the flows in the FDS files, such as packet counts, byte counts, and flow counts (including directionality for biflows).

Lister mode

(Default) Displays individual flow records in a specified format (table or JSON). Records can be sorted and ordered by field values.

Aggregation mode

(TBD) Used for grouping and aggregating flow data.

When <templateInfo> is enabled, Template records are converted with the @type set to ipfix.template. These records define the structure of flow records and include:

• ipfix:templateId: A unique identifier for the Transport Session and Observation Domain.
• ipfix:fields: An array of JSON objects, each specifying a field's ipfix:elementId, ipfix:enterpriseId, and ipfix:fieldLength.

{
    "@type": "ipfix.template",
    "ipfix:templateId": 49171,
    "ipfix:fields": [
        {
            "ipfix:elementId": 16399,
            "ipfix:enterpriseId": 6871,
            "ipfix:fieldLength": 1
        }, 
        {
            "ipfix:elementId": 184,
            "ipfix:enterpriseId": 29305,
            "ipfix:fieldLength": 4
        }
    ]
}

The plugin converts and stores IPFIX flow records into a ClickHouse database using binary-level export (avoiding text SQL conversion) for high performance.

Key behaviors:

• High-Speed Export: Capable of hundreds of thousands to a million records per second.
• Customizable Mapping: You can map any IPFIX/NetFlow items to specific database columns.
• High Availability (HA): Supports multiple ClickHouse endpoints. It uses a failover strategy (not round-robin); if one endpoint fails, data is redirected to the next available one.
• Schema Requirement: The plugin does not automatically create tables. It expects the table and schema to exist beforehand and will error if the schema does not match your configuration.