Deepfence PacketStreamer Documentation

PacketStreamer is designed for lightweight, efficient, and remote raw network data collection.

Use PacketStreamer instead of alternatives like PacketBeat or ksniff when you need to:

• Collect raw network data from multiple machines simultaneously.
• Centralize the collection into a single location for logging and analysis.
• Minimize the performance impact on remote hosts (sensors do not perform additional processing like transaction assembly or parsing).
• Capture traffic across diverse environments including Virtual Machines, Kubernetes, and AWS Fargate (Linux and Windows).

PacketStreamer uses a sensor-receiver model to collect network traffic for forensic analysis:

1. Sensors: Deployed on target servers (bare-metal, Docker hosts, Kubernetes nodes, VMs, or AWS Fargate). Sensors capture raw network packets, apply BPF filters to select specific traffic, and stream the data to a central receiver. Streams can be compressed and/or encrypted using TLS.
2. Receiver: A central process that accepts streams from multiple remote sensors and writes the collected packets into a local pcap capture file.

The resulting pcap file can be processed by tools like Zeek, Wireshark, or Suricata, or fed into Machine Learning models.

Plugins in PacketStreamer allow you to stream captured packets to various external storage services. You can configure plugins to operate at two different stages of the packet pipeline:

1. Sensor level: Locally captured packets on the target host are streamed directly through the plugin.
2. Receiver level: All packets collected from one or more sensors are aggregated and then streamed through the plugin.

Currently, the available plugin is:

• S3 (see S3 documentation for details)

PacketStreamer uses a distributed architecture consisting of sensors and a receiver to collect network traffic:

1. Sensors: Deployed on target hosts (VMs, Kubernetes nodes, Docker hosts, AWS Fargate, etc.). They capture raw network packets, apply BPF filters, and stream the traffic (optionally compressed or encrypted via TLS) to a central receiver.
2. Receiver: A central process that accepts multiple streams from remote sensors and aggregates them into a single local pcap capture file. This file can then be used with tools like Zeek, Wireshark, or Suricata.

This design allows for lightweight, high-performance remote packet capture with minimal impact on the target hosts.

The PacketStreamer quickstart provides several methods for deploying and using the tool depending on your environment. You can follow these guides to get started:

• Building: Instructions for building the project from source.
• Using locally: Instructions for running PacketStreamer on a local host.
• Using with Docker: Instructions for running PacketStreamer within Docker containers.
• Using on Kubernetes: Instructions for deploying PacketStreamer to Kubernetes clusters.
• Using on Vagrant: Instructions for using PacketStreamer within Vagrant environments.

When PacketStreamer is configured to write to stdout, you can pipe the output directly into Suricata. This is useful for real-time processing without intermediate file storage.

./packet-streamer receiver --config ./contrib/config/receiver-stdout.yaml | suricata -v -c /etc/suricata/suricata.yaml -r /dev/stdin

Before running PacketStreamer with the S3 plugin, you must provide AWS credentials using one of the following methods:

1. Environment Variables: Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. This method is recommended when running the sensor as root.
2. AWS Config File: Use the ~/.aws/config file (which can be generated via the aws configure command).

You can build a container image for PacketStreamer using the make command with the docker-image target. To create a smaller, optimized image, use the RELEASE=1 flag to build a stripped release binary.

make docker-image

# Alternatively, build a stripped release binary
make docker-image RELEASE=1

When PacketStreamer is configured to write to stdout, you can pipe the stream directly into Suricata. This avoids writing to a temporary file on disk.

./packet-streamer receiver --config ./contrib/config/receiver-stdout.yaml | suricata -v -c /etc/suricata/suricata.yaml -r /dev/stdin

Start a local development server using yarn start. This command opens a browser window and supports live reloading for most changes. You can specify a custom port using the --port flag if you do not want to use the default port 3000.

$ yarn start

Use the docker-bin target to build packetstreamer inside a Docker container. This method produces a binary that is statically linked with musl and libpcap, ensuring portability across different Linux distributions.

To build a stripped release binary using Docker, combine the docker-bin target with the RELEASE=1 parameter.