cisco-system-traffic-generator/trex-core

Scapy is a powerful Python-based interactive packet manipulation program and library. It allows you to forge or decode packets for a wide variety of protocols, send them on the wire, capture them, and store/read them using pcap files.

Key capabilities include:

• Fast packet prototyping using default values.
• Replacing tools like hping, nmap, arpspoof, arping, tcpdump, and wireshark for many tasks.
• Handling specialized tasks like sending invalid frames, injecting 802.11 frames, or combining techniques like VLAN hopping and ARP cache poisoning.
• Cross-platform support (Linux, OSX, *BSD, and Windows).

TRex is a high-performance traffic generator designed to outperform traditional tools like iperf, netperf, and Linux kernel pktgen. It leverages COTS x86/ARM servers and NICs (Intel, Mellanox, etc.) to provide both stateless and stateful traffic generation.

Key Capabilities:

• High Throughput: Can generate up to 200Gbps of traffic.
• High Scale: Supports millions of real-world TCP/UDP flows and connection rates in the order of millions of connections per second (MCPS).
• Extensibility: Allows emulation of L7 applications (e.g., HTTP, HTTPS, Citrix) using a programmable language and the ability to modify L7 application fields (e.g., HTTP User-Agent).
• Stateful Traffic: Includes a TCP stack for stateful generation and an emulation layer for L7 simulation.

jsonpickle is a Python library designed for the serialization and deserialization of complex Python objects to and from JSON.

Unlike standard Python JSON libraries (such as the built-in json module, simplejson, or demjson) which are limited to Python primitives with direct JSON equivalents (e.g., dict, list, str, int), jsonpickle can handle much more complex data structures. It is highly configurable and extendable, allowing users to select or add different JSON backends.

The Scapy RPC Server provides a way to build and modify network packets using a JSON-RPC 2.0 specification over ZMQ. It operates on a Request-Response basis and does not support batched commands. It is primarily used to translate Scapy-like packet definitions into binary payloads and structured JSON data.

Key characteristics:

• Protocol: JSON-RPC 2.0
• Transport: ZMQ (ZeroMQ)
• Communication Pattern: Request-Response (no batching supported)

The Packet Builder Language uses a YAML-based format to define packet structures for use in a GUI. This format allows for dynamic packet building, field manipulation, and protocol stacking.

Key Capabilities:

• Parsing & Reversibility: Ability to parse buffers created by the tool and load/save packets via PCAP or JSON-RPC formats.
• Dynamic Field Manipulation: Set values for any field in any protocol and vary fields (like IP/MAC addresses) at runtime.
• Arbitrary Protocol Stacking: Define protocol stacks in any order using the YAML structure.
• Supported Layers: Supports L2 (Ethernet, 802.3, VLAN, MPLS), L3 (ARP, IPv4, IPv6, IP-in-IP), L4 (TCP, UDP, ICMP, IGMP), and L7 (Text-based protocols or Binary patterns like random, seq, or hex dumps).

The TRex Stateless (STL) API allows users to control high-performance traffic generation. The API is structured around several core components:

• STLClient: The primary client interface used to interact with the TRex engine.
• Traffic Profiles (STLProfile, STLStream): Abstractions used to define the characteristics of the traffic being generated.
• Packet Builder: Tools for constructing packets, including support for Scapy-based builders.
• Field Engine: A mechanism for dynamic packet field manipulation.
• Linux Namespace: Support for managing traffic within specific Linux network namespaces.
• Bird Integration: Tools for routing configuration, including Bird CFG Creator, PyBirdClient, and Bird Node components.

The Stateless (STL) API is used for high-speed, simple packet generation where each packet is independent of the previous one. It provides two primary interfaces:

1. STL Plugin API: For low-level control and custom packet generation logic via plugins.
2. STL NDR Bench API: For running Network Detection and Response (NDR) benchmarking tasks using stateless traffic.

Refer to the specific stl_plugin_api or stl_ndr_bench documentation for implementation details.

The lockfile package provides a unified API for file locking across Unix (Linux, Mac) and Windows platforms. It abstracts away platform-specific mechanisms like fcntl.flock on Unix and msvcrt.locking on Windows.

Default Behavior

• On Unix/Linux/Mac: Uses linklockfile.LinkLockFile (based on the link(2) system call).
• On Windows: Uses mkdirlockfile.MkdirLockFile (based on the mkdir(2) system call).

Requirements

• Requires Python 2.5 or later for out-of-the-box with statement support.
• The directory containing the target path must exist and be writable.

from lockfile import LockFile
lock = LockFile("/some/file/or/other")
with lock:
    print lock.path, 'is locked.'

TRex performance monitoring has evolved from using hard-coded boundaries (which caused false positives) to a data-driven approach. The current recommended architecture involves sending performance results to an ElasticSearch database and analyzing them using Kibana, Grafana, and Pandas.

This solution enables:

1. Automatic identification of performance breakage.
2. Time-based performance trend reporting.
3. Simple querying of historical performance data.
4. Easy collection and extension of new regression setup fields.

The workflow typically involves:

• Data Collection: Sending test results (JSON format) to ElasticSearch.
• Data Analysis: Using a Python Pandas module to parse JSON data and perform calculations (average, min, max, standard deviation).
• Visualization: Using Grafana for time-series dashboards and Kibana for raw data exploration.

The Advanced Stateful (ASTF) API is used for complex traffic scenarios that require maintaining state (e.g., TCP flows, handshake sequences). It provides two primary interfaces:

1. ASTF Plugin API: For implementing complex, stateful protocol behaviors via plugins.
2. ASTF NDR Bench API: For running Network Detection and Response (NDR) benchmarking tasks using stateful traffic.

Refer to the specific astf_plugin_api or astf_ndr_bench documentation for implementation details.