amzn-drivers

The Elastic Network Adapter (ENA) driver is a FreeBSD kernel driver designed for high-speed, low-overhead network traffic processing. It supports a range of ENA devices across various link speeds (10GbE, 25GbE, 40GbE, etc.) and is link-speed independent.

Key features include:

• Multi-queue support: Provides multiple Tx/Rx queue pairs with dedicated MSI-X interrupt vectors per pair.
• Performance optimizations: CPU cacheline optimized data placement and support for Low-latency Queue (LLQ) mode on compatible devices.
• Scaling: Supports Receive-side scaling (RSS) for multi-core scaling, binding queue pairs to CPU cores and NUMA domains (on FreeBSD 12+).
• SR-IOV: Supports both Physical Function (PF) and Virtual Function (VF) devices.
• Reliability: Implements watchdog health monitoring for transparent device/driver recovery and provides debug logs.

The ENA PMD (Poll Mode Driver) is a driver for the Elastic Network Adapter (ENA) designed to work within the DPDK framework. This repository provides:

1. Release Notes: Detailed ENA PMD release notes for all DPDK releases.
2. vfio-pci Patch: A patch for the vfio-pci module that can be used as an alternative to igb_uio.
3. Driver Backports: The latest ENA PMD driver backports based on supported LTS (Long Term Support) branches.

Note: Backported versions may skip certain DPDK features (like device reset or Tx prepare) if they are not supported by the target DPDK version.

The repository includes the following driver implementations:

• Elastic Network Adapter (ENA):
  • Linux kernel driver
  • FreeBSD kernel driver
  • DPDK userspace driver
• Elastic Fabric Adapter (EFA):
  • Linux kernel driver
  • Peer memory Linux kernel driver (for EFA)

ENA LLQ (Low Latency Queue), also known as ENAv2, is a Tx submission ring located in the ENA device DRAM behind PCI BAR2. The driver maps this area as Write Combined and pushes Tx submission ring descriptors along with packet headers there. This mechanism improves both latency and Packets Per Second (PPS).

Starting with ENA driver v2.0, LLQ is the default mode of operation for the majority of instance types.

ENA Queues

An ENA queue consists of a Tx Submission and Completion ring, and an Rx Submission and Completion ring.

The number of queues available per ENI is generally calculated as: MIN(MAX_NUM_QUEUES_PER_ENI, NUM_OF_VCPUS).

• MAX_NUM_QUEUES_PER_ENI is typically 8 for most instance types, but can be up to 32 for network-accelerated instances.

IRQ Allocation

For each ENI, the driver allocates:

1. 1 IRQ for management (Admin CQ, AENQ).
2. 1 IRQ for each ENA queue (Note: The IRQ for an ENA queue is shared between its Tx and Rx Completion rings).

The choice depends on your security requirements and hardware:

• igb_uio: Simpler to use but provides less security because it lacks IOMMU support.
• vfio-pci: Required if IOMMU is enabled (standard on AWS *.metal instances).

Important Notes for vfio-pci:

• On systems without IOMMU, vfio-pci must be rebuilt with the CONFIG_VFIO_NOIOMMU flag and no-IOMMU mode must be enabled.
• To support Write Combining (improving performance), use the patch from the enav2-vfio-patch repository.
• The Rx interrupt feature of the ENA PMD requires vfio-pci because it provides multiple interrupts (the first is reserved for the ENA admin queue).

The ENA driver supports Hardware Packet Timestamping (PHC).

Depending on the driver version, PHC can be enabled and managed through different interfaces:

• Devlink (r2.16.0+): Use Devlink for PHC enablement and to view PHC stats via Debugfs.
• Module Parameters (r2.8.3+): PHC can be enabled via module parameters.

PHC features include hardware timestamping support, extended cdesc support (r2.15.0), and error statistics (r2.15.0).

ENA hardware may only support a subset of RSS configuration features (Indirection table, Hash key, or Hash fields). Because DPDK's rte_eth_dev_rss_hash_update() attempts to update both the key and the functions simultaneously, the driver may return a warning if one is unsupported by the hardware.

If you see the warning Setting RSS hash fields is not supported..., it means the key was updated successfully, but the driver fell back to default hardware hash functions.