repository·main·Indexed 21 days ago
https://github.com/ofiwg/libfabricA high-performance networking framework for parallel and distributed applications and middleware. It provides a standardized API to access diverse networking hardware and protocols through a provider-based architecture. The project includes the ofi-libfabric-sys Rust crate for low-level FFI bindings, a suite of examples for RDM, MSG, and RMA messaging, and fabtests for comprehensive performance and functional testing across various providers.
Fabtests is a suite of example programs for libfabric providers designed to demonstrate various high-performance fabric features. While most tests report performance metrics, their primary purpose is to verify functionality and familiarize users with the libfabric API.
Most tests operate in a client-server model, meaning they require both a server process and a client process to run. Note that not all providers support every test in the suite.
The Elastic Fabric Adapter (EFA) Reliable Datagram (RDM) communication protocol version 4 (protocol v4) is the protocol used by the libfabric EFA provider's RDM endpoint (starting from libfabric 1.10.0).
This protocol provides a platform-independent definition for RDM communication, separating the protocol specification from specific implementations. It is structured into several functional areas:
Developers implementing or debugging EFA-based communication should refer to this protocol definition to ensure compatibility and correct behavior.
tcp provider runs over TCP (SOCK_STREAM) sockets and is compatible with all operating systems supported by libfabric. It implements a custom protocol over TCP/IP to support libfabric communication APIs. It can leverage platform-specific features like io_uring and zero-copy transfers to improve performance.The CXI provider enables libfabric on Cray's Slingshot network (comprised of Rosetta switches and Cassini NICs). While Slingshot is Ethernet-compliant, the CXI provider uses proprietary extensions to support high-performance computing (HPC) applications.
Key Capabilities:
For exhaustive technical details, refer to the fi_cxi(7) man page.
process_vm_readv/process_vm_writev calls. It is designed for high-performance communication between processes residing on the same physical system.The EFA Libfabric provider enables support for the Amazon Elastic Fabric Adapter (EFA), an OS bypass network interface available on Amazon EC2 instances. It is designed for High-Performance Computing (HPC) and Machine Learning applications to achieve lower latency and higher throughput compared to TCP.
Key features provided by the software layer include:
The provider supports both reliable and unreliable datagram send and receive semantics.
The EFA RDM (Remote Direct Memory Access) communication protocol allows two libfabric endpoints to communicate using an EFA device. While EFA hardware natively supports MTU-sized send/receive, 1GB RDMA reads, and 1GB RDMA writes, Protocol v4 extends these capabilities via software subprotocols to support:
FI_ORDER_SAS) and Delivery Complete (DC) notifications, which are not natively guaranteed by the EFA hardware.FI_PEER capability.In High-Performance Computing (HPC) applications communicating with thousands of peers, memory footprint is a critical factor for scalability. Excessive memory consumption can limit the number of parallel peers an application can support.
Key trade-offs include:
struct sockaddr_in requires 8 bytes per peer; communicating with a million peers consumes ~8 MB just for the address list.Registered memory regions (MRs) associate memory buffers with permissions for fabric resources. A buffer must be registered with a resource domain before it can be used for remote RMA, atomic data transfers, or certain collective operations.
Memory registration behavior is controlled by mr_mode bits. When calling fi_getinfo, an application should check these bits to understand the provider's requirements. If a provider supports a specific mode, it will leave the bit set; otherwise, it may clear it. Applications should be prepared to handle the mode even if the provider clears the bit (indicating the requirement is not strictly necessary for that specific provider).
The emulated long-read write subprotocol uses RDMA read to emulate a write operation. It is initiated with a LONGREAD_RTW packet. Unlike the long-read message subprotocol, the receiver side does not receive a completion event by default.
Key header fields include:
rma_iov_count and rma_iov: Information about the target buffer on the responder side (provided by the application via libfabric's write API).read_iov_count and read_iov: Information about the source buffer on the requester side (constructed by the write requester).### LONGREAD_RTW Mandatory Header
| Name | Length (bytes) | Type | C language type | Notes |
|---|---|---|---|---|
| `type` | 1 | integer | `uint8_t` | part of base header |
| `version` | 1 | integer | `uint8_t` | part of base header |
| `flags` | 2 | integer | `uint16_t` | part of base header |
| `rma_iov_count` | 4 | integer | `uint32_t` | number of RMA iov on the responder |
| `msg_length` | 8 | integer | `uint64_t` | total length of the message |
| `send_id` | 4 | integer | `uint32_t` | ID of the receive operation |
| `read_iov_count` | 4 | integer | `uint32_t` | number of iov on requester (to be read by responder) |
| `rma_iov` | `rma_iov_count` * 24 | array | `efa_rma_iov[]` | write iov information |