DOCSEARCHbyAlgolia
Submit Docs

rdma-core

repository·master·Indexed 25 days ago

https://github.com/linux-rdma/rdma-core

Userspace components for the Linux Kernel's InfiniBand/RDMA subsystem, including essential libraries such as libibverbs, librdmacm, and libibumad, as well as service daemons. The project provides tools for RDMA connection management (ibacm), SRP SCSI target discovery (ibsrpdm), and a CMake-based build system for various Linux distributions.

Tokens
178.8K
Snippets
362
Records
827
Agent score
80%

What's inside rdma-core

  1. Overview of Pyverbs

    master
    Pyverbs is a Python API wrapper for rdma-core, providing access to the Linux userspace C API for the RDMA stack. It aims to simplify the steep learning curve of RDMA by reducing the overhead of initializing complex C structs, providing a smoother user experience, and serving as a test suite for rdma-core features.

  2. Overview of infiniband-diags utilities

    master

    The infiniband-diags package is a suite of utilities used to configure, debug, and maintain InfiniBand fabrics.

    Key Operational Modes:

    • Directed Route MADs: Base utilities use directed route Management Datagrams (MADs), allowing them to function even in unconfigured subnets.
    • LID Routed MADs: Higher-level utilities require LID routed MADs and may require access to the Subnet Administrator (SA) or Subnet Manager (SM).
    • SMPs (QP0): Many tools use Subnet Management Packets (SMPs) via Queue Pair 0 (QP0) to pull data directly from the Subnet Management Agent (SMA). Note that while this is common for diagnostics in broken or partially configured fabrics, it may require an MKey or may be restricted in certain Virtual Machine environments due to security policies.

  4. Use vendstat to query InfiniBand vendor specific functions

    master

    vendstat is a diagnostic tool used to access vendor-specific functionality beyond the standard InfiniBand specification by using vendor-specific Management Datagrams (MADs). It currently supports Mellanox InfiniSwitch-III (IS3) and InfiniSwitch-IV (IS4).

    Usage:

    vendstat [options] <lid|guid>
    vendstat [options] <lid|guid>

  5. What is the MANA Direct Verbs API and when to use it

    master

    The manadv API provides low-level, direct access to MANA devices, bypassing the general branching and abstractions found in libibverbs.

    When to use it: Use manadv when application performance optimization is more critical than portability. While libibverbs provides a portable, provider-agnostic interface, it introduces a performance penalty. manadv allows applications to perform mana-specific low-level operations directly.

    Key Constraints:

    • This version of the driver supports only one Queue Pair (QP) type: IBV_QPT_RAW_PACKET.
    • The IBV_QPT_RAW_PACKET QP shares the same hardware resources as the Ethernet port used in the kernel.
    • Exclusive Access Required: To use IBV_QPT_RAW_PACKET on a specific port, the port must not be used by any other software (including the Kernel). If the port is in use, ibv_create_qp() will fail with errno set to EBUSY.

  6. Use Full Offload Mode for ESP

    master

    When the IB_UVERBS_FLOW_ACTION_ESP_FLAGS_FULL_OFFLOAD flag is set in esp_attr->flags, the hardware automatically handles the ESP header and trailer during the cipher operation.

    Features in Full Offload Mode:

    • Automatic Header Management: The ESP header and trailer are added/removed automatically. The esn and spi are used to populate/check the header.
    • Anti-Replay: For decryption, the hardware performs anti-replay checks. Decryption failures cause the packet to be dropped.
    • Encapsulation: If IB_UVERBS_FLOW_ACTION_ESP_FLAGS_TUNNEL is set, an esp_encap specification is required to provide fields for the encapsulation header (IP tunnel/UDP).

    Fields used exclusively in Full Offload Mode (esp_attr):

    • spi: ESP Security Parameters Index.
    • seq: Initial 32 lower bytes of the sequence number.
    • tfc_pad: Length of Traffic Flow Confidentiality Padding (RFC4303).
    • hard_limit_pkts: Hard lifetime of the SA in number of packets (RFC4301). After this limit, the action drops future packets.

  7. Understand RDMA tag-matching offload

    master

    RDMA tag-matching offload allows hardware (HW) to accelerate MPI-style tag matching by offloading the matching of Send operations to Receive operations. This process is split between HW and Software (SW):

    • Hardware (HW): Holds a bounded prefix of Receive tags. It processes and transfers any 'expected' message that matches a tag held in HW. For the rendezvous protocol, HW also initiates the RDMA-Read data transfer and sends a notification upon completion.
    • Software (SW): Handles any 'unexpected' messages (where no matching Receive is posted) or messages whose tags are not currently held in HW.

    Tag-matching is defined for the RC (Reliable Connected) transport. Messages are encapsulated as standard RDMA-Send operations, meaning the sender requires no special hardware support.

  8. Modify or query Verbs objects via the DEVX interface

    master

    The DEVX API allows direct user-space access to the mlx5 device driver using the KABI mechanism. This enables interoperability where you can create objects using standard Verbs methods (e.g., ibv_create_qp) and then perform advanced modifications or queries using the mlx5dv_devx_* API. This approach allows for new device functionality to be used without requiring kernel changes.

    To use these functions, you must include <infiniband/mlx5dv.h>.

  9. Manage DR Flow Tables

    master

    DR tables are used to manage the execution order of packet processing. Packets begin traversing the steering domain tree at table level 0. Using rules and actions, packets can be redirected to other tables within the same domain.

    • Create a table: mlx5dv_dr_table_create(domain, level)
    • Destroy a table: mlx5dv_dr_table_destroy(table) (call only after all dependent resources are released).

  10. Understand the rdma-core licensing model

    master

    Most of the rdma-core software is provided under a Default Dual License. You can choose to use the software under either:

    1. OpenIB.org BSD (MIT variant)
    2. GNU General Public License (GPL) Version 2

    Files marked with See COPYING file follow this dual license. However, individual source files or specific directories may have different licenses that supersede the default. Always check the specific file or directory for its declared license.

  11. Memory management limitations in Pyverbs

    master
    Because Pyverbs is a Python wrapper, memory management is handled by the Python runtime. When Pyverbs allocates memory (for example, a user buffer for a memory region), the memory is accessible to the user, but the user does not directly control the allocation or deallocation of that memory.

  12. Manage DR Flow Matchers

    master

    A matcher is created within a table at a specific priority (lower values are checked first). A matcher holds multiple rules that all share the same mask (of type struct mlx5dv_flow_match_parameters), which defines the exact attributes compared by hardware steering. Only fields masked in the matcher should be populated in the rules created via mlx5dv_dr_rule_create().

    • Create a matcher: mlx5dv_dr_matcher_create(table, priority, match_criteria_enable, mask)
    • Destroy a matcher: mlx5dv_dr_matcher_destroy(matcher)
    • Set matcher layout: mlx5dv_dr_matcher_set_layout(matcher, matcher_layout)

    Matcher Layout Flags (enum mlx5dv_dr_matcher_layout_flags):

    • MLX5DV_DR_MATCHER_LAYOUT_RESIZABLE: Allows the matcher to resize its scale and resources based on rule insertion/removal.
    • MLX5DV_DR_MATCHER_LAYOUT_NUM_RULE: Provides a hint to the hardware about the expected number of rules, allowing for preallocation of resources (useful for non-resizable layouts).