Unified Communication X (UCX)

Unified Communication X (UCX) is an optimized communication framework designed for modern, high-bandwidth, and low-latency networks. It provides abstract communication primitives that leverage hardware resources and offloads, including:

• RDMA (InfiniBand and RoCE)
• TCP
• GPUs
• Shared memory
• Network atomic operations

UCX provides a high-level API to mask low-level communication details while maintaining high performance and scalability across various message sizes.

Unified Communication X (UCX) is a set of network APIs and implementations designed for high-throughput computing. It acts as a communication middleware that abstracts vendor-specific software and hardware interfaces, bridging the gap between programming models (like MPI, OpenSHMEM, or task-based models) and network hardware.

UCX is designed for functional and performance portability across diverse architectures and programming models. It provides optimized protocols for data transfer between various memories and manages network resources to support high-performance, scalable networking.

UCX offers flexible methods for managing connections and resources:

Connection Models

• Client/Server: Establish connections using a client/server model similar to TCP.
• Direct Connection: Connect directly by passing a remote address blob.

Resource and Progress Models

• Threading: Supports sharing resources between multiple threads or allocating dedicated resources per thread.
• Progress Mechanisms: Supports both event-driven (interrupt-based) and polling-driven (busy-wait) progress models.

UCX provides native support for hybrid programming models (e.g., applications using both OpenSHMEM and MPI). It enables efficient resource sharing and progress engine coordination by allowing users to choose between:

1. A single shared UCX network context across different programming models.
2. Standalone UCX network contexts for each individual programming model.

This flexibility helps optimize memory usage and improves overall network and application performance.

UCX is organized into several functional layers:

UCX provides several high-level communication patterns for data transfer and synchronization. Developers can choose from the following API models depending on their application requirements:

• Stream-oriented operations: Send and receive operations based on a continuous stream of data.
• Tag-matched operations: Send and receive operations that use tags to match specific messages, similar to MPI semantics.
• Remote Memory Access (RMA): Direct access to memory on a remote node.
• Remote Atomic Operations: Performing atomic operations (like fetch-and-add or compare-and-swap) on remote memory.

UCX distinguishes between blocking and non-blocking routines, which affects how you manage resources and control program flow:

• Blocking routines: These return only when the UCX operation is fully complete. Once the routine returns, you can safely reuse the resources involved in that operation.
• Non-blocking routines: These return immediately, regardless of whether the operation has finished. You must not reuse the resources used by the routine until the operation is confirmed complete.

To avoid ambiguity and precedence issues, follow these rules for boolean logic:

• Explicit Comparisons: Use explicit comparisons for non-boolean values (pointers, counts, status/enum codes). Example: if (ptr == NULL) instead of if (!ptr).
• Direct Testing: Test boolean flags (integers holding 0/1, e.g., is_*, has_*) directly. Example: if (is_enabled).
• Parentheses: Add parentheses around every comparison in compound expressions to ensure correct precedence.
• Negation: Use ! without parentheses for single flags or predicate calls (e.g., !is_enabled).

When developing or extending UCX modules, memory optimization should focus on scalability. The goal is to ensure memory usage scales with the number of in-flight operations rather than the number of connections.

Key principles:

• Scalability: Minimize the dependency of memory usage on the number of connections created.
• Structure Size: Avoid enlarging the endpoint structure and remote memory key structure.
• Descriptor Scaling: The number of requests and other descriptors must be proportional to the number of in-flight operations, not the total number of connections.
• Memory Pools: Limit the number of buffers used from memory pools.
• Acceptable Overhead: Adding small fields to structures that exist per thread, process, or device (e.g., iface, worker, context, md) is acceptable.