TensorFusion GPU Virtualization and Pooling

Node splitting and isolation capabilities are configured via nodeManagerConfig within a GPUPool.

Rules are matched sequentially; the first rule that matches a node's selector is applied. Nodes that do not match any rule fall back to the defaultIsolationMode.

Supported modes:

• soft: Soft isolation.
• hard: Hard isolation.
• partitioned: Node partitioning.
• shared: This is a workload allocation strategy (available since v2.13.0), not a node capability mode. Shared workloads can run on nodes using any of the three modes above.

When modifying rules, TensorFusion uses a rolling update process for the Hypervisor. If autoUpdateHypervisor=false, configuration changes are marked as unsynced. If enabled, updates are performed in batches according to batchPercentage and batchInterval.

nodeManagerConfig:
  defaultIsolationMode: soft
  isolationModeRules:
    - mode: partitioned
      selector:
        matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values: [p4d.24xlarge, p5.48xlarge]
    - mode: hard
      selector:
        matchLabels:
          tensor-fusion.ai/gpu-model: H100

The project uses a peer graph + tier model to represent GPU connectivity, moving beyond simple NUMA-only abstractions. This allows compatibility with various hardware vendors.

Core Components:

• GPUTopology: Represents a single GPU, its NUMANode, and its Peers.
• GPUPeerLink: Describes the connection to a peer GPU, including the Tier (distance), LinkType (e.g., NVLink, PCIe), and Bandwidth.

Standardized Affinity Tiers:

• TierSameInterconnect (0): GPUs are on the same high-speed interconnect.
• TierSameNUMA (1): GPUs are within the same NUMA node.
• TierCrossNUMA (2): GPUs are across different NUMA nodes.
• TierUnknown (3): Topology information is unavailable.

Fallback Logic (Auto Mode):

1. Use vendor interconnect tier if available.
2. Fallback to NUMA if vendor tier is missing.
3. Fallback to TierUnknown if NUMA info is missing.

type GPUTopology struct {
    GPUUUID  string        `json:"gpuUUID"`
    NUMANode *int32        `json:"numaNode,omitempty"`
    Peers    []GPUPeerLink `json:"peers,omitempty"`
}

type GPUPeerLink struct {
    PeerGPUUUID string `json:"peerGPUUUID"`
    Tier        int32  `json:"tier"`
    LinkType    string `json:"linkType,omitempty"` // e.g., NVLink, PCIe
    Bandwidth   int64  `json:"bandwidth,omitempty"`
}

type GPUAffinityTier int
const (
    TierSameInterconnect GPUAffinityTier = 0
    TierSameNUMA         GPUAffinityTier = 1
    TierCrossNUMA        GPUAffinityTier = 2
    TierUnknown          GPUAffinityTier = 3
)

The scheduler applies different logic depending on the number of GPUs requested by a Pod:

Multi-GPU Requests (N > 1)

• Goal: Maximize interconnect performance.
• Priority:
  1. Select GPUs within the same NUMA node.
  2. If no same-NUMA combination exists:
     • hard mode: Filter the node.
     • soft mode: Allow the node but reduce the score.
• Scoring: Based on the smallest Tier, highest topology score, and lowest fragmentation.

Single-GPU Requests (N = 1)

• Goal: Minimize 'damage' to high-bandwidth clusters (Least Damage principle).
• Priority:
  1. Select 'isolated' GPUs (those not part of a high-speed interconnect cluster).
  2. Select GPUs in NUMA domains that are already fragmented (cannot form multi-GPU groups).
  3. Select GPUs that, if removed, leave the remaining GPUs in a state that can still form valid multi-GPU groups.
  4. Avoid consuming members of high-quality, complete topology clusters.

TensorFusion uses a soft isolation mechanism that loads the vendor limiter via /etc/ld.so.preload. This allows an application to set its own LD_PRELOAD (e.g., for jemalloc) without removing the TensorFusion limiter.

The Injection Process:

1. A middleware init container copies the limiter.
2. A per-application init container runs within the application's image.
3. This init container reads the original /etc/ld.so.preload (including explicitly mounted configurations) and appends the vendor limiter path.
4. The application receives this merged file as a read-only subPath mount.

Key Characteristics:

• Isolation: Each container gets its own generated file; the original configuration volume remains unmodified.
• Lifecycle: The preload file is a startup snapshot. If the underlying ConfigMap changes, existing Pods must be recreated to receive the new injection.
• Compatibility: Requires a glibc-based application image containing /bin/sh, cat, and chmod.

In v2, you must plan the isolation strategy for each GPUPool within the TensorFusionCluster (TFC). Do not modify individual GPUPool objects directly; instead, update the spec.gpuPools[].specTemplate.nodeManagerConfig in the TFC.

Isolation Modes

• soft: The default mode if no rules match. Recommended for most nodes.
• hard: Strict isolation.
• partitioned: Used for nodes requiring hardware partitioning.

Configuration Rules

1. defaultIsolationMode: Defines the mode for nodes that do not match any isolationModeRules.
2. isolationModeRules: Rules are matched in declaration order. The first match wins.
3. Selectors: Use stable Kubernetes Node labels. Ensure selectors do not overlap in ways that violate your intended priority.
4. Compatibility: Ensure workload isolation requests match the target node capabilities (soft, hard, or partitioned).

Example TFC Configuration Snippet:

spec:
  gpuPools:
    - name: nvidia
      specTemplate:
        nodeManagerConfig:
          defaultIsolationMode: soft
          isolationModeRules:
            - mode: partitioned
              selector:
                matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values: [p4d.24xlarge, p5.48xlarge]
            - mode: hard
              selector:
                matchLabels:
                  tensor-fusion.ai/gpu-model: H100

The implementation must adhere to several strict invariants to ensure stability:

Core Invariants:

1. All committed and assumed allocations on a Dynamic GPU must have the same isolation type.
2. If activeIsolationMode is non-empty, there must be at least one corresponding committed or assumed allocation.
3. When a GPU has no allocations, activeIsolationMode must converge to empty.
4. A shared GPU can only have one allocation and must be completely idle and unpartitioned before allocation.
5. Dynamic GPUs cannot contain partitioned allocations.

Handling Conflicts: If a GPU is found to have multiple conflicting modes during recovery or an abnormal write:

• Do not automatically delete any Pods.
• Mark the GPU Phase/Condition as conflict or unschedulable.
• Exclude the GPU from all new scheduling candidates.
• Emit Kubernetes Events and metrics listing the GPU, Pod UID, and the conflicting modes.
• Recovery is handled automatically by the reconcile loop once an administrator clears the conflicting Pods.

The gputopo (GPU Network Topology Aware) scheduling mechanism is designed as an independent scheduler plugin in TensorFusion. Its primary purpose is to provide intra-node GPU topology awareness for multi-GPU requests without interfering with the existing gpufit (GPU Resources Fit) logic.

Core Responsibilities

• GPUResourcesFit: Responsible for answering "On which nodes are there allocatable GPUs?"
• GPUNetworkTopologyAware: Responsible for answering "Among these allocatable GPUs, which group has the optimal topology?"

Design Principles

• Separation of Concerns: gputopo performs combination selection (choosing the best group) but does not handle actual resource allocation. Final allocation, cache updates, and annotation patching remain the responsibility of GPUResourcesFit.
• Zero Side Effects: When the plugin is disabled, it must have zero impact on existing scheduling paths, performance, or behavior.
• Unified Abstraction: The scheduler consumes normalized topology metadata (tiers/domains) rather than vendor-specific raw formats (like NVLink or HCCS).
• Data/Control Plane Separation: The hypervisor/provider discovers hardware topology, while the scheduler plugin only consumes the normalized metadata.