Slurm Operator

When using scalingMode=StatefulSet, NodeSet pods are loosely mapped to Kubernetes nodes and can be rescheduled. To ensure a pod always returns to the same physical node after eviction, deletion, or maintenance, you can enable Node Pinning.

How it works:

• The pod is initially scheduled normally.
• The controller records the mapping in status.nodeToOrdinal.
• On recreation, the controller adds a Kubernetes node affinity to the pod to force it onto the recorded node.
• If the node is unavailable, the pod stays in Pending until the node returns.

When pinning is removed:

• The Kubernetes node no longer exists.
• The new NodeSet pod template no longer matches the recorded node (e.g., changes to affinity or nodeSelector).

# Example: Enabling node pinning in a NodeSet CR
apiVersion: slinky.slurm.net/v1beta1
kind: NodeSet
metadata:
  name: gpu-workers
spec:
  pinToNode: true
  replicas: 4

Slurm requires a fully connected network with bidirectional communication between all daemons and clients. Because Kubernetes NAT-type networks can impede this, choose one of two networking strategies:

Host Network

Avoids Kubernetes NAT by having Slurm pods use the Kubernetes node host network directly.

• Pros: Simplest methodology.
• Cons: Security implications; only one pod with hostNetwork: true can run on a Kubernetes node at a time; inherits node hostname and namespace.
• Note: Controller and Accounting do not support this option due to Slurm configuration race conditions with Kubernetes.

To use this, configure the pod as follows:

hostNetwork: true
dNSPolicy: ClusterFirstWithHostNet

Network Peering

Configures network peering so internal and external services can communicate directly.

• Pros: Higher security; minimal Slurm configuration required.
• Cons: Most complex methodology.
• Implementation: Typically involves an advanced CNI (like Calico) with network peering (BGP) for bidirectional communication across Kubernetes boundaries. No special configuration is required for the Slurm helm chart itself.

For NVIDIA GB200/GB300 NVL72 systems, Slurm-operator supports IMEX channels via the NVIDIA GPU DRA driver.

Note: Slurm-operator relies on DRA to manage IMEX channels, so do not use custom prolog/epilog scripts for IMEX management as they may interfere with the DRA driver. Per-job IMEX configuration is not supported; instead, use a ComputeDomain to inject channels into the NodeSet.

# 1. Define a ComputeDomain to inject channels
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: slurm-compute-domain
  namespace: slurm
spec:
  channel:
    allocationMode: All
    resourceClaimTemplate:
      name: slurm-test-compute-domain
  numNodes: 0

# 2. Configure Controller for IMEX
controller:
  extraConfMap:
    SwitchType: "switch/nvidia_imex"
    SwitchParameters: ""

# 3. Claim the channel in the NodeSet
nodesets:
  slinky:
    slurmd:
      resources:
        claims:
          - name: compute-domain-channel

The Slurm Client Controller is an internal component of the operator responsible for managing Slurm Clients, which are used by other internal controllers to interact with Slurm.

Lifecycle and Workflow

1. Creation and Initialization

• When a user creates a Controller Custom Resource (CR) via the Kubernetes API, the Slurm Client Controller watches the CRD.
• The controller creates a new Slurm Client instance.
• The Slurm Client instance performs an initial connection to the Slurm REST API to retrieve Slurm nodes and add events to the Slurm Event Channel for cache delta tracking.
• The controller adds the initialized Slurm Client to an internal Slurm Client Map.
• The controller pings the Slurm Control Plane via the Slurm Client to verify connectivity.
• Finally, the controller updates the status of the Controller CR in the Kubernetes API.

2. Deletion and Cleanup

• When a user deletes the Controller CR, the Slurm Client Controller watches the deletion event.
• The controller looks up the corresponding Slurm Client in the Slurm Client Map and removes it.
• The Slurm Client instance is then garbage collected (GC) and destroyed.