OpenHPC Documentation

An OpenHPC recipe is a combination of two files located in recipes/ that define a specific system configuration:

1. *.conf: An ordered list of YAML configuration files from the config/ directory that are merged together.
2. *.yaml (Optional): Per-recipe overrides. These are primarily used for Confluent recipes to provide values that cannot be derived from the standard configuration hierarchy (e.g., distro_id or distro_iso_image).

The build process uses yq to perform a deep merge of these files into a single build/*.yaml file, which then serves as the input for documentation generation via mkdoc.py.

# recipes/rocky10-x86_64-warewulf-slurm.conf
config/base.yaml
config/distro/el10.yaml
config/distro/rocky.yaml
config/arch/x86_64.yaml
config/provisioner/warewulf.yaml
config/scheduler/slurm.yaml

# recipes/rocky10-x86_64-confluent-slurm.yaml
distro_id: "rocky-10.1-x86_64-default"
distro_iso_image: "Rocky-10.1-x86_64-dvd1.iso"

Since proxy and mirror configurations are highly site-specific, OpenHPC uses static placeholders that are intended to be replaced by a pre-processing script before recipe.sh is executed. These placeholders appear as valid shell comments in the generated script.

Placeholder Types:

• #<<< ohpc_proxy:head >>>#: Used for head node setup (CA certs, dnf.conf, etc.).
• #<<< ohpc_proxy:compute >>>#: Used for compute image or node setup.
• #<<< ohpc_proxy:image >>>#: Used for OpenCHAMI image-builder configuration.

Implementation Pattern: To add a new static placeholder, use an ohpc_command marker inside an ohpc_begin block without any ohpc_if wrappers to ensure it is always present for the pre-processor to find.

<!-- ohpc_begin -->
<!-- ohpc_comment Configure site authentication (site-specific) -->
<!-- ohpc_command #<<< ohpc_auth:head >>># -->
<!-- ohpc_end -->

In environments without IPMI (like VMs), compute nodes cannot be reset using ipmitool. Instead, OpenHPC uses runtime-emitted placeholders that the script prints to stdout during execution. An external monitoring process must wrap the script execution to intercept these lines and trigger a hypervisor-level reset.

Placeholder Format: When the has_ipmi flag is 0, the script emits lines in the following format: #<<< ohpc_reset:index,node_name,bmc_address >>>#

Example Output:

#<<< ohpc_reset:0,c1,192.168.1.101 >>>#
#<<< ohpc_reset:1,c2,192.168.1.102 >>>#

Implementation Example (Monitoring Wrapper):

#!/bin/bash
# Run recipe.sh and intercept ohpc_reset lines to trigger VM resets
bash recipe.sh 2>&1 | while IFS= read -r line; do
  printf '%s\n' "$line"
  pat='^#<<<[[:space:]]ohpc_reset:([0-9]+),([^,]+),(.+)[[:space:]]>>>#$'
  if [[ "$line" =~ $pat ]]; then
    idx="${BASH_REMATCH[1]}"
    node_name="${BASH_REMATCH[2]}"
    bmc_addr="${BASH_REMATCH[3]}"
    echo ">>> Triggering reset for ${node_name} [${idx}] (${bmc_addr})" >&2
    openstack server reboot --hard "${node_name}"
  fi
done

This environment provides a single-user cluster for learning and testing Slurm on OpenHPC 4.x with Rocky10. The cluster consists of a head node (openhpc-head), a login node (openhpc-login), and 8 compute nodes (openhpc-node-[0-7]) using shared Docker networking and storage.

Cluster Architecture:

• Shared Storage: /project (volume openhpc-container-project) and /scratch (volume openhpc-container-scratch).
• Non-shared Storage: The /home directory is not shared across containers.

Setup and Execution:

1. Container Engine: If using podman instead of Docker, set the CONTAINER environment variable.
2. Start Cluster: Run ./run.sh. This creates the network, shared storage, and starts the 8 nodes. It will automatically connect you to the login node. Exiting this shell will shut down the cluster.
3. Accessing Nodes:
   • To connect to the login node from a new terminal: ./ssh.sh.
   • To login to the head node as root: USER=root ./ssh.sh (Note: root access is temporary and lost when ./run.sh is exited).
4. Cleanup: Run ./delete.sh to remove the cluster network, storage, and container images.

# Use podman instead of docker
export CONTAINER=podman

# Build and run the cluster
./run.sh

# Connect to the login node from another terminal
./ssh.sh

# Clean up everything when finished
./delete.sh