DOCSEARCHbyAlgolia
Submit Docs

HAMi (Heterogeneous AI Computing Virtualization Middleware)

repository·master·Indexed 23 days ago

https://github.com/project-hami/hami

A Kubernetes-native solution for GPU virtualization and heterogeneous accelerator scheduling. HAMi enables sharing AI accelerators (GPUs, NPUs, etc.) across multiple workloads with resource isolation and device-aware scheduling without requiring application code changes. It supports a wide range of hardware including NVIDIA, Cambricon MLU, Hygon DCU, Metax GPU, Enflame GCU, Kunlunxin XPU, AWS Neuron, Huawei Ascend, and Iluvatar devices.

Tokens
17.8K
Snippets
24
Records
98
Agent score
88%

What's inside HAMi

  2. Overview of HAMi (Heterogeneous AI Computing Virtualization Middleware)

    master

    HAMi is a Kubernetes-native middleware designed for GPU virtualization and heterogeneous accelerator scheduling in AI infrastructure. It allows platform teams to share expensive GPUs and other AI accelerators across Kubernetes workloads by partitioning device memory and compute.

    Key capabilities include:

    • Device Sharing: Allocate portions of physical accelerators based on memory, cores, or device count.
    • Resource Isolation: Enforce memory and compute limits per workload (if supported by the backend).
    • Device-Aware Scheduling: Support for topology-aware, binpack, spread, and device-specific scheduling policies.
    • Heterogeneous Support: Manage multiple accelerator types (NVIDIA GPU, NPU, DCU, MLU, etc.) through a single workflow.
    • No Application Changes: Uses standard Kubernetes resource requests and limits.

  3. Use the HAMi vGPU Metrics Summarizer skill

    master

    The hami_vgpu_metrics_summarizer is an analysis skill designed to process HAMi vGPU metrics (from Prometheus-style /metrics output) to provide a structured summary of GPU resource usage.

    It can analyze:

    1. Raw HAMi Prometheus metrics pasted directly by the user.
    2. Metrics collected from a live Kubernetes cluster.

    The tool produces reports on:

    • Node and GPU usage status.
    • Per-GPU core and memory allocation.
    • Allocation-based utilization per GPU.
    • Sharing density (number of containers sharing each GPU).
    • Pod and namespace consumption.
    • Namespace-level totals for gpucores and gpumem.
    • Cluster capacity and resource distribution patterns (concentrated, fragmented, or even).

    Note: Unless explicit real-time GPU utilization metrics are provided, the tool treats metrics as allocation-based utilization, not actual SM/compute runtime utilization from nvidia-smi.

  4. Understand HAMi GPU Resource Accounting for Init Containers

    master

    HAMi implements a specific accounting logic for Pods that contain both init containers and app containers. Because Kubernetes runs init containers sequentially before app containers, they do not overlap in execution.

    To prevent over-provisioning or incorrect scheduling rejections, HAMi calculates the effective GPU footprint of a pod using the following formula for each resource dimension (GPU count, memory, cores, and per-device UUID):

    effective = max( sum(app container requests), max(single init container request) )

    This ensures that if an init container requests 20Gi and an app container requests 10Gi, HAMi accounts for 20Gi of usage rather than the sum (30Gi), reflecting the actual peak resource usage.

  5. Understand the HAMi Architecture

    master

    HAMi operates as a 'chart' architecture consisting of four primary components that manage the lifecycle of pod scheduling and device allocation:

    1. MutatingWebhook: Intercepts pod creation to validate resource requests. If HAMi recognizes the requested sharable devices, it sets the schedulerName to HAMi scheduler. Otherwise, it allows the task to pass to the default Kubernetes scheduler.
    2. Scheduler: An extender for kube-scheduler or volcano-scheduler. It implements Filter (to find nodes with available sharable devices) and Score (to rank those nodes) methods. The decision is communicated via pod annotations.
    3. DevicePlugin: A customized Device Plugin (DP) that runs on the node. Once the scheduler makes a decision, the DP generates the necessary environment variables and volume mounts based on the pod annotations.
    4. InContainer Control: Manages hard limits within the container. This is device-specific (e.g., HAMi-Core for NVIDIA devices, libvgpu-control.so for Iluvatar devices) and relies on specific environment variables passed during the scheduling process.

  6. How HAMi handles Init Container resource accounting

    master

    HAMi applies the effective resource formula across three main stages of the Pod lifecycle:

    1. Admission (Quota Check): The fitResourceQuota check uses the effectiveReq (the maximum of the peak init container request or the sum of app container requests) to validate against namespace quotas. Note that currently, GPU count is computed for consistency but is not used as an independent quota dimension; only memory and cores are checked against quotas.
    2. Scheduling (Fit & Scoring): The scheduler performs two passes. First, it fits init containers independently against a fresh copy of node state to find the peak per-device usage. Second, it fits app containers cumulatively. It then merges these results using the max() function per device UUID.
    3. Usage Recording: HAMi collapses raw per-container PodDevices into an accounting-only view using CollapseInitContainerUsage. This prevents the usage tracker from summing init and app container requests simultaneously.

  7. Diagnose ImagePullBackOff and ErrImagePull

    master

    If pods fail to start due to image issues, verify the image reference and pull errors.

    Common Error Patterns:

    • repository does not exist: Verify the image name and tag.
    • unauthorized: Fix imagePullSecrets to provide credentials.
    • manifest unknown: Verify the tag exists in the registry.
    • connection refused: Check network/DNS connectivity to the registry.
    • x509: certificate: Fix the CA trust chain.

    Verification Commands:

    • Check the image being used.
    • Inspect pod events for specific pull errors.
    • Verify if imagePullSecrets are configured for private registries.

  8. Troubleshoot Pending Pods (Scheduling Failures)

    master

    If pods are stuck in the Pending state, use kubectl describe pod to inspect the Events and Conditions sections.

    Common Causes and Remediation:

    • Insufficient cpu/memory: Resource exhaustion. Reduce requests, scale the cluster, or check resource quotas.
    • Taint/toleration mismatch: 0/N nodes are available + taint. Add a toleration to the pod or untaint the node.
    • Node selector mismatch: 0/N nodes are available + node selector. Fix the nodeSelector or add matching nodes.
    • Affinity rule unsatisfiable: 0/N nodes are available + affinity. Relax affinity rules.
    • Missing PVC: persistentvolumeclaim "X" not found. Create the PVC or fix the claim name.
    • Unbound PVCs: pod has unbound immediate PersistentVolumeClaims. Check StorageClass and PV availability.
    • Quota limit hit: exceeded quota. Check resource quotas.
    kubectl describe pod <pod-name> -n <namespace>

  9. Generate a HAMi vGPU Usage Summary Report

    master

    A comprehensive vGPU usage report should follow a structured template to provide both high-level cluster status and granular workload analysis. The recommended structure is:

    1. Executive Summary: A one-paragraph overview of the cluster state.
    2. Cluster / Node Overview: Node count, GPU count, and total core/memory allocation ratios.
    3. Per-GPU Summary: A table including Node, GPU Index, UUID, Core/Mem Allocated vs Limit, Core %, Mem %, Shared Containers, and Status.
    4. Per-Namespace Summary: Namespace usage of cores and memory (MiB/GiB), core/memory share %, and main pods.
    5. Pod / Workload Breakdown: Detailed mapping of Namespace, Pod, Container, Node, GPU UUID, Core, Memory, and Allocation Type.
    6. Real vs. Allocated Usage (nvidia-smi): Comparison of actual GPU utilization and memory used (via nvidia-smi) against the allocated reservation to identify over-allocated or idle pods.
    7. Capacity & Scheduling Assessment: Identification of idle, fully allocated, and fragmented GPUs, and whether new jobs (full or fractional) can fit.
    8. Key Findings & Caveats.

  10. Install HAMi via Helm

    master

    To install HAMi on your Kubernetes cluster, follow these steps:

    1. Label GPU nodes: HAMi requires nodes to be labeled so it can manage them.
    2. Add the Helm repository: Add the official hami-charts repository.
    3. Install the chart: Deploy HAMi into the kube-system namespace.
    4. Verify installation: Ensure both hami-device-plugin and hami-scheduler are in the Running state.

    Prerequisites for NVIDIA users:

    • NVIDIA driver >= 440
    • nvidia-docker version > 2.0
    • NVIDIA configured as the default runtime (containerd, Docker, or CRI-O)
    • Kubernetes >= 1.23
    • glibc >= 2.17
    • Linux kernel >= 3.10
    • Helm > 3.0

  11. Install HAMi using Helm

    master

    Follow these steps to install HAMi on your Kubernetes cluster.

    Prerequisites (for NVIDIA devices):

    • NVIDIA Driver >= 440
    • nvidia-docker > 2.0
    • NVIDIA configured as default runtime (containerd, Docker, or CRI-O)
    • Kubernetes >= 1.23
    • glibc >= 2.17
    • Linux Kernel >= 3.10
    • Helm > 3.0

    Installation Steps:

    1. Label your GPU nodes to allow HAMi management:
      kubectl label nodes <node-name> gpu=on
    2. Add and update the HAMi Helm repository:
      helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update
    3. Install HAMi into the kube-system namespace:
      helm install hami hami-charts/hami -n kube-system
    4. Verify that hami-device-plugin and hami-scheduler are running:
      kubectl get pods -n kube-system
    5. Deploy a sample workload to test:
      kubectl apply -f examples/nvidia/default_use.yaml
    kubectl label nodes <node-name> gpu=on
helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update
helm install hami hami-charts/hami -n kube-system
kubectl get pods -n kube-system
kubectl apply -f examples/nvidia/default_use.yaml

  12. Use Customized Device Plugins for HAMi

    master

    To ensure proper device virtualization and resource allocation, you must use the customized Device Plugin (DP) provided by HAMi for your specific hardware. Standard official Kubernetes Device Plugins are generally incompatible with HAMi and will cause unexpected behavior.

    Refer to the main README for installation instructions specific to your device type.