DOCSEARCHbyAlgolia
Submit Docs

Clusterscope Documentation

website·Indexed 30 days ago

https://facebookresearch.github.io/clusterscope/

A CLI and Python library designed to extract cluster information and enable cluster-agnostic design. It provides tools to detect cluster capabilities, identify Slurm versions, and manage GPU/CPU resources, including utilities like cscope-job-gen for generating GPU and CPU jobs and functions to retrieve memory, CPU counts, and temporary directories.

Tokens
4.6K
Snippets
36
Records
40
Agent score
75%

What's inside Clusterscope

  1. Overview of Clusterscope Cluster-Agnostic Design

    Clusterscope provides a cluster-agnostic design that allows users to 'write once, run anywhere' by automatically detecting cluster capabilities. This ensures high portability across heterogeneous computing environments, removing the need for users to manually handle specific HPC cluster operations or write environment-specific job submissions.

  2. Inspect available GPUs using cscope gpus

    The cscope gpus command displays information about available GPUs. When executed on a Slurm cluster, it provides details for every partition; otherwise, it defaults to the local node. By default, it outputs the GPU generation, count, vendor, and the associated Slurm partition.
    $ cscope gpus
GPU Gen, GPU Count, GPU Vendor, Slurm Partition:
h100, 8, nvidia, h100
h200, 8, nvidia, h200

  3. Get local node GPU generation and count

    Use the clusterscope.local_node_gpu_generation_and_count() function to retrieve information about the GPUs on the local node. This function returns either a single GPUInfo instance or a list of GPUInfo instances containing details such as the GPU generation, the number of GPUs, the vendor, and the partition.
    import clusterscope
gpu_info = clusterscope.local_node_gpu_generation_and_count()
print(gpu_info)
# Example Output: GPUInfo(gpu_gen='H100', gpu_count=8, vendor='nvidia', partition=None)

  4. Retrieve system memory information using clusterscope.mem()

    The clusterscope.mem() function provides information about available system memory. Its behavior depends on the environment:

    • Slurm Cluster: Returns a list of MemInfo objects, one for each available partition.
    • Local Node: Returns a single MemInfo instance for the local node (where partition is None).

    The MemInfo object contains the following attributes:

    • mem_total_MB: Total memory in Megabytes.
    • mem_total_GB: Total memory in Gigabytes.
    • partition: The name of the Slurm partition, or None if not applicable.
    import clusterscope

# Get memory info for all partitions (if on Slurm) or local node
mem_info = clusterscope.mem()
print(mem_info)

  5. Retrieve available CPU core information using cpus()

    The clusterscope.cpus() function returns information about the number of CPU cores available in the system.

    • On a Slurm Cluster: It returns a list of CPUInfo objects, one for each available partition.
    • On a Local Node (Non-Slurm): It returns a single CPUInfo instance where the partition attribute is None.

    Note: To find CPUs specifically assigned to a current job rather than total system availability, use get_job().

    import clusterscope

# Get all available CPU info
cpu_info = clusterscope.cpus()
print(cpu_info)

# Example Slurm output: [CPUInfo(cpu_count=192, partition='cpu'), ...]
# Example Local output: CPUInfo(cpu_count=16, partition=None)

  6. Configure pre-commit hooks for ClusterScope

    ClusterScope uses pre-commit to integrate linters, formatters, and typecheckers. These checks are also enforced via GitHub CI. To automate these checks locally before committing code, install the pre-commit hooks.
    pre-commit install

  7. Update ClusterScope project dependencies

    When adding new dependencies, first add them to the appropriate section in the pyproject.toml file. Then, regenerate the requirements files using the provided make targets.
    # Update the standard requirements file
make requirements.txt

# Update the development requirements file
make dev-requirements.txt

  8. Generate Slurm job requirements using cscope job-gen

    The cscope job-gen task slurm command calculates the proportionate amount of CPUs and Memory to allocate for a Slurm job based on the available resources in a specified partition. Users only need to specify either the number of GPUs or CPUs per task; the tool automatically derives the corresponding memory and CPU requirements.
    # Example: Request 4 GPUs in the h100 partition with slurm_cli output
$ cscope job-gen task slurm --partition=h100 --gpus-per-task=4 --format=slurm_cli

  11. Retrieve current job metadata using get_job

    Use clusterscope.get_job() to obtain a job object that provides methods for querying the current execution environment, such as CPU/GPU counts, job IDs, and distributed training ranks.
    import clusterscope
job = clusterscope.get_job()

  12. Set up the ClusterScope development environment

    To set up the development environment, install the required dependencies using uv and install ClusterScope in editable mode from the root of the repository. Once installed, you can activate the environment to access the clusterscope library and the cscope CLI tool.
    # Install development requirements and clusterscope in editable mode
make install-dev-requirements

# Activate the environment
source <(make activate)