DOCSEARCHbyAlgolia
Submit Docs

batchspawner

repository·main·Indexed 42 days ago

https://github.com/jupyterhub/batchspawner

A JupyterHub spawner for cluster environments using batch scheduling software such as SLURM, Torque, or LSF. It provides a unified abstraction layer via BatchSpawnerBase to manage user sessions by submitting jobs to resource managers including Torque, Moab, SLURM, SGE, HTCondor, and LSF.

Tokens
2K
Snippets
4
Records
9
Agent score
3%

What's inside batchspawner

  1. How BatchSpawnerBase works

    main

    The BatchSpawnerBase is an abstraction layer for batch job queueing systems. It provides a unified way to interact with various resource managers (Torque, Moab, SLURM, SGE, HTCondor, LSF).

    Key features include:

    • Template Variables: Configurable traits prefixed with req_ (e.g., req_foo) are exposed as {foo} in job template scripts.
    • Template Engines: Templates automatically use jinja2 if they contain {{ or {% syntax; otherwise, they use standard Python str.format().
    • Job Tracking: Uses a generic concept of job-IDs and ID-based state tracking rather than PIDs.
    • Customizable Commands: Allows configuring command templates for submitting, querying, and cancelling jobs.

  2. Provide multiple configurations using ProfilesSpawner

    main

    To allow users to choose between different resource allocations (e.g., different CPU/Memory profiles) or different spawners (e.g., Local vs. Cluster), use ProfilesSpawner from the wrapspawner package.

    In c.ProfilesSpawner.profiles, you provide a list of tuples with the following signature: List(Tuple( Unicode, Unicode, Type(Spawner), Dict ))

    1. Display Name: The name shown to the user in the dropdown.
    2. Unique Key: An internal identifier for the profile.
    3. Spawner Class: The class of the spawner to use.
    4. Config Dictionary: A dictionary of configuration options specific to that profile.
    import batchspawner
# Note: ProfilesSpawner is in the wrapspawner package
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'

c.ProfilesSpawner.profiles = [
   ( "Local server", 'local', 'jupyterhub.spawner.LocalProcessSpawner', {'ip':'0.0.0.0'} ),
   ('Mesabi - 2 cores, 4 GB, 8 hours', 'mesabi2c4g12h', 'batchspawner.TorqueSpawner', 
      dict(req_nprocs='2', req_queue='mesabi', req_runtime='8:00:00', req_memory='4gb')),
   ('Mesabi - 12 cores, 128 GB, 4 hours', 'mesabi128gb', 'batchspawner.TorqueSpawner', 
      dict(req_nprocs='12', req_queue='ram256g', req_runtime='4:00:00', req_memory='125gb')),
]

  3. Install batchspawner

    main

    To install batchspawner, you can install it directly from PyPI or via an editable install from the repository root.

    After installation, you must register the batchspawner interface in your jupyterhub_config.py by importing it, even if you don't call it directly. This is required for the spawner to function correctly.

    # Option 1: Standard installation
pip install batchspawner

# Option 2: Editable installation from repo root
pip install -e .
    # In jupyterhub_config.py
c = get_config()
c.JupyterHub.spawner_class = 'batchspawner.TorqueSpawner'
import batchspawner  # Required to register the interface

  4. Configure SlurmSpawner environment and termination

    main

    The SlurmSpawner uses srun by default to wrap the spawner command, which enforces the environment.

    • To use the user's existing environment: Set req_srun='' in your configuration. Note that this may allow shell aliases or startup scripts to interfere with the command.
    • Termination: Use of srun is required to ensure the singleuser server terminates gracefully.
    c.SlurmSpawner.req_srun = ''

  5. Configure BatchSpawnerBase parameters

    main

    Parameters defined under BatchSpawnerBase are used to populate the job script templates. These typically represent resource requirements or environment settings. Common parameters include:

    • req_nprocs: Number of processes.
    • req_queue: The name of the batch queue.
    • req_host: The host/address of the resource manager.
    • req_runtime: The requested runtime (e.g., '12:00:00').
    • req_memory: The requested memory (e.g., '4gb').

  6. Troubleshoot batchspawner issues

    main

    If jobs are failing to start or Jupyter is not appearing, follow this debugging workflow:

    1. Check JupyterHub Logs: Look for errors in the main Hub log. Specifically, verify the batch script that was submitted and the command used to submit it.
    2. Check the Batch System: Verify if the job was actually scheduled.
      • If the job never starts: Check for bad scheduler options.
      • If the job stays in queue too long: It may exceed start_timeout, causing JupyterHub to kill it.
    3. Check Job Output: If the job starts but fails, check the scheduler's stdout/stderr files. You can add env or set -x to your batch script for more verbosity.
    4. Check Jupyter Logs: If the job runs but Jupyter fails, check the error messages in the batch script output. Ensure Jupyter is starting with the correct options and can communicate with the Hub.
    5. NFS Warning: If running on an NFS filesystem, Jupyter may experience issues with fcntl() calls. Check the JupyterHub documentation regarding SQLite on NFS.

  7. Verify SIGTERM support in batch spawners

    main

    To ensure a spawner handles graceful termination correctly, it must send SIGTERM to the jupyterhub-singleuser process before sending SIGKILL.

    You can verify this by adding echo "terminated gracefully" to the end of your batch script template. If you see this message in your singleuser server logs during a stop request, the spawner is correctly receiving SIGTERM and terminating gracefully.

  8. Example: Configure TorqueSpawner

    main

    This example demonstrates how to configure a TorqueSpawner. Note that when using batch systems, it is often necessary to increase c.Spawner.http_timeout because batch jobs may take significant time to transition from 'submitted' to 'running'.

    import batchspawner

# Select the Torque backend and increase the timeout
c.JupyterHub.spawner_class = 'batchspawner.TorqueSpawner'
c.Spawner.http_timeout = 120

# BatchSpawnerBase configuration (used in the template below)
c.BatchSpawnerBase.req_nprocs = '2'
c.BatchSpawnerBase.req_queue = 'mesabi'
c.BatchSpawnerBase.req_host = 'mesabi.xyz.edu'
c.BatchSpawnerBase.req_runtime = '12:00:00'
c.BatchSpawnerBase.req_memory = '4gb'

# TorqueSpawner specific configuration
c.TorqueSpawner.batch_script = '''#!/bin/sh
#PBS -q {queue}@{host}
#PBS -l walltime={runtime}
#PBS -l nodes=1:ppn={nprocs}
#PBS -l mem={memory}
#PBS -N jupyterhub-singleuser
#PBS -v {keepvars}
module load python3
{cmd}
'''

# Munge the execution hostname returned by qstat if necessary
c.TorqueSpawner.state_exechost_exp = r'int-\1.mesabi.xyz.edu'