nvidia-cuda-checkpoint

A process's CUDA state is initially running.

Suspension (Running $\rightarrow$ Suspended): When cuda-checkpoint suspends CUDA:

1. Lock: Any CUDA driver APIs that launch work, manage resources, or impact GPU state are locked.
2. Complete: Already-submitted CUDA work (including stream callbacks) is completed.
3. Copy-out: Device memory is copied to the host into allocations managed by the CUDA driver.
4. Release: All of CUDA's GPU resources are released.

Note: CPU threads are not suspended and can continue to interact with CUDA by calling APIs (which may block) or accessing host memory (e.g., cudaMallocHost).

Resumption (Suspended $\rightarrow$ Running): When cuda-checkpoint resumes CUDA:

1. Acquire: GPUs are re-acquired by the process.
2. Copy-in: Device memory is copied back to the GPU, and memory mappings are restored to original addresses.
3. Restore: CUDA objects like streams and contexts are restored.
4. Unlock: CUDA driver APIs are unlocked.

Once resumed, CUDA calls unblock and execution continues on the GPU.

The cuda-checkpoint utility (located in the bin directory) allows for transparently checkpointing and restoring CUDA state within a running Linux process. It can be used alone to suspend/resume CUDA state or combined with CRIU to fully checkpoint CUDA applications by ensuring the GPU resources are released before CRIU captures the process state.

Key Workflow:

1. Suspend: Use cuda-checkpoint --toggle --pid <pid> to move a process from a running CUDA state to a suspended state. This locks driver APIs, completes submitted work, copies device memory to the host, and releases GPU resources.
2. CPU Checkpoint: Use a tool like criu to checkpoint the process tree while the CUDA state is suspended.
3. Restore: Use criu to restore the process.
4. Resume: Use cuda-checkpoint --toggle --pid <pid> to move the process back to a running CUDA state. This re-acquires GPUs, copies device memory back to the GPU, and restores CUDA objects (streams, contexts).

# Suspend CUDA state
cuda-checkpoint --toggle --pid $PID

# Resume CUDA state
cuda-checkpoint --toggle --pid $PID

For display driver version 610 and higher, cuda-checkpoint supports cuIpcGetMemHandle-based CUDA IPC. This can be enabled in two ways:

1. Direct Launch: Launch the application directly using the --launch-job option.
2. Job File: Copy the file created by cuda-checkpoint and point the CUDA_CHECKPOINT_JOB_FILE environment variable to that copy.

Important Notes:

• Job files should never be reused.
• When using the second method, job files should only be used in the environment where they were created.
• While --launch-job is available in earlier versions, it is recommended to use it only with driver version 610 or higher.
• cuda-checkpoint must be invoked on the processes in a job sequentially.

# Method 1: Direct launch (Recommended for driver 610+)
cuda-checkpoint --launch-job ./your_application

# Method 2: Using a job file
export CUDA_CHECKPOINT_JOB_FILE=/path/to/copied/job_file
./your_application

The cuda-checkpoint utility provides the following command-line interface for managing CUDA process states. It supports display driver version 550 and higher.

Operations:
--get-state --pid <pid>
        Prints the current checkpoint state of the process specified by <pid>

--action lock | checkpoint | restore | unlock --pid <pid> [--timeout <ms>]
        Performs the specified action on <pid>. For the lock action a timeout can be provided, the lock operation will wait up to <ms> milliseconds for the operation to succeed.

--toggle --pid <pid>
        Toggles the CUDA state in the specified process between the running and checkpointed states

--get-restore-tid --pid <pid>
        Retrieves the CUDA restore thread ID of the process specified by <pid>

Options:
--pid|-p <pid>
        The pid upon which to perform the operation

--timeout|-t <timeout>
        Optional timeout that can be specified for the lock action in milliseconds

--help|-h
        Print this help message