repository·main·Indexed 22 days ago
https://github.com/chelsea0x3b/cudarcA minimal and safe Rust API wrapper over the CUDA toolkit, providing safe abstractions for the CUDA driver, NVRTC, cuBLAS, cuDNN, and other NVIDIA libraries. It features a three-tier API design (sys, result, and safe), a Bindings Generator for automating CUDA bindings, and support for various linking behaviors and CUDA, CUDNN, and NCCL versions via Cargo features.
Each wrapper in cudarc (e.g., driver, nvrtc, cublas) is organized into three distinct layers. It is heavily recommended to use the safe APIs for most use cases.
sys: The raw FFI APIs generated via bindgen. These are low-level and unsafe.result: A thin wrapper around sys that converts return codes into Rust Result types.safe: High-level, ergonomic, and safe abstractions built on top of result and sys.You can specify which CUDA version cudarc builds against using Cargo features.
-F cuda-<major>0<minor>0 to target a specific version (e.g., -F cuda-12010).-F cuda-version-from-build-system to automatically detect the CUDA toolkit version via nvcc at build time.-F fallback-latest to prevent the build script from panicking if detection fails; this will cause the crate to use the highest available bindings instead.cargo build --features cuda-12010You can control how cudarc links to CUDA libraries using Cargo features:
-F dynamic-loading: (Default) Does not require libraries to be present at build time.-F dynamic-linking: Links to libraries dynamically.-F static-linking: Links to libraries statically.Specific versions for CUDNN and NCCL can be selected using Cargo features:
CUDNN versions:
cudnn-08970 (8.9.7)cudnn-09102 (9.10.2)cudnn-09211 (9.21.1)NCCL versions:
-F nccl-<version> for versions in the range 2.22 to 2.30 (e.g., -F nccl-02220).The Bindings Generator is a Rust binary designed to automate the creation of CUDA bindings. It performs the following steps:
To run the generator, use cargo run with the --release flag to ensure optimal performance during the download and generation process.
cargo run --releaseThe safe API manages multi-stream synchronization automatically when using CudaSlice, CudaView, or CudaViewMut.
Each of these types contains internal CudaEvents that record when data is read from or written to. When you pass these types as arguments via .arg() in a LaunchArgs builder, cudarc automatically:
wait events to the current stream to ensure previous operations on that data are complete.record events to the current stream to ensure the kernel's operations are tracked for future dependencies.This prevents race conditions where multiple kernels on different streams might attempt to access the same memory simultaneously.
Result<(), CurandError>. The CurandError struct wraps the underlying sys::curandStatus_t from the CUDA toolkit. If an operation fails, the error variant will contain the specific status code returned by cuRAND.cuTENSOR uses a two-step process for execution: first, you define a plan preference, and then you create an execution plan based on that preference and the required workspace size.
create_plan_preference to specify algorithm preferences and JIT modes.estimate_workspace_size to determine how much memory is needed for the operation.create_plan to generate the actual plan used for execution.Safety Requirements:
plan must not have been freed already when calling destroy_plan.// 1. Create preference
let pref = unsafe {
create_plan_preference(handle, algo, jit_mode).expect("Failed to create preference")
};
// 2. Estimate workspace
let workspace_size = unsafe {
estimate_workspace_size(handle, op_desc, pref, workspace_pref).expect("Failed to estimate workspace")
};
// 3. Create plan
let plan = unsafe {
create_plan(handle, op_desc, pref, workspace_size).expect("Failed to create plan")
};
// ... execute ...
// Cleanup
unsafe {
destroy_plan(plan).expect("Failed to destroy plan");
destroy_plan_preference(pref).expect("Failed to destroy preference");
}The driver API is the foundation of cudarc. It provides safe abstractions for managing GPU resources and execution. Use the following mental model to map CPU concepts to CUDA:
| Concept | CPU | Cuda |
|---|---|---|
| Memory allocator | std::alloc::GlobalAlloc | driver::CudaContext |
| List of values on heap | Vec<T> | driver::CudaSlice<T> |
| Slice | &[T] | driver::CudaView<T> |
| Mutable Slice | &mut [T] | driver::CudaViewMut<T> |
| Function | Fn | driver::CudaFunction |
| Calling a function | my_function(a, b, c) | driver::LaunchArgs::launch() |
| Thread | std::thread::Thread | driver::CudaStream |
Key structs:
driver::CudaContext: A handle to a specific device ordinal (e.g., 0, 1, 2, ...).driver::CudaStream: Used to submit work to a device.driver::CudaSlice<T>: Represents a Vec<T> on the device, which can be allocated via a CudaStream.cuBLAS operations return a Result<(), CublasError>. The CublasError struct wraps the underlying sys::cublasStatus_t error code.
If a function returns Err(CublasError(status)), the status contains the specific cuBLAS error code which can be used for troubleshooting.
CUPTI allows you to subscribe to events using callbacks. Use the following functions to manage these subscriptions:
subscribe: Initializes a callback subscriber with a callback function and user data.unsubscribe: Unregisters an existing callback subscriber.enable_all_domains: Enables or disables all callbacks across all domains.enable_domain: Enables or disables callbacks for a specific domain.enable_callback: Enables or disables callbacks for a specific domain and callback ID.Note: Most of these functions are unsafe and require that the subscriber or callback handles provided are valid.
// Example: Subscribing to callbacks
// unsafe {
// subscribe(
// &mut subscriber_handle,
// my_callback_func,
// user_data_ptr
// )?;
// }Most modules in cudarc are organized into three distinct layers of abstraction to balance safety and flexibility:
safe module: Provides high-level, safe Rust abstractions. This is the recommended layer for most users.result module: A thin wrapper around the sys module that ensures all functions return a Result type.sys module: Contains the raw FFI (Foreign Function Interface) bindings to the underlying CUDA C libraries.While the result and sys levels are often interchangeable, the safe APIs are preferred to ensure memory safety and proper resource management. If a feature is missing from the safe API, you may need to drop down to the result level.