fastsafetensors

Performance

fastsafetensors is designed for high-throughput model loading. Key performance characteristics include:

• Standalone loading: Up to 4.8x-7.5x faster than the default safetensors deserializer.
• NVMe Throughput: Can reach up to 26.4 GB/s with GDS (GPUDirect Storage).
• vLLM Integration: Significant reductions in startup time (e.g., from ~12s to ~4.7s for Llama-2-13B on 4x L40S).
• AMD ROCm: Provides a nogds path that outperforms standard mmap by 2.6x to 4.7x.

Platform Support

The library supports various environments including:

• Linux/CUDA
• ROCm (AMD)
• Windows
• 3FS
• Unified-memory systems (e.g., DGX Spark)

Compatibility

• Python: 3.10 - 3.14
• PyTorch: 2.11.0

To prevent Out-of-Memory (OOM) errors, use these two parameters:

• max_batch_bytes: Caps the size of each sub-file chunk. It must be at least as large as the largest selected tensor, as tensors cannot be split across chunks.
• device_memory_budget: Bounds the total memory used by resident tensors and transient chunk buffers. In distributed broadcast, every rank must use the same value.

Error Handling: If the requested queue depth cannot fit within the budget, the loader automatically reduces the depth down to queue_size=-1. If even serial loading is impossible, it raises fastsafetensors.BudgetInfeasibleError (a subclass of ValueError).

Best Practices: When calculating a budget from free device memory, leave a reserve for allocator rounding and copier fixed pools. A recommended starting point is max(5% of free memory, 1 GiB).

The queue_size parameter controls how the loader manages buffers and concurrency:

Important Notes:

• Setting use_pipeline: false forces queue_size=-1 (serial mode with minimal GPU memory usage).
• Buffer counts exclude resident tensors, broadcast receive tensors, yield clones, and copier staging memory (which are handled by the memory planner).

To compare the performance of safetensors versus fastsafetensors when loading model weights on AMD ROCm platforms (e.g., MI300X), follow these steps:

1. Clear system cache to ensure a consistent cold-start baseline:

   sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'

2. Launch vLLM using the --load-format fastsafetensors flag. Ensure you set the necessary environment variables for ROCm and vLLM V1 support.

Note: The benchmark uses --tensor-parallel-size 8 for multi-GPU setups and specific compilation configs for optimal performance.

MODEL=EmbeddedLLM/deepseek-r1-FP8-Dynamic

VLLM_USE_V1=1 \
VLLM_ROCM_USE_AITER=1 \
vllm serve $MODEL \
--tensor-parallel-size 8 \
--disable-log-requests \
--compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
--trust-remote-code \
--load-format fastsafetensors \
--block-size 1

Direct storage-to-GPU loading is supported on ROCm via hipFile (requires ROCm >= 7.2).

• If libhipfile.so is available in the environment, the GDS code path is used transparently.
• On older ROCm versions without hipFile, the loader automatically falls back to the bounce-buffer (nogds) path.

You can run the project's test suite locally using the provided Makefile commands. The repository includes support for standard unit tests, parallel execution, and vLLM-specific tests.

make unittest
make unittest-parallel
make test-vllm

To ensure code formatting and linting comply with the CI workflow (which uses Python 3.13), set up pre-commit hooks. This automates checks for black, isort, flake8, mypy, and several file-format validators.

1. Install development dependencies:

pip install -e ".[dev]"

2. Install the hooks:

pre-commit install

To manually run all checks on all files:

pre-commit run --all-files

AutoLoader resolves configuration using the following priority (highest first):

1. Environment variable: Set FASTSAFETENSORS_CONFIG to the path of your .json config file.
2. Default path: A file named ./fastsafetensors.json in your current working directory.
3. Built-in defaults: The values defined in the LoaderConfig() dataclass.

All fields are optional. If a field is omitted from your configuration file, the loader falls back to the built-in default.

# Set via environment variable
export FASTSAFETENSORS_CONFIG=/path/to/config.json

# Or place a file at ./fastsafetensors.json

On Windows, fastsafetensors uses Microsoft DirectStorage to achieve high-throughput loading by bridging D3D12 resources with CUDA via external memory interop.

Setup

If the DirectStorage DLLs are not in the standard search path, you must provide the directory containing dstoragecore.dll and dstorage.dll using the following environment variable:

FASTSAFETENSORS_DSTORAGE_DLL_DIR (must be an absolute directory).

Fallback and CUDA Runtime

• If DirectStorage DLLs are missing, the loader falls back to the bounce-buffer (nogds) path.
• The loader automatically discovers the CUDA runtime (checking PyTorch's torch\lib first, then system locations).
• You can explicitly specify the CUDA runtime DLL path using: FASTSAFETENSORS_CUDART_LIB (absolute path, takes precedence).

This project requires all commits to comply with the Developer Certificate of Origin (DCO). Every commit must include a valid Signed-off-by line. To sign off a commit, use the -s flag with git commit.

git commit -s

To install the latest development version directly from the GitHub repository, use pip with the git URL.

pip install git+https://github.com/foundation-model-stack/fastsafetensors.git