DOCSEARCHbyAlgolia
Submit Docs

bitsandbytes

repository·main·Indexed 27 days ago

https://github.com/bitsandbytes-foundation/bitsandbytes

A PyTorch library providing k-bit quantization and matrix multiplication routines to reduce memory consumption during LLM inference and training. Key features include 8-bit Optimizers via bitsandbytes.optim, LLM.int8() 8-bit quantization via bitsandbytes.nn.Linear8bitLt, and QLoRA 4-bit quantization via bitsandbytes.nn.Linear4bit. It supports multiple accelerators including NVIDIA GPU (cuda), AMD GPU (cuda), Intel GPU (xpu), Intel Gaudi (hpu), and Apple Silicon (mps).

Tokens
17K
Snippets
32
Records
140
Agent score
93%

What's inside bitsandbytes

  1. Overview of bitsandbytes quantization features

    main

    bitsandbytes provides k-bit quantization for PyTorch to reduce memory consumption for large language model (LLM) inference and training. It offers three primary features:

    • 8-bit optimizers: Uses block-wise quantization to maintain 32-bit performance while significantly reducing memory costs.
    • LLM.int8() (8-bit quantization): Enables LLM inference with approximately half the required memory and no performance degradation. It uses vector-wise quantization to quantize most features to 8-bits, while treating outliers separately with 16-bit matrix multiplication.
    • QLoRA (4-bit quantization): Enables LLM training using 4-bit quantization combined with trainable low-rank adaptation (LoRA) weights, allowing for memory-efficient training without compromising performance.

  2. Overview of bitsandbytes features

    main

    bitsandbytes provides k-bit quantization for PyTorch to reduce memory consumption during LLM inference and training. The core features include:

    • 8-bit Optimizers: Uses block-wise quantization to maintain 32-bit performance at a fraction of the memory cost via the bitsandbytes.optim module.
    • LLM.int8() (8-bit Quantization): Enables inference with approximately half the required memory using vector-wise quantization and 16-bit matrix multiplication for outliers. Accessible via bitsandbytes.nn.Linear8bitLt.
    • QLoRA (4-bit Quantization): Enables efficient training by quantizing models to 4-bits and using trainable low-rank adaptation (LoRA) weights. Accessible via bitsandbytes.nn.Linear4bit.

  3. Use Stochastic Gradient Descent (SGD) optimizers in bitsandbytes

    main
    bitsandbytes provides implementations of Stochastic Gradient Descent (SGD) to minimize loss by updating model parameters in the opposite direction of the gradient using randomly sampled mini-batches. These implementations support momentum and Nesterov momentum to accelerate convergence by incorporating a weighted average of past gradients.

  7. Use RMSprop optimizers in bitsandbytes

    main

    bitsandbytes provides implementations of the RMSprop adaptive learning rate optimizer. RMSprop stores a weighted average of the squared past gradients for each parameter to scale the learning rate, preventing it from diminishing.

    Available implementations include:

    • bitsandbytes.optim.RMSprop: The standard implementation.
    • bitsandbytes.optim.RMSprop8bit: An 8-bit quantized version for reduced memory usage.
    • bitsandbytes.optim.RMSprop32bit: A 32-bit version.

  9. Use Adam optimizers in bitsandbytes

    main

    bitsandbytes provides several implementations of the Adam (Adaptive moment estimation) optimizer. These implementations allow for adaptive learning rates by maintaining a weighted average of past gradients (first-moment) and squared past gradients (second-moment).

    Available variants include:

    • Standard Adam: Adam, Adam8bit, and Adam32bit.
    • Paged Adam: PagedAdam, PagedAdam8bit, and PagedAdam32bit. Paged optimizers utilize CUDA's unified memory to offload optimizer states from GPU memory to CPU memory when GPU memory is exhausted, helping to prevent Out-of-Memory (OOM) errors.

  10. Use the Lion optimizer

    main

    Lion (Evolved Sign Momentum) is a memory-efficient and fast optimizer that uses the sign of the gradient to determine the update direction of the momentum. It is an alternative to AdamW that avoids tracking and storing second-order moments, reducing memory overhead.

    bitsandbytes provides several variants of the Lion optimizer:

    • Lion: Standard implementation.
    • Lion8bit: 8-bit quantized version for reduced memory usage.
    • Lion32bit: 32-bit implementation.
    • PagedLion: Paged version (useful for managing memory spikes by offloading to CPU).
    • PagedLion8bit: 8-bit quantized paged version.
    • PagedLion32bit: 32-bit paged version.

  11. Use LLM.int8() quantization for large language models

    main
    LLM.int8() is a quantization method designed to make large language model (LLM) inference more accessible with minimal accuracy degradation. It works by dynamically identifying outliers in inputs and weights. These outliers are processed using 16-bit precision, while all other values are processed in 8-bit precision before being dequantized back to 16-bit. The results are then combined to produce the final output, preserving critical information that naive 8-bit quantization might lose.

  12. Use AdamW optimizers in bitsandbytes

    main

    bitsandbytes provides several variants of the AdamW optimizer, which separates weight decay from the gradient update. You can choose between standard versions and 'Paged' versions.

    Paged Optimizers: These take advantage of CUDA's unified memory to automatically transfer memory from the GPU to the CPU when GPU memory is exhausted, helping to prevent Out-of-Memory (OOM) errors.