llama-cpp-python

To enable Metal (MPS) acceleration on MacOS:

Build from source:

CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

Pre-built wheels: Requirements:

• MacOS 11.0 or later
• Python 3.10, 3.11, or 3.12

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal

You can download models in gguf format directly using the Llama.from_pretrained method. This requires the huggingface-hub package (pip install huggingface-hub). Models are downloaded to the Hugging Face cache directory by default.

llm = Llama.from_pretrained(
    repo_id="lmstudio-community/Qwen3.5-0.8B-GGUF",
    filename="*Q8_0.gguf",
    verbose=False
)

The API supports OpenAI-compatible tool/function calling. This is typically achieved using the chatml-function-calling format or functionary models.

For Functionary v2 models, you must provide a LlamaHFTokenizer (initialized via LlamaHFTokenizer.from_pretrained) to handle tokenizer discrepancies between llama.cpp and Hugging Face.

from llama_cpp import Llama
from llama_cpp.llama_tokenizer import LlamaHFTokenizer
llm = Llama.from_pretrained(
  repo_id="meetkai/functionary-small-v2.2-GGUF",
  filename="functionary-small-v2.2.q4_0.gguf",
  chat_format="functionary-v2",
  tokenizer=LlamaHFTokenizer.from_pretrained("meetkai/functionary-small-v2.2-GGUF")
)

Speed up generation by using a draft model via the LlamaPromptLookupDecoding class. Pass this instance to the draft_model parameter during Llama initialization.

• num_pred_tokens=10: Generally good for GPU.
• num_pred_tokens=2: Performs better for CPU-only machines.

from llama_cpp import Llama
from llama_cpp.llama_speculative import LlamaPromptLookupDecoding

lama = Llama(
    model_path="path/to/model.gguf",
    draft_model=LlamaPromptLookupDecoding(num_pred_tokens=10)
)

This option automates the process of downloading an Apache V2.0 licensed 3B parameter Open LLaMA model and installing it into a Docker image running an OpenBLAS-enabled llama-cpp-python server.

Steps:

1. Navigate to the directory: cd ./open_llama
2. Run the build script: ./build.sh
3. Run the start script: ./start.sh

$ cd ./open_llama
./build.sh
./start.sh

Use the cuda_simple Dockerfile to build and run a server with NVIDIA GPU acceleration via CuBLAS. This method keeps the model file outside the Docker image.

Requirements:

• NVIDIA GPU with sufficient VRAM.
• Docker NVIDIA support installed (NVIDIA Container Toolkit).
• The model file must exist on the host system.

Steps:

1. Navigate to the directory: cd ./cuda_simple
2. Build the image: docker build -t cuda_simple .
3. Run the container using the --gpus=all flag and mapping your host model path to /var/model inside the container.

cd ./cuda_simple
docker build -t cuda_simple .
docker run --gpus=all --cap-add SYS_RESOURCE -e USE_MLOCK=0 -e MODEL=/var/model/<model-path> -v <model-root-path>:/var/model -t cuda_simple

You can configure llama.cpp build options (like hardware acceleration backends) using the CMAKE_ARGS environment variable or the -C / --config-settings CLI flag during installation.

Using Environment Variables (Linux/Mac):

CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python

Using Environment Variables (Windows PowerShell):

$env:CMAKE_ARGS = "-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS"
pip install llama-cpp-python

Using CLI flags:

pip install llama-cpp-python -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

You can use llama-cpp-python as a backend for GitHub Copilot. This requires a code completion model in GGUF format and a significantly increased context size (--n_ctx).

1. Run the server with high context:

python3 -m llama_cpp.server --model <model_path> --n_ctx 16192

2. Update your .vscode/settings.json to point to your local server host and port:

{
    "github.copilot.advanced": {
        "debug.testOverrideProxyUrl": "http://<host>:<port>",
        "debug.overrideProxyUrl": "http://<host>:<port>"
    }
}

Structured function calling is supported via the OpenAI function calling API. To use models like functionary, you must specify the functionary-v1 or functionary-v2 chat_format. Because these models require a HuggingFace tokenizer, you must also provide the path to the tokenizer using --hf_pretrained_model_name_or_path.

python3 -m llama_cpp.server --model <model_path_to_functionary_v2_model> --chat_format functionary-v2 --hf_pretrained_model_name_or_path <model_path_to_functionary_v2_tokenizer>