DOCSEARCHbyAlgolia
Submit Docs

Distributed Llama

repository·main·Indexed 25 days ago

https://github.com/b4rtaz/distributed-llama

A system for accelerating LLM inference by connecting multiple devices into a cluster using tensor parallelism and high-speed synchronization over Ethernet. It supports Linux, macOS, and Windows, optimizing for ARM and x86_64 AVX2 CPUs, and includes optional GPU support via the Vulkan API. The system utilizes a Root Node for model management and synchronization, and Worker Nodes for processing neural network slices. It provides a CLI for inference and chat, an API server, and tools to convert Hugging Face models (llama, mistral, qwen3, qwen3_moe) to the Distributed Llama format.

Tokens
9K
Snippets
28
Records
58
Agent score
84%

What's inside distributed-llama

  1. Understand Distributed Llama Architecture

    main

    Distributed Llama uses a cluster architecture to accelerate LLM inference via tensor parallelism over Ethernet. The system consists of two node types:

    • Root Node: Responsible for loading model weights, forwarding them to workers, and synchronizing the neural network state. The root node also acts as a worker, processing its own slice of the network.
    • Worker Node: Processes its own slice of the neural network. Workers do not require model configuration.

    Key constraints:

    • You must run on $2^n$ nodes (e.g., 1, 2, 4, 8...).
    • The maximum number of nodes is limited by the number of KV heads in the model.
    • The root node requires more RAM than worker nodes because it manages the model and synchronization.

  2. Set up Distributed Llama on Raspberry Pi

    main

    To run Distributed Llama across multiple Raspberry Pi devices, follow these hardware and software preparation steps:

    1. OS Installation: Install Raspberry Pi OS Lite (64 bit) on all devices (Root and Workers).
    2. Networking: Connect all devices to a switch or router via Ethernet. For two devices, a direct Ethernet connection is preferred.
    3. Dependencies: Install git on all devices:
      sudo apt install git
    4. Compilation: Clone the repository and compile both the main binary and the API server on all devices:
      git clone https://github.com/b4rtaz/distributed-llama.git
cd distributed-llama
make dllama
make dllama-api
    5. IP Configuration: Assign unique static IP addresses to each device on the same subnet (e.g., 10.0.0.x/24) using ip addr.
    sudo apt install git

git clone https://github.com/b4rtaz/distributed-llama.git
cd distributed-llama
make dllama
make dllama-api

# Example IP assignment for Root
sudo ip addr add 10.0.0.1/24 dev eth0

  3. Setup the Root Node using launch.py

    main

    If you have Python 3 and a C++ compiler installed, you can set up a Root Node and automatically download the required model and tokenizer using the launch.py script. This is the fastest way to get started with pre-configured models.

    Supported models include various Llama, DeepSeek, and Qwen versions. Use the specific model identifier as an argument to the script.

  4. Build Distributed Llama with GPU support

    main

    To enable GPU support via the Vulkan API, you must install the Vulkan SDK for your platform and build the project with the DLLAMA_VULKAN=1 environment variable.

    Prerequisites:

    Build Commands: Build the main CLI and the API server using:

    DLLAMA_VULKAN=1 make dllama
DLLAMA_VULKAN=1 make dllama-api

  5. Run Distributed Llama Workers

    main

    On every WORKER device, start the worker process. It is recommended to use nice with a negative value to grant the process high priority.

    Example command for a worker on port 9999 using 4 threads:

    sudo nice -n -20 ./dllama worker --port 9999 --nthreads 4
    sudo nice -n -20 ./dllama worker --port 9999 --nthreads 4

  6. Download and Manage Models with launch.py

    main

    Use the launch.py script on the ROOT device to manage model downloads. You do not need to download models on worker devices.

    • To see a list of available models: python3 launch.py
    • To download a specific model (e.g., llama3_2_3b_instruct_q40): python3 launch.py llama3_2_3b_instruct_q40
    python3 launch.py llama3_2_3b_instruct_q40

  7. Run Distributed Llama on GPU

    main

    Once built with Vulkan support, you can run the root node, worker node, or API server on a GPU device.

    Important Requirements:

    • The Vulkan backend requires a single thread. You must set --nthreads 1 when using GPU.
    • Use the --gpu-index <index> flag to select the device (e.g., --gpu-index 0 for the first device).

    Usage Examples:

    ./dllama inference ... --nthreads 1 --gpu-index 0 
./dllama chat      ... --nthreads 1 --gpu-index 0 
./dllama worker    ... --nthreads 1 --gpu-index 0 
./dllama-api       ... --nthreads 1 --gpu-index 0

  11. Start the Distributed Llama API Server

    main

    To expose the distributed model via an API, run the dllama-api binary on the ROOT device. This allows external clients (like web UIs) to interact with the model.

    Key Flags:

    • --host: The interface to bind to (use 0.0.0.0 to allow external connections).
    • --port: The API port.
    • --model: Path to the .m model file.
    • --tokenizer: Path to the .t tokenizer file.
    • --buffer-float-type: Floating point type (e.g., q80).
    • --nthreads: Number of CPU threads.
    • --max-seq-len: Maximum sequence length.
    • --workers: Space-separated list of IP:PORT for all worker nodes.
    sudo nice -n -20 ./dllama-api \
  --host 0.0.0.0 \
  --port 9999 \
  --model models/llama3_2_3b_instruct_q40/dllama_model_llama3_2_3b_instruct_q40.m \
  --tokenizer models/llama3_2_3b_instruct_q40/dllama_tokenizer_llama3_2_3b_instruct_q40.t \
  --buffer-float-type q80 \
  --nthreads 4 \
  --max-seq-len 4096 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999