DOCSEARCHbyAlgolia
Submit Docs

RamaLama Documentation

repository·main·Indexed 25 days ago

https://github.com/containers/ramalama

RamaLama is an open-source command line tool for local AI LLM model inference using OCI containers. It simplifies the process by automatically detecting host hardware (GPUs) to pull optimized container images, allowing users to run models securely via REST APIs or chatbots. It supports multiple transports including HuggingFace, ModelScope, Ollama, and OCI registries, and integrates with Podman and Docker for container isolation.

Tokens
38.5K
Snippets
100
Records
323
Agent score
81%

What's inside RamaLama

  1. Overview of RamaLama

    main

    RamaLama is an open-source tool designed to simplify the local use and serving of AI models for inference using OCI containers. It allows developers to apply container-centric development patterns to AI workflows.

    Key features include:

    • Automatic Hardware Optimization: Detects host GPUs and pulls specific accelerated container images to handle dependencies and hardware optimization automatically.
    • Registry Support: Works with multiple AI model registries, including OCI Container Registries.
    • Container-Native Workflow: Treats models similarly to how Podman and Docker treat container images, allowing the use of common container commands.
    • Security and Isolation: Runs models in rootless containers to isolate them from the host. By default, it provides no network access and removes all temporary data upon application exit.
    • Flexible Interaction: Models can be interacted with via a REST API or a chatbot interface.

  2. Understand the RamaLama oci:// Image Format

    main
    RamaLama uses the oci:// transport to store AI models within OpenContainers image registries. Each model is stored as a standard single-platform container image (using an OCI Image Manifest rather than an OCI Image Index). The model data is wrapped in standard tar layers.

  3. Understand RamaLama security model

    main

    RamaLama runs AI models inside rootless containers using Podman or Docker to isolate models from the host system. The security footprint includes:

    • Container Isolation: Models run in isolated containers.
    • Read-Only Volume Mounts: AI models are mounted as read-only volumes, preventing processes inside the container from modifying host files.
    • No Network Access: The ramalama run command uses --network=none, preventing outbound connectivity and data leakage.
    • Auto-Cleanup: Containers run with the --rm flag, ensuring temporary data is deleted when the session ends.
    • Privilege Restriction: Containers are run with dropped Linux capabilities and the No New Privileges kernel feature enabled to prevent privilege escalation.

  4. Quickstart: Pull and Run models with RamaLama

    main

    Once installed, use the following commands to start using models:

    1. Check version: ramalama version
    2. Pull a model: ramalama pull <model_name>
    3. Run a chatbot: ramalama run <model_name>
    4. View help: ramalama --help
    # Pull a model
ramalama pull tinyllama

# Run a chatbot
ramalama run tinyllama

  6. Configure GPU acceleration on macOS

    main

    On macOS, you can choose between two GPU acceleration paths:

    1. Podman with krunkit (Containerized)

    Uses GPU passthrough via krunkit.

    Prerequisites:

    1. Add provider = "libkrun" to the [machine] section of your containers.conf (usually $HOME/.config/containers/containers.conf).
    2. Install krunkit and podman via Homebrew.
    3. Initialize and start the Podman machine.

    2. Native MLX (Non-containerized)

    Runs directly on the host using Apple's MLX framework. This is faster than containerized GPU passthrough.

    Prerequisites:

    1. Install mlx-lm via Homebrew: brew install mlx-lm.
    2. Configure RamaLama to use the MLX runtime. You can do this via CLI flag --runtime=mlx or by updating $HOME/.config/ramalama/ramalama.conf:
    [ramalama]
runtime = "mlx"

    Note: If Podman is installed, RamaLama defaults to it. To force MLX, use the --nocontainer flag or set container = false in your config:

    [machine]
container = false
    [machine]
provider = libkrun

  7. Run RamaLama on Fedora Silverblue or Toolbox

    main

    Since Fedora Silverblue is an immutable system, you have two primary ways to run RamaLama:

    1. Toolbox: Create a Toolbox container and install RamaLama inside it (via pip install ramalama or dnf install ramalama). Ensure the toolbox has access to the host's container engine (e.g., by bind-mounting the socket or configuring it to use the host's podman command).
    2. Host Installation: Install via rpm-ostree install ramalama if available.

    Note: The model store defaults to ~/.local/share/ramalama, which is writable on Silverblue. If running in a toolbox, ensure the model store is pointed to a writable location like your home directory using the --store option.

  9. Prerequisites for RamaLama with CUDA in WSL2

    main

    Before setting up CUDA support for RamaLama in WSL2 using Docker, ensure the following requirements are met:

    1. NVIDIA Game-Ready Drivers: Install appropriate NVIDIA game-ready drivers on your Windows host system.
    2. Docker Desktop: Install Docker Desktop on Windows and verify that Use the WSL 2 based engine is enabled in the General settings.

  10. Reset RamaLama installation

    main
    Because RamaLama is currently in alpha and subject to breaking changes, you may need to reset your installation. To do this, follow the instructions in the Uninstall section of the documentation to remove RamaLama and clean up all associated data files, then perform a fresh installation.

  11. List models served by an inference server with `ramalama models`

    main

    Use the ramalama models command to list model identifiers currently exposed by a running inference server (e.g., one started with ramalama serve).

    Unlike ramalama list (which shows models stored locally), ramalama models queries a remote or local server. It first attempts to query the OpenAI-compatible /v1/models endpoint (compatible with llama.cpp, MLX, etc.) and falls back to the llama.cpp native /models endpoint if necessary.

    $ ramalama models
tinyllama

  12. Install the NVIDIA Container Toolkit on Debian-based distros (Ubuntu)

    main

    To install the NVIDIA Container Toolkit on Ubuntu or Debian-based systems, configure the production repository, update your package list, and then install the toolkit.

    # Configure the Production Repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Update and Install
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit