DOCSEARCHbyAlgolia
Submit Docs

Ollama Operator

repository·main·Indexed 42 days ago

https://github.com/nekomeowww/ollama-operator

A Kubernetes operator that simplifies running large language models (LLMs) on clusters. It automates the deployment of model inferencing servers and manages model image storage using Ollama. It includes the kollama CLI for streamlined deployment and supports the Model CRD (ollama.ayaka.io/v1) for advanced configuration of scaling, storage, and resource limits (CPU, Memory, NVIDIA/AMD GPUs).

Tokens
12.5K
Snippets
45
Records
62
Agent score
52%

What's inside Ollama Operator

  1. Overview of Ollama Operator

    main

    The Ollama Operator is a Kubernetes operator designed to bring the ease of use of the Ollama CLI to Kubernetes clusters. It allows users to manage multiple large language models (LLMs) across a cluster with complex resource configurations.

    Key benefits include:

    • Multi-model support: Run multiple models on the same cluster.
    • Ollama Compatibility: Fully compatible with all Ollama models, APIs, and the standard Ollama CLI.
    • Environment Simplification: Leverages llama.cpp to eliminate the need for manual Python environment setup or CUDA driver management.
    • Kubernetes Native: Works on general Kubernetes clusters, K3s (Raspberry Pi, TrueNAS SCALE), kind, and minikube.
    • Efficient Image Management: Pulls images once and shares them across nodes.
    • Standard Networking: Easily exposed using standard Kubernetes Services and Ingress controllers.

  2. Use the `kollama` CLI to manage Ollama deployments

    main

    The kollama CLI is used to manage the lifecycle of Ollama models within the cluster. It provides commands to deploy models, remove deployments, and expose models for access.

    Available primary commands:

    • kollama deploy: Deploys a specific model.
    • kollama undeploy: Removes a deployed model.
    • kollama expose: Exposes a deployed model to make it accessible.

  3. What is a Model CRD

    main

    The Model Custom Resource Definition (CRD) represents an Ollama server instance within a Kubernetes cluster. Creating a Model resource automatically triggers the creation of a Kubernetes Deployment and a Service.

    On its first creation, the operator also creates a StatefulSet and a PersistentVolumeClaim (PVC) to ensure the downloaded Ollama images are stored and persisted.

  4. How the Ollama Operator architecture works

    main

    The Ollama Operator manages two primary components to enable model inferencing in a Kubernetes cluster:

    1. Model Inferencing Server: A gRPC server that runs the model and serves the model's API. This is implemented as a Kubernetes Deployment.
    2. Model Image Storage: A dedicated service used to cache and reuse downloaded models. It is implemented using a StatefulSet and a PersistentVolumeClaim (PVC) to ensure models are stored on a dynamically provisioned PersistentVolume rather than being re-downloaded for every request.

    Why Model Image Storage is required: Ollama models created via a Modelfile are valid OCI format images, but they use an incompatible contentType and structure that prevents them from being run directly by standard container runtimes. To solve this, the operator starts a standalone service using the ollama serve command. When a new model inference service is created, it sends an image pull request directly to this storage service to retrieve the cached model.

  5. Supported Kubernetes environments

    main

    The Ollama Operator can be installed on various Kubernetes distributions, including:

    • Standard Kubernetes clusters (Cloud or self-managed)
    • K3s (e.g., on Raspberry Pi or TrueNAS SCALE)
    • kind (Kubernetes in Docker)
    • minikube

    For local development on macOS or Windows, you can use Docker Desktop or OrbStack in combination with kind or minikube to create a local cluster for the operator.

  7. Install the Ollama Operator

    main

    To use the Ollama Operator, you must first install it into your Kubernetes cluster. The installation process typically involves setting up a Kubernetes cluster (e.g., using kind) and then applying the operator manifests to the cluster.

    Refer to the specific installation commands provided in the official documentation snippets for the exact kubectl commands required for your environment.

  10. Install and use the kollama CLI

    main

    The kollama CLI is a native tool shipped by the Ollama Operator to simplify interactions. You can install it using go install.

    To deploy a model (e.g., phi) with an exposed NodePort on port 30001, use the deploy command.

    # Install kollama
go install github.com/nekomeowww/ollama-operator/cmd/kollama@latest

# Deploy a model
kollama deploy phi --expose --node-port 30001

  11. Configure resource limits for a Model

    main

    You can specify resource limits for a deployed Model using the --limit flag. This flag supports multiple uses to set different resource types. The format is <resource>=<quantity>.

    CPU and Memory

    kollama deploy phi --limit=cpu=1 --limit=memory=1Gi

    GPU Resources

    To request specific GPU vendors, use their Kubernetes resource labels:

    • NVIDIA: --limit=nvidia.com/gpu=1
    • AMD: --limit=amd.com/gpu=1

    Updating existing limits

    If a model is already deployed, you can update its resource limits using kubectl set resources with a label selector:

    # Update CPU
kubectl set resources deployment -l model.ollama.ayaka.io/name=<model name> --limits cpu=4

# Update Memory
kubectl set resources deployment -l model.ollama.ayaka.io/name=<model name> --limits memory=8Gi
    kollama deploy phi --limit=nvidia.com/gpu=1

  12. Use out-of-the-box supported models

    main

    You can deploy pre-configured models by specifying the model image in your Model Custom Resource Definition (CRD). The operator supports a wide range of models from the Ollama library, including Llama 3, Mistral, and Phi-3.

    # Example usage in a Model CRD
# (Note: The exact YAML structure depends on your CRD definition)
model:
  image: "llama3"