repository·main·Indexed 20 days ago
https://github.com/docker/model-runnerA toolset for managing, running, and deploying AI models (such as LLMs) using Docker. It provides a unified interface via a Docker CLI plugin or a standalone binary (dmr) to pull and serve models from OCI-compliant registries. The toolset supports deployment on Docker Desktop and Kubernetes clusters via Helm, offers GPU support for NVIDIA and AMD, and includes capabilities for packaging GGUF models into OCI artifacts and integrating with Open WebUI.
GGUF and Safetensors model formats and provides mechanisms for local model storage and metadata management.The docker model command is the entrypoint for the Docker Model Runner CLI. It allows you to run, manage, and interact with AI models directly from your command line. You can use it to pull models from Docker Hub or HuggingFace, run them in chat mode, benchmark performance, and manage the Model Runner lifecycle (start, stop, restart, install, uninstall).
docker model <subcommand> [options]The model-cli gateway is a lightweight, OpenAI-compatible LLM proxy designed to sit in front of Docker Model Runner (DMR) or other providers. It provides several enterprise-grade features including:
/v1/chat/completions, /v1/models, and /v1/embeddings.To optimize your development workflow with Docker Model Runner:
ai/smollm2) for faster response times during development.--detach flag when running models to pre-load them in the background, which is better for automated scripts.The dmr tool is designed to be engine-independent by bypassing the standard Docker Engine connection logic used by the docker model CLI plugin.
Key architectural components:
dmr serve runs the pkg/server in-process. It listens on TCP port 12434 by default, or a Unix socket if specified via the --socket flag.dmr CLI communicates with its daemon via the MODEL_RUNNER_HOST environment variable (which defaults to http://localhost:12434). This forces the "manual host" code path, ensuring it does not attempt to probe a Docker Engine or Docker Desktop.dmr matches the docker model CLI plugin command tree, providing full feature parity for operations like run, pull, push, tag, inspect, logs, bench, and configure.You can define fallback behavior in the general_settings section of your configuration file. This allows the gateway to automatically route a request to a secondary provider if the primary one fails.
Example Configuration:
In this setup, if fast fails, it tries local. If smart fails, it tries fast, and if that fails, it tries local.
model_list:
- model_name: fast
params:
model: groq/llama-3.1-8b-instant
api_key: os.environ/GROQ_API_KEY
- model_name: smart
params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
- model_name: local
params:
model: docker_model_runner/ai/smollm2
api_base: http://localhost:12434/engines/llama.cpp/v1
general_settings:
num_retries: 2
fallbacks:
- fast: [local]
- smart: [fast, local]general_settings:
num_retries: 2
fallbacks:
- fast: [local]
- smart: [fast, local]The distribution package uses a layered interface hierarchy to separate low-level storage from high-level inference. This allows the system to support different storage backends (like OCI registries) and different model formats (like Docker or CNCF ModelPack) while providing a consistent interface for inference runtimes.
oci.Image: The foundation. Handles low-level OCI artifact operations (layers, manifests, digests) for registry storage.types.ModelArtifact: Extends oci.Image. Used for the building and pushing phases of a model's lifecycle.partial.BaseModel: A common implementation used to bridge artifacts to usable models by managing layer lists and configuration files.types.Model: Represents a stored model. It is used to resolve file paths (GGUF, Safetensors, etc.) for inference.types.ModelBundle: The final stage. Represents an unpacked model ready for immediate execution by a runtime.┌─────────────────────────────────────────────────────────────────┐
│ oci.Image │
│ (Low-level OCI artifact: Layers, Manifest, Digest, etc.) │
└───────────────────────────────┬─────────────────────────────────┘
│ embeds
▼
┌─────────────────────────────────────────────────────────────────┐
│ types.ModelArtifact │
│ (Building & pushing: ID, Config, Descriptor) │
└───────────────────────────────┬─────────────────────────────────┘
│ implemented by
▼
┌─────────────────────────────────────────────────────────────────┐
│ partial.BaseModel │
│ (Common implementation with LayerList, ConfigFile) │
└─────────────────────────────────────────────────────────────────┘The /metrics endpoint aggregates metrics from all active llama.cpp runners. While metrics retain their original names and types, the model-runner adds three specific labels to every metric to allow for granular filtering in Prometheus:
backend: The inference backend (e.g., llama.cpp).model: The specific model name (e.g., llama3.2:latest).mode: The operation mode, which is either completion or embedding.# HELP llama_prompt_tokens_total Total number of prompt tokens processed
# TYPE llama_prompt_tokens_total counter
llama_prompt_tokens_total{backend="llama.cpp",model="llama3.2:latest",mode="completion"} 4934
llama_prompt_tokens_total{backend="llama.cpp",model="ai/mxbai-embed-large:335M-F16",mode="embedding"} 4525
# HELP llama_generation_tokens_total Total number of tokens generated
# TYPE llama_generation_tokens_total counter
llama_generation_tokens_total{backend="llama.cpp",model="llama3.2:latest",mode="completion"} 2156
# HELP llama_requests_total Total number of requests processed
# TYPE llama_requests_total counter
llama_requests_total{backend="llama.cpp",model="llama3.2:latest",mode="completion"} 127
llama_requests_total{backend="llama.cpp",model="ai/mxbai-embed-large:335M-F16",mode="embedding"} 89# HELP llama_prompt_tokens_total Total number of prompt tokens processed
# TYPE llama_prompt_tokens_total counter
llama_prompt_tokens_total{backend="llama.cpp",model="llama3.2:latest",mode="completion"} 4934To deploy Docker Model Runner on Docker Desktop, apply the provided static manifest and wait for the deployment to become available. Once ready, you can run a model by setting the MODEL_RUNNER_HOST environment variable to point to your local instance.
kubectl apply -f static/docker-model-runner-desktop.yaml
kubectl wait --for=condition=Available deployment/docker-model-runner --timeout=5m
MODEL_RUNNER_HOST=http://localhost:31245 docker model run ai/smollm2:latestTo deploy on a standard Kubernetes cluster, apply the docker-model-runner.yaml manifest and use kubectl port-forward to expose the deployment to your local machine. After forwarding, set MODEL_RUNNER_HOST to access the runner via the CLI.
kubectl apply -f static/docker-model-runner.yaml
kubectl wait --for=condition=Available deployment/docker-model-runner --timeout=5m
kubectl port-forward deployment/docker-model-runner 31245:12434
# Run a model
MODEL_RUNNER_HOST=http://localhost:31245 docker model run ai/smollm2:latestIf you are developing or building the CLI manually, follow these steps:
make build to compile the binary.git clone https://github.com/docker/model-cli.git
cd model-cli
make build
./model-cli install-runnerYou can build all published target binaries for dmr locally using make. Since dmr has no cgo dependencies, all targets can be cross-compiled from any host. The resulting binaries are placed in dist/dmr/<os>-<arch>/dmr.
make build-dmr-cross