LLMKube

The ModelRouter is a Kubernetes Custom Resource Definition (CRD) that provides a single, OpenAI-compatible HTTP endpoint to dispatch LLM requests across multiple backends. It acts as a cross-engine handoff layer for agentic chains, allowing an agent to move between local models and cloud providers based on declarative policies.

Key Capabilities:

• Backend Support: Routes to local InferenceService instances, external providers (Anthropic, OpenAI), or a LiteLLM proxy.
• Policy-Driven Routing: Uses rules based on data classification, task complexity, or headers to decide where a request goes.
• Fail-Closed Semantics: Ensures sensitive data (e.g., pii, phi) never leaves the cluster by refusing requests if local backends are unhealthy, rather than falling back to cloud providers.
• Audit Logging: Emits one audit-log line per request to stdout.
• Streaming: Supports SSE/chunked response streaming without buffering.

Status: ModelRouter is in alpha (as of LLMKube 0.8). The CRD resides in inference.llmkube.dev/v1alpha1.

apiVersion: inference.llmkube.dev/v1alpha1
kind: ModelRouter
metadata:
  name: coding-router
spec:
  backends:
    - name: local-qwen
      tier: local
      inferenceServiceRef:
        name: qwen3-coder
    - name: cloud-opus
      tier: cloud
      external:
        provider: anthropic
        model: claude-opus-4-7
        url: https://api.anthropic.com
        credentialsSecretRef:
          name: anthropic-key

  rules:
    - name: pii-stays-local
      match:
        dataClassification: ["pii", "phi"]
      route:
        backends: ["local-qwen"]
      failClosed: true

    - name: complex-to-cloud
      match:
        taskComplexity: complex
      route:
        backends: ["cloud-opus", "local-qwen"]
        strategy: primary-fallback

  defaultRoute: local-qwen

Foreman is an opt-in orchestrator for agentic workloads built on top of LLMKube. It uses its own API group (foreman.llmkube.dev) and introduces four CRDs: Workload, AgenticTask, Agent, and FleetNode.

Foreman is designed for high-scale, autonomous agentic tasks (like automated coding/testing pipelines) where on-prem hardware or sovereignty constraints make cloud-only models unsuitable. It features a capability-aware scheduler and a native Go agent loop that executes OpenAI function-calling against local inference endpoints.

A typical Workload defines an intent, a repository, specific issues to address, and a set of specialized agents (coder, verifier, reviewer) to execute the task pipeline.

apiVersion: foreman.llmkube.dev/v1alpha1
kind: Workload
metadata:
  name: fix-small-bugs
  namespace: default
spec:
  intent: "Fix small open issues"
  repo: defilantech/LLMKube
  issues: [510, 526, 449]
  coderAgentRef:    { name: qwen36-35b-carnice-mtp-coder }
  verifierAgentRef: { name: shadowstack-gate }
  reviewerAgentRefs:
    - { name: qwen36-35b-a3b-reviewer }
    - { name: devstral-24b-reviewer }

LLMKube supports three primary model cache topologies depending on your cluster requirements and storage capabilities:

Key Concept: The Downloader Lifecycle When using claimName with a remote URL, the downloader is an init container within the inference pod itself. This ensures the downloader inherits the same node selection and GPU tolerations as the serving container, allowing it to function even on strictly tainted nodes.

LLMKube uses a persistent model cache to avoid re-downloading large models (e.g., 13B-70B) every time an InferenceService is recreated.

Architecture

• Per-Namespace PVCs: A PersistentVolumeClaim (PVC) named llmkube-model-cache is automatically created in every namespace where you deploy models. This provides namespace isolation and simple RBAC.
• Lifecycle:
  • The first pod in a namespace triggers an init-container to download the model into the llmkube-model-cache PVC.
  • Subsequent pods mount the PVC as read-only and skip the download process, allowing deployments to complete in seconds rather than minutes.
• Cache Keys: Models are indexed in the PVC using a SHA256 hash (first 16 characters) of their source URL. The resulting path follows the pattern /models/<cache-key>/model.gguf.

Example Cache Mapping:
Source: https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf
Cache Key: a3b8c9d4e5f67890
Path: /models/a3b8c9d4e5f67890/model.gguf

The Foreman Reviewer Agent is a pipeline step (Step 3) that follows the Coder Agent and the automated gate job. Its purpose is to perform a qualitative review of a code diff to determine if it addresses the intended issue in a way a human maintainer would accept.

Workflow Context:

1. Coder Agent: Produces a branch on a fork.
2. Gate Job: Runs make fmt vet lint test and codegen-sync against that branch.
3. Reviewer Agent: Evaluates the branch. If the gateVerdict is not GATE-PASS, the reviewer should still perform reads and provide a useful review, but must not approve the change.

Input Payload: The agent receives the following data in its task payload:

• repo: GitHub owner/name (e.g., defilantech/LLMKube)
• issue: The integer issue number being addressed.
• branch: The branch name on the fork (e.g., foreman/issue-510).
• gateVerdict: Either GATE-PASS or GATE-FAIL.
• coderSummary: A one-line summary provided by the coder.