OpenClaw Kubernetes Operator

repository·main·Indexed 18 days ago

https://github.com/paperclipinc/openclaw-operator

A production-grade tool for self-hosting OpenClaw AI agents on Kubernetes. It automates the deployment of a secure, observable, and adaptive agent stack using the OpenClawInstance Custom Resource, managing components such as StatefulSets, Services, RBAC, NetworkPolicies, and PVCs. Features include autonomous agent self-configuration, flexible configuration modes (merge vs overwrite), automated skill and plugin installation, and disk-aware readiness probes.

Tokens
51.6K
Snippets
122
Records
193
Agent score
63%

What's inside openclaw-operator

  1. Overview of the OpenClaw Operator

    main

    The OpenClaw Operator is a Kubernetes operator designed to run OpenClaw AI agents with production-grade security, observability, and lifecycle management.

    Instead of managing individual Kubernetes components, you define a single OpenClawInstance Custom Resource (CR). The operator then automatically reconciles this resource into a complete, hardened stack including:

    • Workloads: StatefulSet, PodDisruptionBudget
    • Networking: Service, Ingress, NetworkPolicy
    • Storage: PersistentVolumeClaim (PVC)
    • Security: RBAC
    • Observability: ServiceMonitor
  2. Core features and security hardening in OpenClaw Operator

    main

    As of v0.33.0, the operator provides a hardened environment for OpenClaw fleet management. Key capabilities include:

    Security & Hardening

    • Default Security Posture: Non-root UID 1000, readOnlyRootFilesystem, dropping ALL capabilities, and seccomp: RuntimeDefault.
    • Network Security: Default-deny NetworkPolicy and per-instance RBAC.
    • Process Management: Shared PID namespace (enabled by default via spec.shareProcessNamespace: true) for zombie reaping.

    Management & Observability

    • Lifecycle: Full StatefulSet lifecycle management with auto-update and rollback capabilities.
    • Configuration: Support for inline and external config, JSON5, and OpenClawClusterDefaults for singleton cluster-wide settings.
    • Observability: OTel Collector sidecar for operator metrics and ServiceMonitor support.
    • Storage: PVC management with backup/restore-on-delete and SecretsReady conditions.

    Agent Capabilities

    • Skills & Plugins: Declarative skill installation and declarative plugin installation via spec.plugins.
    • Sidecars: Support for Chromium (with anti-bot CDP proxy), Tailscale (with persistent state), ttyd web terminal, and Nginx gateway proxy.
    • Self-Configuration: OpenClawSelfConfig allows agents to perform self-modifications based on an allowlist policy.
  3. How configuration changes trigger rollouts

    main

    The operator ensures that configuration updates are applied automatically by computing a SHA-256 hash of the configuration. This hash is stored as the annotation openclaw.rocks/config-hash on the pod template.

    When you update the configuration (either via configMapRef or raw), the hash changes. This change to the pod template triggers a standard Kubernetes rolling update of the Deployment, ensuring the new configuration is picked up without requiring a manual restart.

  4. Configure Health Probes and Disk Readiness

    main

    The ProbesSpec manages the health checks for an OpenClawInstance.

    Standard Probes:

    • liveness: Checks if the container is alive.
    • readiness: Checks if the container is ready to accept traffic.
    • startup: Checks if the application has finished starting up.

    Each probe uses ProbeSpec which includes enabled, initialDelaySeconds, periodSeconds, timeoutSeconds, and failureThreshold.

    Disk Readiness Guard: diskReadiness is an optional, opt-in feature for PVC-backed workspaces. When enabled, it adds an exec probe that verifies the workspace volume is writable and has sufficient free space. If the disk is full or read-only, the pod is marked NotReady (draining it from Service endpoints) to prevent silent failures, without triggering a CrashLoopBackOff (as liveness/startup probes remain unaffected).

  5. Configure additional workspaces for multi-agent setups

    main

    Additional workspaces can be declared to host their own scoped skills. These skills are seeded into ~/.openclaw/workspace-<name>/.

    • ClawHub skills: Installed into the specific workspace's skills/ directory using clawhub --workdir.
    • npm skills: Installed as global binaries in ~/.local/bin and shared by all agents.
    • Skill Packs: Resolve exactly like top-level packs but are tracked in a per-workspace manifest (/data/.skillpack-manifest-ws-<name>).

    Changing a workspace's skills triggers a pod rollout, keyed by the workspace name.

    spec:
      workspace:
        additionalWorkspaces:
          - name: secondary
            skills:
              - "pack:example-org/openclaw-skills/skills/example-skill@example-skill-v1.2.0"
              - "@acme/browser-use"
              - "npm:@acme/cli-tool"
  6. How agent self-configuration works

    main

    OpenClaw agents can autonomously modify their own environment (installing skills, patching config, adding env vars, or seeding workspace files) by interacting with the Kubernetes API.

    To enable this capability, you must explicitly allow it in the OpenClawInstance specification using the selfConfigure field. Once enabled, the agent can create OpenClawSelfConfig resources to request changes. The operator validates every request against an allowlist policy to ensure protected configuration keys are not overwritten and unauthorized actions are denied.

    # 1. Enable self-configure on the instance
    spec:
      selfConfigure:
        enabled: true
        allowedActions: [skills, config, envVars, workspaceFiles]
    
    # 2. The agent creates this to install a skill at runtime
    apiVersion: openclaw.rocks/v1alpha1
    kind: OpenClawSelfConfig
    metadata:
      name: add-fetch-skill
    spec:
      instanceRef: my-agent
      addSkills:
        - "@anthropic/mcp-server-fetch"
  7. Automatic management behaviors

    main

    The OpenClaw operator automatically manages several configurations and settings without requiring explicit user setup:

    BehaviorDetails
    gateway.bindWhen the gateway proxy sidecar is enabled (default), binds to loopback and an nginx reverse proxy handles external access. When disabled (spec.gateway.enabled: false), binds to 0.0.0.0 so the gateway is reachable directly.
    Gateway auth tokenAuto-generated Secret per instance; injected into config and env
    Control UI originsgateway.controlUi.allowedOrigins auto-injected from localhost + ingress hosts + spec.gateway.controlUiOrigins
    OPENCLAW_GATEWAY_HANDSHAKE_TIMEOUT_MS10000 (10s) to work around upstream timeout regression
    OPENCLAW_DISABLE_BONJOUR=1Always set (mDNS does not work in Kubernetes)
    Browser profilesWhen Chromium is enabled, "default" and "chrome" profiles are auto-configured with the sidecar's CDP endpoint
    Tailscale serve configWhen Tailscale is enabled, a tailscale-serve.json key is added to the ConfigMap for the sidecar's TS_SERVE_CONFIG
    Tailscale state persistenceWhen Tailscale is enabled, node identity and TLS certs are persisted to a <instance>-ts-state Secret via TS_KUBE_SECRET
    Config hash rolloutsConfig changes trigger rolling updates via SHA-256 hash annotation
    Config restorationThe init container restores config on every pod restart (overwrite or merge mode)
  8. Server-Side Apply and Field Ownership in OpenClawSelfConfig

    main

    The SelfConfig controller utilizes Kubernetes Server-Side Apply (SSA) with the field manager name openclaw-selfconfig. This allows for granular ownership of specific fields, preventing accidental overwrites of configuration managed by other entities.

    Ownership Granularity

    • Skills (+listType=set): Each skill name is individually owned. Multiple field managers can own different skills on the same instance.
    • Env vars (+listType=map, key: name): Each environment variable is individually owned by the field manager that last set it.
    • Workspace files (map fields): Each file entry under initialFiles is individually owned.
    • Config raw: This is owned atomically as a single field.

    Handling Conflicts

    If a SelfConfig request attempts to remove an item owned by a different field manager, the removal is skipped. In this case, the operator emits a Warning event with the code SelfConfigSkippedRemoval, and the resource status message will include details about the skipped removal.

  9. Use OpenClawClusterDefaults for Global Settings

    main

    The OpenClawClusterDefaults is a cluster-scoped singleton (must be named cluster) used to provide default values that are merged into every OpenClawInstance during reconciliation.

    Use Cases:

    • Setting image registry mirrors for air-gapped environments.
    • Defining shared environment variables (e.g., NPM_CONFIG_REGISTRY, PIP_INDEX_URL).
    • Configuring runtime-dep init containers globally.

    Precedence Rule: Per-instance fields always win over cluster defaults. A default from OpenClawClusterDefaults is only applied if the corresponding field in the OpenClawInstance is unset.

  10. How OpenClawSelfConfig works

    main

    The OpenClawSelfConfig Custom Resource Definition (CRD) allows a running agent to request modifications to its own OpenClawInstance specification. This process is governed by an allowlist policy to ensure security and stability.

    Lifecycle

    1. Creation: An agent creates an OpenClawSelfConfig resource. The status is initially Pending.
    2. Validation: The operator checks the parent OpenClawInstance for:
      • selfConfigure.enabled must be true (if false, status becomes Denied).
      • Requested action categories must be present in allowedActions (if not, status becomes Denied).
      • Protected config keys (e.g., gateway.*) or restricted environment variables are requested (if so, status becomes Failed).
    3. Application: The operator applies the requested changes to the parent instance spec.
    4. Completion: Status transitions to Applied (success) or Failed (error).
    5. Cleanup: An owner reference is set to the parent instance for garbage collection, and terminal requests are automatically deleted after 1 hour.
    apiVersion: openclaw.rocks/v1alpha1
    kind: OpenClawSelfConfig
    metadata:
      name: add-fetch-skill
    spec:
      instanceRef: my-agent
      addSkills:
        - "mcp-server-fetch"
      addEnvVars:
        - name: MY_CUSTOM_VAR
          value: "hello"
  11. Configure Chromium and Ollama capabilities

    main

    OpenClaw supports specialized sidecar/integrated capabilities via the spec:

    Chromium

    Enables headless browser capabilities. You can specify a custom image, resource requirements, and persistence settings.

    Ollama

    Enables local LLM capabilities. You can specify models to load, resource limits, storage size limits, and the number of GPUs to allocate.

    chromium:
      enabled: true
      image:
        repository: chromedp/headless-shell
        tag: "stable"
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: "2"
          memory: 4Gi
      persistence:
        enabled: true
        size: 2Gi
    
    ollama:
      enabled: true
      models:
        - llama3.2
        - nomic-embed-text
      resources:
        requests:
          cpu: "2"
          memory: 4Gi
        limits:
          cpu: "8"
          memory: 16Gi
      storage:
        sizeLimit: 40Gi
      gpu: 1