Jenkins Kubernetes Plugin

repository·master·Indexed 25 days ago

https://github.com/jenkinsci/kubernetes-plugin

Automates the scaling of Jenkins agents by running them as dynamic pods within a Kubernetes cluster (version 1.14 or later). It provides capabilities for configuring Kubernetes Pod Templates, using the `podTemplate` pipeline step for ephemeral pods, managing sidecar containers, and implementing garbage collection for orphaned pods. Supports various authentication methods including kubeconfig secret files, tokens for OpenShift, and Google Service Accounts for GKE.

Tokens
14.9K
Snippets
33
Records
75
Agent score
79%

What's inside kubernetes-plugin

  1. Overview of the Jenkins Kubernetes Plugin

    master

    The Jenkins Kubernetes plugin automates the scaling of Jenkins agents running in Kubernetes. Based on the 'Scaling Docker with Kubernetes' concept, it creates a Kubernetes Pod containing a Docker image when a node starts and stops the Pod when the build completes.

    Key features:

    • Supports Jenkins controllers running both inside and outside of Kubernetes.
    • Automatically injects environment variables into the agent for connectivity.
    • Uses JNLP to start nodes and connect them to the Jenkins controller.
  2. How Pod Template Inheritance works

    master

    Pod templates can inherit from existing templates using the inheritFrom field.

    Inheritance Rules:

    • Simple values (strings): The child template overrides the parent.
    • Complex values (lists, maps): The child template merges with the parent. The yamlMergeStrategy in the child controls this behavior.
    • Service account & Node selector: These are completely substituted by the child.
    • Container templates & Volumes: If a child defines a container or volume with the same name as one in the parent, it inherits/overrides the parent's configuration. If the name is new, it is added.
    • Image Pull Secrets: These are combined (all secrets from both parent and child are used).
    • Multiple Inheritance: inheritFrom can accept multiple templates separated by spaces. They are processed in order, with later items overriding earlier ones. If a template is not found, it is ignored.
    podTemplate(inheritFrom: 'mypod', containers: [
        containerTemplate(name: 'maven', image: 'maven:3.9.9-eclipse-temurin-21')
      ]) {
      node(POD_LABEL) {
        …
      }
    }
  3. Compose Pod Templates via Nesting

    master

    In scripted pipelines, you can nest podTemplate blocks to compose complex pods. Each level of nesting adds to the pod definition. This is particularly useful for creating reusable pipeline library functions that wrap specific capabilities (e.g., a dockerTemplate and a mavenTemplate).

    Note: When nesting, you must include a node statement in the innermost block to execute steps. The POD_LABEL in the innermost block refers to the generated label for the fully composed pod.

    podTemplate(containers: [containerTemplate(image: 'docker', name: 'docker', command: 'sleep', args: '99d')]) {
        podTemplate(containers: [containerTemplate(image: 'maven', name: 'maven', command: 'sleep', args: '99d')]) {
          node(POD_LABEL) {
            // gets a pod with both docker and maven
            …
          }
        }
    }
  4. How Kubernetes agents and pod templates work

    master

    The Kubernetes plugin allocates Jenkins agents within Kubernetes pods.

    Core Concepts

    • The Agent Container: Every pod contains at least one special container running the Jenkins agent. While it defaults to the name jnlp, you can specify a different name in the pod template.
    • Sidecar Containers: You can run arbitrary processes in other containers within the same pod. In Pipeline jobs, you can execute commands in these specific containers using the container step.
    • Label-based Allocation: When you define a Pod Template in the Jenkins UI, you assign it a label. When a job (Freestyle or Pipeline) requests a node with that specific label (e.g., node('some-label')), the Kubernetes Cloud automatically allocates a new pod to run that build.

    Best Practice: While global pod templates are useful for migrating existing freestyle projects, new users should prefer using the podTemplate step within Pipelines for better control.

  5. Enable Garbage Collection for orphaned pods

    master

    In some cases, agent pods may be left running in the Kubernetes cluster without a corresponding Jenkins agent in the controller. These orphaned pods will continuously attempt to reconnect.

    The plugin includes a garbage collection mechanism to clean up these pods. Note that this feature is disabled by default because it can generate additional load on the Kubernetes API server.

  6. Important constraints for Kubernetes containers

    master

    When defining pods in the Kubernetes plugin, keep these constraints in mind:

    1. The jnlp container: Jenkins automatically creates a container named jnlp to serve as the Jenkins JNLP agent. It uses parameters ${computer.jnlpmac} ${computer.name}.
    2. Custom jnlp images: If you want to provide your own Docker image for the JNLP agent, you must name the container jnlp. If you use a different name, Jenkins will attempt to connect two different nodes to the master simultaneously, causing conflicts.
    3. Process persistence: All other containers must run a continuous process. If the default entrypoint or command exits immediately, you should override it using cat and ttyEnabled: true to keep the container running.
  7. How to nest Pod Templates for complex compositions

    master

    You can nest podTemplate blocks within each other to compose complex environments. This is particularly useful when creating reusable Pipeline Library functions.

    Crucial Requirement: When nesting, you must use the label generated by the innermost podTemplate to call the node step. This ensures the agent runs on a pod that contains all the combined containers from the outer templates.

    // Example of nesting templates in a Pipeline Library
    package com.foo.utils
    
    public void dockerTemplate(body) {
      def label = "worker-${UUID.randomUUID().toString()}"
      podTemplate(label: label,
            containers: [containerTemplate(name: 'docker', image: 'docker', command: 'cat', ttyEnabled: true)],
            volumes: [hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')]) {
        body.call(label)
      }
    }
    
    public void mavenTemplate(body) {
      def label = "worker-${UUID.randomUUID().toString()}"
      podTemplate(label: label,
            containers: [containerTemplate(name: 'maven', image: 'maven', command: 'cat', ttyEnabled: true)],
            volumes: [secretVolume(secretName: 'maven-settings', mountPath: '/root/.m2'),
                      persistentVolumeClaim(claimName: 'maven-local-repo', mountPath: '/root/.m2nrepo')]) {
        body.call(label)
      }
    }
    
    return this
    
    // --- Usage in a Pipeline ---
    
    import com.foo.utils
    
    slaveTemplates = new PodTemplates()
    
    slaveTemplates.dockerTemplate {
      slaveTemplates.mavenTemplate { label ->
        node(label) {
          container('docker') {
            sh 'echo from docker'
          }
          container('maven') {
            sh 'echo from maven'
          }
         }
      }
    }
  8. Configure Pod Template features

    master

    The Kubernetes plugin supports several advanced Pod Template configurations:

    • Volumes: Supports PersistentVolumeClaims (PVC), NFS volumes, secrets, emptyDir, and hostPath volumes.
    • Resource Management: Define resource requests and limits at the container level.
    • Scheduling: Use nodeSelector to constrain pods to specific nodes.
    • Security:
      • Define image pull secrets for the pod template.
      • Set a serviceAccount when creating new pods.
      • Enable pseudo-TTY at the container level.
      • Configure image pull policy via checkbox.
    • Inheritance: Templates support nesting via inheritFrom to allow for template inheritance of containers and volumes.
  9. Use multiple containers in a single Pod (Container Groups)

    master

    You can define multiple containers within a single agent pod using the containers parameter in podTemplate. Containers in the same pod share resources like mount points, and each container's ports are accessible via localhost.

    Note: The container directive for executing commands in specific containers is currently in ALPHA status and may have issues with concurrency and pipeline recovery.

    def label = "mypod-${UUID.randomUUID().toString()}"
    podTemplate(label: label, containers: [
        containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
        containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')
      ]) {
    
        node(label) {
            stage('Get a Maven project') {
                git 'https://github.com/jenkinsci/kubernetes-plugin.git'
                container('maven') {
                    stage('Build a Maven project') {
                        sh 'mvn -B clean install'
                    }
                }
            }
    
            stage('Get a Golang project') {
                git url: 'https://github.com/hashicorp/terraform.git'
                container('golang') {
                    stage('Build a Go project') {
                        sh """
                        mkdir -p /go/src/github.com/hashicorp
                        ln -s `pwd` /go/src/github.com/hashicorp/terraform
                        cd /go/src/github.com/hashicorp/terraform && make core-dev
                        """
                    }
                }
            }
    
        }
    }