Kubenix

repository·main·Indexed 19 days ago

https://github.com/hall/kubenix

A tool for generating Kubernetes manifests using Nix, designed for compatibility with standard Kubernetes tooling and GitOps workflows. It supports defining Custom Resource Definitions (CRDs), managing Helm releases with patching capabilities, and building custom container images via dockerTools. Kubenix allows for the creation of resources across multiple namespaces using submodules and provides mechanisms to integrate with ArgoCD, Flux, or k3s auto-deploying manifests.

Tokens
3K
Snippets
14
Records
17
Agent score
68%

What's inside kubenix

  1. Define Custom Resource Definitions (CRDs) with kubernetes.customTypes

    main

    You can register Custom Resource Definitions (CRDs) in Kubenix to enable type-safe definitions of custom resources using Nix options. To do this, add an entry to the kubernetes.customTypes list. Each entry must specify the group, version, kind, and a Nix module that defines the schema for that resource.

    # Conceptual structure for kubernetes.customTypes
    kubernetes.customTypes = [
      {
        group = "stable.example.com";
        version = "v1";
        kind = "CronTab";
        module = ./path/to/your/schema.nix;
      }
    ];
  2. Resolve secrets at runtime within the cluster

    main

    For a more robust security model, resolve secrets from within the Kubernetes cluster itself. This can be achieved in two ways:

    1. Reference external sources: Use a controller running inside the cluster (such as external-secrets) to fetch secrets from an external provider.
    2. Decrypt inline secrets: Use a controller or tool within the cluster to decrypt secrets that were included in the manifest (such as sealed-secrets or using sops with external keys).
  3. Create resources in multiple namespaces using kubenix submodules

    main

    Kubenix uses a submodule system (built on top of the NixOS submodule system) to allow for the creation of resources across multiple Kubernetes namespaces. By defining a submodule, you can encapsulate resource definitions and then instantiate them multiple times with different namespace parameters.

    To implement this pattern:

    1. Define a submodule using Nix syntax to specify the required parameters (like namespace).
    2. Use the submodule within a main module to instantiate resources for specific namespaces.
    # Example pattern for a submodule definition
    # (Referenced in the documentation via namespaced.nix)
    
    # Example pattern for usage
    # (Referenced in the documentation via module.nix)
  4. Automate Kubenix deployments with k3s auto-deploying manifests

    main

    You can leverage k3s's ability to automatically deploy manifests located in /var/lib/rancher/k3s/server/manifests. By using Nix, you can write the resultYAML from a Kubenix evaluation to a file in /etc and then create a symbolic link from that file into the k3s manifests directory.

    Important Considerations:

    • Security: This method writes all manifests to the Nix store, making it unsuitable for manifests containing inline sensitive data.
    • Resource Cleanup: k3s will not automatically delete Kubernetes resources if the corresponding files are removed from the manifests directory.
    {
      # let's write `resultYAML` to an arbitrary file under `/etc`
      environment.etc."kubenix.yaml".source = 
      (kubenix.evalModules.x86_64-linux {
        module = { kubenix, ... }: {
          imports = [ kubenix.modules.k8s ];
          kubernetes.resources.pods.example.spec.containers.example.image = "nginx";
        };
      }).config.kubernetes.resultYAML;
    
      # now we can link our file into the appropriate directory
      # and k3s will handle the rest
      system.activationScripts.kubenix.text = ''
        mkdir -p /var/lib/rancher/k3s/server/manifests
        ln -sf /etc/kubenix.yaml /var/lib/rancher/k3s/server/manifests/kubenix.yaml
      ';
    }
  5. Instantiate Custom Resources using kubernetes.resources

    main

    After defining a custom type in kubernetes.customTypes, you can instantiate specific resources by adding them to the kubernetes.resources.<attrName> attribute set, where <attrName> matches the name used during definition.

    For example, if you define a type with the name crontabs, you would define your resource instances under kubernetes.resources.crontabs. The resulting Kubernetes manifest will include the apiVersion, kind, metadata, and the spec defined by your Nix schema.

    {
      "apiVersion": "stable.example.com/v1",
      "kind": "CronTab",
      "metadata": {
        "name": "my-new-cron-object"
      },
      "spec": {
        "cronSpec": "* * * * */5",
        "image": "my-awesome-cron-image",
        "replicas": 1
      }
    }
  6. Patch Helm chart resources using kubernetes.resources

    main

    Instead of relying solely on values.yaml for templating, Kubenix allows you to patch resources generated by a Helm release by merging configuration during evaluation.

    To patch a resource (like a Deployment) created by a Helm chart:

    1. Identify the resource type and name used by the chart.
    2. Define a resource in your configuration using kubernetes.resources.<type>.<name>.
    3. Ensure you match the metadata.namespace of the original resource.
    4. Apply your desired changes to the spec or other fields. This bypasses the limitations of the chart's values.yaml.
    {
      # define a resource with the same name as the one in the Helm chart
      kubernetes.resources.deployments.nginx = {
        # be sure to match the corresponding namespace as well
        metadata.namespace = "default";
        # configure anything directly, bypassing values.yaml constraints
        spec.template.spec.containers.nginx.env = [{
          name = "MY_VARIABLE";
          value = "100";
        }];
      };
    }
  7. Inject secrets at deploy time using vals

    main

    The simplest method is to inject secrets after manifests are generated but before they are applied to the cluster. You can pipe your generated manifests through vals to resolve secret values (e.g., using the file provider) before passing them to kubectl.

    pkgs.writeShellScript "apply" ''
      cat manifest.json | ${pkgs.vals}/bin/vals eval | ${pkgs.kubectl}/bin/kubectl -f -
      ''
  8. Deploy Kubenix using GitOps (ArgoCD or Flux)

    main
    Kubenix is compatible with any tooling that can ingest standard Kubernetes manifests. For GitOps workflows, you can use tools like ArgoCD or Flux to pull the manifests generated by Kubenix from a git repository. This allows you to use Git as the source of record, perform easy rollbacks, and manage resource pruning. Once your GitOps tool is configured to watch the repository where Kubenix outputs its manifests, deployments are as simple as committing new manifests to Git.
  9. Define a Helm release with kubernetes.helm.releases

    main

    To define a Helm release in Kubenix, use the kubernetes.helm.releases option. This allows you to fetch and render Helm charts as part of your configuration evaluation, similar to how plain Kubernetes manifests are handled.

    # Example of defining a Helm release
    kubernetes.helm.releases = { ... };