KOTS (Kubernetes Off-The-Shelf)

Replicated KOTS (Kubernetes Off-The-Shelf) is a suite of tools for distributing and managing Kubernetes applications. It consists of two primary components:

1. Kots CLI: A kubectl plugin used as a client-side binary to configure and build dynamic Kubernetes manifests. It also acts as the bootstrapper for the in-cluster Admin Console.
2. kotsadm (Admin Console): An in-cluster application that automates core Kots CLI tasks, including license verification, configuration, updates, image renaming, version controlling changes, and deployment. It also handles preflight checks and support bundle analysis.

The kotsstore is a backing store implementation that manages application archives, support bundles, and metadata. It utilizes a hybrid storage strategy:

• S3: Used for storing application archives and support bundles.
• rqlite: Used for storing all metadata and cache.
• Kubernetes Cluster: Used for storing sensitive information (such as gitops data) and for performance optimization by storing certain data locally in the cluster.

Note: The architecture is progressively migrating away from S3 and PG towards Kubernetes-native storage components.

KOTS requires an object store to persist deployable application versions. When a new upstream version is detected (via polling or airgap bundle upload), kotsadm performs the following lifecycle:

1. Templating: The new upstream version is templated using existing user-supplied Config, Application, and License state (stored in upstream/userdata) and any custom kustomize changes (stored in overlays/).
2. Bundling: The resulting upstream, base, and overlays directories are bundled into a tarball.
3. Storage: The bundle is pushed to a specific key in the object store.

These stored versions are used for:

• Deployment: Deploying a specific processed version of the application.
• Rollback: Rolling back to previous versions (if allowRollback is enabled in the application specification).

The lifecycle of conditional status informers follows these steps:

1. Discovery: During the application deployment loop in TypeScript, the list of statusInformers is retrieved from the Application spec.
2. Rendering: The statusInformers array is processed using the RenderFile function from the KOTS FFI. This allows template functions to resolve the array entries.
3. Communication: The rendered list is sent via a socket to the KOTS operator.
4. Application: The operator evaluates the entries. If a template function resolves to an empty string "" or an invalid entry, the operator excludes it from the active informers.

By default, KOTS uses MinIO as a standalone object store for application archives and support bundles. You can disable MinIO usage depending on your installation type:

• Standard KOTS Install/Upgrade: Use the --with-minio=false flag during installation or upgrade.
• Embedded Cluster: Use the disableS3 option within the KOTS add-on.

Note that removing MinIO changes how application archives and support bundles are stored.

To enable future detection of deployed application versions in GitOps workflows, KOTS uses 'tracers' in the form of Kubernetes annotations. These tracers are injected into application pods via Kustomize during the manifest rewriting process. This allows the KOTS Admin Console to eventually identify which specific application slug and sequence are currently running in a cluster managed by GitOps.

When KOTS performs a pull or rewrite operation, it injects specific annotations into the kustomization.yaml file, which are then applied to all downstream manifests via kustomize build.

commonAnnotations:
  kots.io/app-slug: <app-slug>
  kots.io/app-sequence: <app-sequence>

The kotsstore is designed with specific operational constraints regarding how it interacts with Kubernetes objects:

• Predictable Resource Usage: The store uses a fixed number of ConfigMaps and Secrets per application. The number of objects does not scale with the number of application versions, uptime, or other metrics not controlled by the user.
• Stable Object Count: Activity on an application must not increase the total number of objects stored in the cluster.
• Concurrency Safety: The store is designed to be safe for multiple replicas to perform simultaneous read and write operations on the objects.
• Data Separation: Sensitive data is strictly stored in Kubernetes Secrets, while non-sensitive data is stored in ConfigMaps.
• Minimal Ephemeral Storage: The use of ephemeral storage within the pod is limited and discouraged.

Variadic (or dynamic) configuration allows vendors to define resources that can be created multiple times or extended with an unknown number of properties at install time. This solves common use cases such as:

• Template Resources: Creating multiple instances of a resource (e.g., deploying multiple Kafka instances) where the count is determined by the user during installation.
• Extending Resources: Adding an arbitrary number of configuration properties to an existing resource (e.g., mounting $N$ files to a pod or adding $N$ environment variables).

KOTS implements this through two new schema capabilities: repeatable Config Items and repeatable Config Groups.

A repeatable Config Item allows a vendor to define a field that accepts an array of values instead of a single scalar value. This is used to extend existing resources with multiple configuration properties.

To implement this, vendors add a repeatable attribute to the Config Item schema. The item can also include a template property, which specifies the YAML document or sub-document that should be used/copied for each element in the array. The underlying value types (integer, string, boolean, etc.) remain the same, but the field now expects an array of those types.