DOCSEARCHbyAlgolia
Submit Docs

CNDI

repository·main·Indexed 42 days ago

https://github.com/polyseam/cndi

A tool for deploying GitOps-enabled Kubernetes application clusters on AWS, GCP, Azure, or locally. CNDI uses a CLI and YAML templates to provide a wizard-like experience for infrastructure setup, utilizing Terraform for provisioning and ArgoCD for manifest management. Key commands include `cndi create` for project bootstrapping, `cndi overwrite` for generating manifests, and `cndi run` for executing deployments via GitHub Actions.

Tokens
44.1K
Snippets
152
Records
227
Agent score
46%

What's inside CNDI

  1. Configure Airflow DAG storage and GitOps

    main

    During the cndi create process, you can configure how Airflow DAGs are synchronized. This allows you to manage your DAGs in a separate Git repository, enabling a GitOps workflow for your data pipelines.

    Required inputs during setup:

    • Git Username for Airflow DAG Storage: Your GitHub handle.
    • Git Password for Airflow DAG Storage: A personal access token for DAG synchronization.
    • Git Repo for Airflow DAG Storage: The URL of the repository where your Airflow DAGs are stored.
    • Use cluster credentials for Airflows Git Sync: An option to determine if Airflow uses the cluster's credentials to pull DAGs.

  3. Configure CNDI network deployment modes

    main

    CNDI provides two modes for deploying your cluster network via the infrastructure.cndi.network configuration block:

    • create mode: CNDI creates a new Virtual Network (VNet/VPC) and one or more subnets. This is the default mode and is considered the most stable as it does not depend on existing network configurations.
    • insert mode: CNDI inserts your cluster into an existing VNet. Use this mode if your cluster needs to communicate with pre-existing infrastructure located within the same network.

  4. Manage DNS for Load Balancers with ExternalDNS

    main

    If you enable and configure external-dns during the cndi create process, CNDI will automatically manage your DNS records.

    Instead of manually creating CNAME records in your DNS provider (like Route 53), ExternalDNS will detect the Load Balancer's public host and create the necessary CNAME record to point your chosen domain (e.g., airflow.untribe.com) to that host automatically.

    Manual Fallback: If you do not use ExternalDNS, you must:

    1. Locate the public host (CNAME address) of the load balancer in the resource groups output after cndi run completes.
    2. Manually create a CNAME record in your DNS provider (e.g., Google Domains) to route your domain to that public host.

  5. How to use CNDI Blocks in Templates

    main

    CNDI Blocks are reusable components that can be plugged into your Templates. To include a block, use the {{ $cndi.get_block(<identifier>) }} macro.

    When referencing blocks from a remote repository, it is recommended to use the GitHub Raw URL. For stability and to prevent unexpected changes from breaking your configuration, consider using a specific commit SHA instead of a branch name like main in the URL.

    outputs:
  cndi_config:
    cluster_manifests:
      my_ingress: "{{ $cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/common/cluster/default-ingress.yaml) }}"

  6. Use the dev deployment target for local development

    main
    When specifying the dev deployment target provider, cndi prepares your workload to run in development mode. This mode is optimized for rapid iteration and experimentation, typically assuming the workload will run on a single virtualized node or a local cluster.

  7. Use blocks for reusable YAML fragments

    main

    The blocks section allows you to define reusable fragments of YAML content. These blocks can be conditionally inserted into the final project configuration (like cndi_config) based on user responses from prompts.

    Each block has:

    • name: A unique identifier.
    • content: The YAML fragment to be used.

    To include a block in an output file, use the $cndi.get_block(block_name) function. You can wrap this in a condition to decide which block to include.

    blocks:
  - name: git_sync_dedicated
    content:
      apiVersion: v1
      kind: Secret
      metadata:
        name: airflow-git-credentials
      stringData:
        GIT_SYNC_USERNAME: $cndi_on_ow.seal_secret_from_env_var(GIT_SYNC_USERNAME)

outputs:
  cndi_config:
    cluster_manifests:
      git-sync-credentials-secret:
      $cndi.get_block(git_sync_dedicated):
        condition:
          - "{{ $cndi.get_prompt_response(airflow_share_credentials) }}"
          - ==
          - false

  8. Project structure of a CNDI project

    main

    A generated CNDI project follows this structure:

    ├── 📁 cndi
│   ├── 📁 cluster_manifests
│   │   ├── 📁 applications
│   │   │   ├── cnpg.application.yaml
│   │   │   ├── pgadmin.application.yaml
│   │   │   ├── public_nginx.application.yaml
│   │   │   ├── jupyter-notebook-deployment.yaml
│   │   │   └── etc
│   │   ├── argo-ingress.yaml
│   │   ├── jupyter-notebook-ingress.yaml
│   │   ├── cert-manager-cluster-issuer.yaml
│   │   └── etc
│   └── 📁 terraform
│       └── cdk.tf.json
├── cndi_config.yaml
├── .env
├── .gitignore
├── .github
└── README.md

  9. Configure infrastructure using the terraform escape hatch

    main
    The infrastructure.terraform key allows you to deploy resources using Terraform that are not covered by the built-in CNDI modules. This should be used for complex infrastructure requirements that fall outside the standard CNDI scope.

  10. Understand the project structure

    main

    A standard CNDI project structure includes:

    • cndi/terraform/: Contains Terraform Resource files used to provision cloud infrastructure and configure virtual machines.
    • .github/: Contains GitHub Actions workflows that trigger cndi run on pushes to the main branch.
    • .env: Local environment variables for Terraform and Kubernetes configuration (not committed to git).
    • .gitignore: Configured to prevent sensitive files and directories from being committed.
    • cndi_config.yaml: The primary configuration file for the project.

  11. Understand the CNDI project structure

    main

    When you run cndi create, the tool bootstraps a standardized project structure designed to manage infrastructure and software deployment. The core components are:

    • cndi_config.yaml: The central configuration file. It defines the hardware, software, and configuration settings for your entire project.
    • .env: A file for sensitive values and credentials (using dotenv syntax). These are consumed by cndi run for cluster secrets and deployment credentials.
    • cndi/terraform/: Contains Terraform files generated by CNDI that describe the infrastructure to be deployed.
    • cndi/cluster_manifests/: Contains Kubernetes manifests applied by ArgoCD once the cluster is operational.
    • cndi/cluster_manifests/applications/: Contains Kubernetes manifests for core applications (like cert-manager, external-dns, reloader, and nginx-ingress) and individual folders for each Application defined in your cndi_config.yaml.
    • .github/workflows/: A default GitHub Action workflow that wraps the cndi run command. If using a different CI system, you should call the cndi run command directly.
    • README.md: Project documentation explaining the template used and deployment details.
    • .gitignore: Configured to prevent sensitive files (like .env) from being published to your repository.

  12. Manage projects with `cndi overwrite`

    main

    Once a project is initialized, it is managed via two primary files:

    • .env: Contains all secret values (not included in source control).
    • cndi_config.yaml: The central configuration file that unifies infrastructure, applications, and configuration.

    The cndi overwrite (or cndi ow) command transforms these files into machine-readable formats:

    1. It processes cndi_config.yaml and .env.
    2. It generates Kubernetes manifests and Terraform objects.
    3. It outputs the resulting files to the ./cndi folder at the root of your repository.

    cndi_config.yaml Structure

    • metadata: Project name, CNDI version, provider, and distribution.
    • applications: Defines Helm Charts to be installed and configured.
    • cluster_manifests: Used for arbitrary Kubernetes configuration.
    • infrastructure: Defines infrastructure exposed by CNDI and optionally raw Terraform objects.
    cndi ow