DOCSEARCHbyAlgolia
Submit Docs

DevOps Portfolio Infrastructure Documentation

website·Indexed 31 days ago

https://el-jefe.me

Technical documentation and blog for a GitOps-driven homelab infrastructure. Covers a 3-node K3s cluster featuring ArgoCD, Helm, Traefik, and a comprehensive monitoring stack with Prometheus, Grafana, Mimir, and Loki. Includes details on a self-hosted AI inference stack using NVIDIA Triton, Langfuse, and a Node.js/Express AI Gateway, as well as CI/CD pipelines via GitHub Actions and multi-stage Docker builds.

Tokens
26.5K
Snippets
139
Records
196
Agent score
51%

What's inside DevOps Portfolio

  1. Overview of the Logging Pipeline architecture

    The logging pipeline collects Pod logs cluster-wide using Alloy (deployed as a DaemonSet), which aggregates and forwards them to Loki (running in single-binary mode). MinIO is used as the S3-compatible object storage backend for log chunks.

  2. Overview of the AI Inference Stack Architecture

    The platform utilizes a self-hosted AI inference stack designed to provide embedding generation, text generation, and LLM observability across all portfolio applications. The architecture follows a shared infrastructure model where a single gateway serves all apps to avoid duplicating AI logic.

  3. Overview of the Shared AI Gateway

    The Shared AI Gateway is a Node.js/Express API gateway (Runtime: Node.js 20 Alpine) that abstracts multiple LLM backends behind a single API. It provides intelligent fallback, Redis caching, Kafka event streaming, and Prometheus metrics. It runs on port 8002 and is distributed as the Docker image maxjeffwell/shared-ai-gateway.

  4. Overview of the multi-layer backup architecture

    The platform implements a defense-in-depth backup strategy using four primary layers, all targeting Backblaze B2 for off-cluster storage:

    1. Velero: Kubernetes-native volume snapshots.
    2. Database CronJobs: Application-consistent logical dumps.
    3. Cold Storage: Backups of DNS records and secrets.
    4. CloudCasa: A SaaS-managed secondary disaster recovery layer.

  5. Implement GitOps deployment with ArgoCD

    The platform uses a GitOps-driven architecture where all infrastructure and application state is declared in the devops-portfolio-manager repository and reconciled by ArgoCD. Application repositories are restricted to source code and CI workflows and do not interact directly with the cluster. This ensures a full audit trail via Git commits, allows rollbacks using git revert, and maintains Git as the single source of truth for cluster state.

  6. Map Doppler secrets to Kubernetes Secrets using ExternalSecret

    Use an ExternalSecret resource to map specific keys from Doppler to a target Kubernetes Secret. The refreshInterval determines how often ESO polls Doppler for updates (default in this architecture is 1 minute).
    apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: bookmarked-secrets
  namespace: default
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: doppler-secret-store
    kind: ClusterSecretStore
  target:
    name: bookmarked-secrets
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: DATABASE_URL
    - secretKey: JWT_SECRET
      remoteRef:
        key: JWT_SECRET

  7. Manage Kubernetes secrets using Doppler and External Secrets Operator

    The architecture uses Doppler as the single source of truth for all sensitive configuration. Secrets are managed via the Doppler dashboard or CLI and are synced into the Kubernetes cluster using the External Secrets Operator (ESO). This prevents secrets from being stored directly in Kubernetes manifests or created manually via kubectl create secret.

  8. Langfuse Architecture Overview

    Langfuse is an open-source LLM observability platform that provides tracing, analytics, and prompt management. Its architecture consists of the following components:

    • Web UI: Dashboard for visualizing traces and analytics.
    • API: Ingestion endpoint for trace data.
    • ClickHouse: Analytics database used for fast aggregations.
    • PostgreSQL: Handles metadata and user management.
    • S3-compatible storage: Used for storing large payloads (e.g., MinIO).

  9. AI Design Principles for Cost and Reliability

    The AI architecture is governed by the following design principles:

    • Cost Optimization: Prioritize free or low-cost backends (e.g., HuggingFace, Groq) and use expensive providers (e.g., Anthropic, RunPod) only when necessary.
    • Reliability: Implement multi-tier fallbacks to ensure AI features remain available.
    • Observability: Ensure every LLM call is traced via LiteLLM through Langfuse.
    • Self-hosting: Reduce external dependencies by hosting Triton on local GPUs and llama.cpp on VPS.
    • Shared Infrastructure: Use a centralized gateway to serve all portfolio applications.

  10. Verify backup integrity with restore tests

    The architecture includes automated restore tests to validate backup health and data integrity. These tests run on a weekly or monthly basis.
    | CronJob | Schedule | Test Action |
| :--- | :--- | :--- |
| velero-backup-verify | Weekly Sun | BSL health, backup age, test restore of default namespace |
| postgresql-restore-test | 1st Sun monthly | Download + pg_restore of dumps |
| mongodb-restore-test | 1st Sun monthly | Download + mongorestore of all 5 DBs |
| dns-zone-restore-test | 1st Sun monthly | Download, extract tarball, validate JSON vs Cloudflare |
| doppler-secrets-restore-test | 1st Sun monthly | Download, AES-256 decrypt, validate JSON vs Doppler |

  11. Understand the K3s cluster node architecture

    The platform is deployed on a 3-node K3s cluster (a lightweight, certified Kubernetes distribution). The architecture consists of one server node acting as the control plane and two worker nodes.

  12. Grafana Mimir Architecture Overview

    Grafana Mimir provides long-term metric storage with a 90-day retention period. In this architecture, Prometheus writes metrics to Mimir via remote write, and Grafana queries Mimir as a Prometheus-compatible data source. The data flow is as follows: Prometheus → Distributor → Ingester (local TSDB blocks) → Compactor (merges/deduplicates and uploads to MinIO) → Store Gateway (serves historical blocks from MinIO) → Querier (aggregates recent and historical data) → Query Frontend → Grafana.