DOCSEARCHbyAlgolia
Submit Docs

Kadeck Documentation

website·Indexed 32 days ago

https://docs.kadeck.com

Documentation for Kadeck, a cloud-native platform designed to scale with Apache Kafka infrastructure. Includes guides on KSQL Lab for streaming data analytics, alert systems for infrastructure and topic monitoring, provisioning profiles, and a Public API for managing topics, licenses, and data catalog metadata.

Tokens
18.7K
Snippets
86
Records
167
Agent score
59%

What's inside Kadeck

  1. Overview of Kadeck KSQL Lab

    KSQL Lab provides a data warehouse-like environment for data and analytics teams to interact with streaming data using KSQL. It allows users to perform ad-hoc exploration of streams and tables, prototype stream processing development, conduct data quality checks against live traffic, and build real-time operational BI. A key feature is 'Data ownership transparency,' where Kadeck surfaces the Kafka topics a statement depends on and links them to ownership context (domains and teams) via the 'Data dependencies' section in an active worksheet.

  2. Overview of Kadeck Public API capabilities

    The Kadeck Public API allows for programmatic administration and operational control of the platform, enabling integration into Infrastructure-as-Code (IaC) pipelines and automated workflows. Key capabilities include:

    • Infrastructure Provisioning: Configuring connections to Kafka clusters, Kinesis streams, and Schema Registries, as well as managing Kafka topics.
    • Metadata Management: Organizing and updating metadata for topics and data streams in the Kadeck catalog.
    • Access Control: Automating the management of users, groups, roles, and permissions.
    • License Management: Activating licenses in offline or automated environments via a challenge-response process.
    • System Monitoring: Accessing health endpoints and operational status.

  3. Overview of Topic Data Arrival Alerts

    Topic Data Arrival Alerts monitor Kafka topics and notify users when the total number of records arriving within a rolling time window crosses a configured threshold. These alerts are used for DLQ monitoring, pipeline health checks, and delivery verification. They can detect unexpected spikes (Greater than), unexpected drops or silence (Less than), and confirm expected data delivery.

  4. Overview of Kadeck for Kafka Operations

    Kadeck is a command center for real-time data operations that provides a unified interface to access, govern, and orchestrate streaming architectures across clouds, data centers, and service providers. It integrates directly with Apache Kafka clusters (managed, self-hosted, or API-compatible) and operates in place, meaning it does not extract or replicate data from the streaming infrastructure. The platform is supported by Argus for monitoring and Machina for distributed computation.

  5. Understand Kadeck Service and Domain concepts

    Kadeck uses a hierarchical structure to manage resources based on Data Mesh principles:

    • Service: A collection of one or more applications (producers/consumers) operating in a single environment (e.g., Dev, QA, Prod). It encompasses related resources such as topics, schemas, ACLs, and consumer groups. Each Service is managed by designated Service Owners.
    • Domain: A logical grouping of Services and topics. Domain Owners are responsible for managing resources and ownership within the Domain.
    • Provisioning Profile: Technical constraints defined by the Platform Team for a specific environment within a domain. These profiles enforce rules on topic configuration (retention, replication), allowed schema types (e.g., Avro), permitted ACL actions, and resource naming conventions.

  6. Understand Kadeck's architecture and role

    Kadeck is a cloud-native control plane for distributed real-time data systems, designed to scale from a single-container deployment to a fully clustered configuration. It provides a web-based interface to unify access, governance, and operations across multiple Apache Kafka clusters, Schema Registries, and Connect clusters. Kadeck runs entirely within the user's controlled infrastructure (private cloud, on-premise, or public cloud containers) and can manage multiple separate data streaming environments from a single unified platform.

  7. Configure Kadeck Master Nodes

    Master nodes form the backbone of the Kadeck cluster. It is recommended to use at least three master nodes to tolerate up to two simultaneous failures. All master nodes must be configured to know about each other via a shared list of nodes. Hostnames must resolve correctly within the network or Kubernetes cluster.
    # Example for Kadeck-1
XEOTEK_KADECK_CLUSTER_NODES=kadeck-1:2551,kadeck-2:2552,kadeck-3:2553
XEOTEK_KADECK_CLUSTER_HOSTNAME=kadeck-1
XEOTEK_KADECK_CLUSTER_PORT=2551

# Example for Kadeck-2
XEOTEK_KADECK_CLUSTER_NODES=kadeck-1:2551,kadeck-2:2552,kadeck-3:2553
XEOTEK_KADECK_CLUSTER_HOSTNAME=kadeck-2
XEOTEK_KADECK_CLUSTER_PORT=2552

# Example for Kadeck-3
XEOTEK_KADECK_CLUSTER_NODES=kadeck-1:2551,kadeck-2:2552,kadeck-3:2553
XEOTEK_KADECK_CLUSTER_HOSTNAME=kadeck-3
XEOTEK_KADECK_CLUSTER_PORT=2553

  9. Configure Kadeck alert notification architecture

    Kadeck uses a two-step integration-based architecture for alerts:

    1. Integration Configuration: Administrators define notification channels (e.g., Slack, Microsoft Teams, PagerDuty, email) via the Kadeck Administration interface.
    2. Alert Configuration: Individual alerts are mapped to one or more of these integrations. This allows for targeted routing to specific teams, redundancy for critical alerts, and notification strategies based on alert severity and type.

  10. Understand ksqlDB Core Concepts: Streams and Tables

    ksqlDB uses two primary abstractions to handle data from Apache Kafka:

    • Stream: An append-only, unbounded sequence of events. It acts like a transaction log or event ledger. Typical use cases include orders, payments, and IoT telemetry.
    • Table: A materialized view of the current state per key. It behaves like a traditional database table. Typical use cases include customer profiles and product catalogs.

  11. Kadeck 5.1.0 Platform Features

    Kadeck 5.1.0 introduced the following core features:

    • Alert Notifications: Integrations with Slack, Microsoft Teams, PagerDuty, and Email.
    • Incident Tracking: New system for tracking and resolving Kafka incidents.
    • Health Assistant: Enhanced detection for Kafka-related issues.
    • Data Catalog: New columns for record count and retention settings.
    • Security: Implementation of CORS, CSP, CSRF, and brute force mitigation.

  12. Understand Kadeck Provisioning Profiles

    Provisioning Profiles are centrally managed templates created by the platform team to define what resources (such as topics, ACLs, and users) a team or domain is allowed to provision within Kadeck. They act as guardrails for initializing streaming services and infrastructure, supporting both self-service and governed provisioning.

    Crucially, Provisioning Profiles define what can be provisioned, but they do not automatically create the resources; the user or service owner must still choose to create the available resources. Additionally, if a user lacks explicit security permissions to create a resource, the action will be blocked even if the Provisioning Profile allows it.