Kafbat UI Documentation

repository·main·Indexed 25 days ago

https://github.com/kafbat/kafka-ui

A versatile, fast, and lightweight web interface for monitoring and managing Apache Kafka clusters. It provides visibility into topics, messages, consumer groups, and schema registries, with support for Avro, JSON Schema, and Protobuf. Features include pluggable authentication (OAuth 2.0, LDAP, Basic Auth), RBAC, data masking, and support for managed services like AWS MSK, Azure EventHub, and Google Cloud Managed Service for Apache Kafka.

Tokens
4K
Snippets
10
Records
20
Agent score
83%

What's inside Kafbat UI

  1. Overview of Kafbat UI features

    main

    Kafbat UI is a web interface for managing Apache Kafka clusters with the following capabilities:

    • Topic Management: Create, configure, and view topic details (partitions, replication, settings).
    • Message Browser: Explore messages in JSON, plain text, or Avro. Supports live view and CEL message filters.
    • Consumer Groups: Analyze offsets and monitor lag (combined and partition-specific).
    • Schema Registry: Support for Avro, JSON Schema, and Protobuf.
    • Security: Pluggable authentication (OAuth 2.0, LDAP, Basic Auth), Cloud IAM (GCP, Azure, AWS), and Role-Based Access Control (RBAC).
    • Managed Services: Support for Azure EventHub, Google Cloud Managed Service for Apache Kafka, and AWS MSK (server-based and serverless).
    • Data Privacy: Data masking to obfuscate sensitive information in topic messages.
  2. Install and set up Kafbat UI frontend

    main

    To set up the Kafbat UI frontend for development, ensure you have docker and nvm (with the Node.js version specified in .nvmrc) installed. Follow these steps:

    1. Navigate to the frontend directory:
      cd ./frontend
    2. Install pnpm globally:
      npm install -g pnpm
      Note: If you need a specific version, use npm install -g pnpm@<version>.
    3. Install project dependencies:
      pnpm install
    4. Generate API clients from the OpenAPI document:
      pnpm gen:sources
    cd ./frontend
    npm install -g pnpm
    pnpm install
    pnpm gen:sources
  3. Run Kafbat UI using Docker Compose

    main

    You can run Kafbat UI and your Kafka clusters using Docker Compose. This command must be executed from the root directory of the repository.

    docker-compose -f ./documentation/compose/kafbat-ui.yaml up

    Important: Ensure that none of your .env* files contain the DEV_PROXY variable when running via Docker.

    docker-compose -f ./documentation/compose/kafbat-ui.yaml up
  4. Run the Subject Strategy Demo locally

    main

    To run the Subject Strategy Demo, you must first build the API with the frontend included, then use Docker Compose to launch the environment.

    1. Build the API: From the repository root, run the Gradle build command with the frontend flag enabled.
    2. Launch via Docker Compose: Navigate to the demo directory and start the containers in detached mode.
    3. Access the UI: Open your browser to http://localhost:8080/ to interact with message production and reproduction.
  5. Explore Docker Compose configurations for Kafka UI

    main

    Kafka UI provides several pre-configured docker-compose.yaml files to support different deployment scenarios, including security protocols, authentication methods, and proxy integrations. Use these files as templates for your own Docker Compose setups.

    Available configurations include:

    • Default Setup: kafka-ui.yaml (includes 2 Kafka clusters, 2 Schema Registry nodes, 1 Kafka Connect, and dummy topics).
    • Security & Authentication:
      • kafka-ssl.yml: Connect to Kafka via TLS/SSL.
      • cluster-sr-auth.yaml: Schema Registry with authentication.
      • auth-context.yaml: Basic (username/password) authentication with a custom URL path.
      • ui-sasl.yaml: SASL authentication for Kafka.
      • ui-jmx-secured.yml: Kafka's JMX with SSL and authentication.
    • Proxy & Networking:
      • nginx-proxy.yaml: Running the app behind a reverse proxy like Nginx.
      • traefik-proxy.yaml: Traefik-specific proxy configuration.
    • Monitoring & Observability:
      • ui-with-jmx-exporter.yaml: 2 Kafka clusters with Prometheus JMX exporters enabled.
    • Other Scenarios:
      • e2e-tests.yaml: Includes various connectors (github-source, s3, sink-activities, source-activities) and KSQL functionality.
      • kafka-zookeeper.yaml: Kafka using ZooKeeper instead of KRaft.
  6. Persistent installation using Docker Compose

    main

    For a permanent installation, use Docker Compose and mount a configuration file to ensure settings persist across container restarts. The following example uses a volume to map a local config.yml to the container's dynamic configuration path.

    services:
      kafbat-ui:
        container_name: kafbat-ui
        image: ghcr.io/kafbat/kafka-ui:latest
        ports:
          - 8080:8080
        environment:
          DYNAMIC_CONFIG_ENABLED: 'true'
          SWAGGER_UI_ENABLED: 'true'
        volumes:
          - ~/kui/config.yml:/etc/kafkaui/dynamic_config.yaml
  7. Quick start with Docker (Demo run)

    main

    To quickly try out Kafbat UI without a persistent setup, run the following Docker command. This enables dynamic configuration and the Swagger UI API documentation.

    Access the web interface at http://localhost:8080 after the container starts.

    docker run -it -p 8080:8080 -e DYNAMIC_CONFIG_ENABLED=true -e SWAGGER_UI_ENABLED=true ghcr.io/kafbat/kafka-ui
  8. Set up kafbat-e2e-playwright for local execution

    main

    To run end-to-end UI tests locally, you need Node.js (>= 18) and a running instance of Kafbat. Follow these steps:

    1. Start Kafbat: Use the provided docker compose file to spin up the environment: docker compose -f ./documentation/compose/e2e-tests.yaml up -d
    2. Install dependencies: Run npm install.
    3. Install Playwright browsers: Run npx playwright install.

    The test suite uses Playwright, Cucumber.js, and TypeScript.

    # Start Kafbat environment
    docker compose -f ./documentation/compose/e2e-tests.yaml up -d
    
    # Install dependencies and browsers
    npm install
    npx playwright install
  9. Run kafbat-e2e-playwright tests

    main

    Use the following npm scripts to execute the test suite in different modes:

    • Normal Test Run: Executes the staged tests.
    • Debug Mode: Runs tests with the Playwright Inspector enabled for debugging.
    • Rerun Failed Tests: Automatically reruns only the tests that failed in the previous run.
  10. Run Kafbat UI in development mode with API proxying

    main

    To run the development server with API proxying, you must configure the proxy target in a .env.local file.

    1. Create or update .env.local in the frontend directory:
      VITE_DEV_PROXY=https://api.server # replace with your actual API server URL
    2. Start the development server:
      pnpm dev