kafkactl Documentation

kafkactl supports several strategies for supplying sensitive credentials (SASL, TLS passphrases, Schema Registry).

1. Interactive Prompting

If a credential is missing from the config/environment, kafkactl will prompt you in the terminal.

2. OS Keyring (Recommended)

By default, kafkactl stores successfully entered passwords in your OS keyring (GNOME Keyring, macOS Keychain, or Windows Credential Manager). Subsequent commands will retrieve them automatically.

• Disable Keyring: Set keyring.enabled: false in config.yml.
• Clear Credentials: Use the --clear-keyring flag to delete a stored entry and force a re-prompt.

  kafkactl get topics --clear-keyring

3. Configuration File

You can store credentials in plaintext in config.yml. Not recommended for production.

4. Kubernetes Secrets

When running in Kubernetes mode, use the saslSecret configuration to avoid passing credentials as plaintext environment variables in the pod spec.

keyring:
  enabled: true

If a schemaRegistry is configured in your context, kafkactl automatically detects and handles Avro and JSON schema encoded topics.

Avro Support

• Production: kafkactl looks up the topic in the registry. If you need to produce using a specific version, use --key-schema-version or --value-schema-version.
• Consumption: Automatically decodes messages. Use --print-schema to see the schema used.

JSON Schema Support

• Production: Input JSON is validated against the registered schema before being produced using the Confluent wire format.
• Consumption: Automatically decodes messages. Use --print-schema to see the schema used.

Configuration Example

contexts:
  local:
    schemaRegistry:
      url: localhost:8081

kafkactl produce avro_topic --value {"next":{"next":{}}}
kafkactl consume avro_topic --from-beginning --print-schema -o yaml

If your Kafka cluster is only accessible via a Kubernetes cluster, you can configure kubernetes support in your context.

When enabled, kafkactl deploys a pod into the specified namespace and kubeContext to act as a proxy between your shell and the Kafka brokers.

Configuration Options

• enabled: Set to true to use Kubernetes mode.
• binary: Path to kubectl (optional).
• kubeConfig: Path to kubeconfig (optional).
• kubeContext: The Kubernetes context to use.
• namespace: The Kubernetes namespace to deploy the pod in.
• image: The Docker image to use (defaults to private.registry.com/deviceinsight/kafkactl).
• saslSecret: Use to avoid plaintext credentials in the pod.
  • name: Reference an existing K8s secret containing username and password keys.
  • create: true: Tells kafkactl to create a short-lived secret from your local config and delete it after the pod exits.

Usage Modes

1. kafkactl attach: Creates a pod and gives you a bash shell inside the container. This provides full auto-completion but takes longer to start.
2. Direct Commands: Running any other command (e.g., kafkactl get topics) deploys a short-lived pod using a scratch image, which is faster but lacks the interactive bash environment.

contexts:
  kafka-cluster:
    brokers:
      - broker1:9092
    kubernetes:
      enabled: true
      kubeContext: k8s-cluster
      namespace: k8s-namespace
      saslSecret:
        create: true
    sasl:
      enabled: true
      mechanism: scram-sha256
      username: my-user
      password: my-password

Configuration is managed via a YAML file. If no config is found, a default is generated at $HOME/.config/kafkactl/config.yml.

Configuration File Resolution Order

1. Command line argument: --config-file=$PATH_TO_CONFIG
2. Environment variable: KAFKA_CTL_CONFIG
3. Project config file in the current working directory (kafkactl.yml or .kafkactl.yml).
4. Default locations:
   • $HOME/.config/kafkactl/config.yml
   • $HOME/.kafkactl/config.yml
   • $APPDATA/kafkactl/config.yml
   • /etc/kafkactl/config.yml

Contexts

Contexts allow you to define multiple Kafka clusters. Each context can have its own brokers, tls, sasl, kubernetes, avro, schemaRegistry, protobuf, producer, and consumer settings.

Environment Variable Overrides

Every key in config.yml can be overridden using environment variables. Convert the key to uppercase, replacing . and - with _. Example: contexts.default.tls.certKey becomes CONTEXTS_DEFAULT_TLS_CERTKEY.

To set the default context specifically, you can omit the CONTEXTS_DEFAULT_ prefix (e.g., TLS_CERTKEY).

contexts:
  default:
    brokers:
      - localhost:9092
  remote-cluster:
    brokers:
      - remote-cluster001:9092
    tls:
      enabled: true
      ca: my-ca
      cert: my-cert
      certKey: my-key
      insecure: false

Use the produce command to send messages to a Kafka topic.

Input Formats

1. Standard (Separator-based): Use a separator (default is newline) to split key and value.

   echo "my-key#my-value" | kafkactl produce my-topic --separator=#

2. JSON Format: Each line must be a JSON object with key, value, and optional headers.

   echo '{"key": "k", "value": "v"}' | kafkactl produce my-topic --input-format=json

3. File Input: Use the --file flag to read messages from a file.

Common Flags

• --key=<val> / --value=<val>: Specify key and value directly.
• --separator=<char>: Custom delimiter between key and value.
• --lineSeparator=<string>: Custom delimiter between messages.
• --partition=<n>: Force a specific partition.
• --partitioner=<type>: Change partitioning scheme (e.g., random).
• --header <key>:<value>: Add a message header.
• --rate=<n>: Limit production to n messages per second.
• --null-value: Produce a tombstone record.
• --required-acks=<level>: Set acknowledgement level (e.g., WaitForAll).

Protobuf Production

Convert JSON to Protobuf messages on the fly:

kafkactl produce my-topic --value='{"field": 1}' --value-proto-type MyMsg --proto-file msg.proto

kafkactl produce my-topic --key=my-key --value=my-value --header key1:value1