DOCSEARCHbyAlgolia
Submit Docs

Yōzefu

repository·main·Indexed 43 days ago

https://github.com/maif/yozefu

An interactive TUI and CLI tool for exploring Kafka cluster data in real-time. It features a SQL-inspired query language and supports extensible search filters via WebAssembly (WASM) implemented in Rust, Golang, or JavaScript. The project includes several Rust crates: yozefu-app for consumer logic, yozefu-command for CLI management, yozefu-lib for core definitions like KafkaRecord, and yozefu-tui for the Ratatui-based interface.

Tokens
36K
Snippets
129
Records
166
Agent score
53%

What's inside Yōzefu

  1. Overview of yozefu-wasm-types

    main
    The yozefu-wasm-types library provides the data structures required to define a WebAssembly (Wasm) module for the Yozefu search engine. It facilitates data exchange between the Yozefu host and the Wasm module using json as the communication format.

  2. Overview of Yōzefu features

    main

    Yōzefu is a TUI (Terminal User Interface) designed to explore Kafka clusters. Key capabilities include:

    • Real-time + Multi-topic: Live access to records and the ability to search across multiple topics simultaneously.
    • Powerful Querying: Uses a SQL-inspired query language with fine-grained filters. The querying engine is extensible via WebAssembly (Wasm).
    • Dual Modes: Can be used as an interactive TUI or as a CLI in headless mode using the --headless flag.
    • One-key Export: Supports quick exporting of Kafka records for further analysis.

  3. What is Yōzefu?

    main
    Yōzefu is a fast and efficient terminal-based tool designed to explore Kafka clusters. It provides real-time access to data published to topics and allows for fine-grained filtering using a SQL-inspired search query language. It can be used as an interactive Terminal User Interface (TUI) or as a headless Command Line Interface (CLI).

  4. Yōzefu limitations and features overview

    main

    Features

    • Real-time access: View data published to topics as it arrives.
    • Search Query Language: SQL-inspired syntax for fine-grained filtering.
    • Multi-topic search: Search across multiple Kafka topics simultaneously.
    • Wasm Extensions: Extend the engine with user-defined filters.
    • Export: One-keystroke export of Kafka records for analysis.
    • Headless Mode: CLI-first operation for automation.

    Limitations

    • Consumption Only: Designed only to consume records; cannot produce records or manage clusters.
    • Serialization Support: Supports json, xml, and plain text. Avro support is currently experimental. Protobuf is not supported.
    • Buffer Size: Uses a ring buffer that stores only the last 500 Kafka records.
    • Platform Support: Tested on macOS Silicon; Linux and Windows support is not explicitly verified.

  5. Core components of yozefu-lib

    main

    The yozefu-lib crate provides the fundamental building blocks for working with Yozefu. Key components include:

    • KafkaRecord: The primary structure used throughout the ecosystem to represent a Kafka record.
    • Error Definitions: A set of error types representing various failure states that can occur during operation.
    • Query Engine: Structures and functions designed to parse and execute search queries.

  6. What is a search filter and how does it work

    main

    A search filter is a WebAssembly (Wasm) module used to extend Yōzefu's search engine with custom logic that the standard query syntax cannot express (e.g., checking if a Kafka key ends with a specific suffix).

    To use a filter in a query, the name of the function in your SQL-like syntax must match the name of the .wasm file. For example, a query using key-ends-with('1234') requires a Wasm file named key-ends-with.wasm.

    Search filters are implemented using Extism and must export two specific functions:

    1. parse_parameters: Validates user-provided arguments during the parsing phase.
    2. matches: Evaluates every Kafka record against the provided parameters to determine if it should be included in the results.

    You can locate your existing Wasm filters by running yozf config get filters-dir.

    -- Example usage of a custom filter named 'key-ends-with'
from begin where key-ends-with('1234')

  7. Understand Yōzefu limitations and capabilities

    main

    Before using Yōzefu, be aware of the following constraints and features:

    Capabilities

    • Real-time access: View data as it is published to topics.
    • Multi-topic search: Search across multiple Kafka topics.
    • Extensibility: Use WebAssembly (via Extism) to write user-defined search filters.
    • Cluster Management: Register multiple Kafka clusters with specific consumer properties.
    • Data Export: Export Kafka records with a single keystroke.

    Limitations

    • Consumption Only: Yōzefu is designed only to consume records. It cannot produce records or manage Kafka clusters.
    • Serialization Support: Supports json, xml, and plain text. Avro support is experimental. Protobuf is not supported.
    • Memory/Buffer: Yōzefu uses a ring buffer to store only the last 500 Kafka records; it does not keep the entire history in memory.
    • Platform Support: Primarily tested on macOS Silicon; Linux and Windows support may vary.

  8. Query language syntax and operators

    main

    The Yōzefu query language supports several operators and keywords for filtering Kafka records:

    Keywords

    • from begin: Starts searching from the earliest available record.
    • from end - [N]: Starts searching from the N-th record from the end of the log.
    • from "[timestamp]": Starts searching from a specific ISO 8601 timestamp.
    • where: Introduces a clause for filtering conditions.
    • limit [N]: Restricts the number of returned records.
    • between [A] and [B]: Filters values within a range.

    Operators

    • ==: Equality comparison.
    • !=: Inequality comparison.
    • >= / <=: Greater than or equal / Less than or equal.
    • > / <: Greater than / Less than.
    • contains: Substring or membership check (used for key or value).
    • ||: Logical OR.
    • and: Logical AND.

    Field Access

    • partition: The Kafka partition number.
    • offset: The Kafka record offset.
    • key: The record key.
    • value: The record value (supports dot notation for JSON properties, e.g., value.album.title).
    • size: The size of the record in bytes.
    • timestamp: The record timestamp.

  9. Run Yozefu in headless mode

    main
    Yozefu supports a headless mode. Unlike the default TUI (Terminal User Interface) which uses Ratatui, headless mode runs the application without a graphical terminal interface and prints all results directly to stdout. This is useful for automation or simple text-based output.

  10. Configure syntax highlighting (Highlighter)

    main

    Yōzefu uses the Syntect library for syntax highlighting. You can choose from 7 built-in Syntect themes (such as base16-ocean.dark, base16-mocha.dark, or Solarized (dark)) or provide a path to a custom Sublime Text .tmTheme file.

    To select a highlighter theme, you have two options:

    1. Global Configuration: Set the highlighter_theme property in your config.json file.
    2. Theme-Specific Configuration: Set the highlighter_theme property within a specific theme definition in themes.json under /<theme-name>/highlighter_theme.

    To disable syntax highlighting entirely, set the highlighter_theme property to null in either config.json or themes.json.

  11. Open Kafka records in a web browser from the TUI

    main

    Yōzefu allows you to view a Kafka record in a web browser directly from the TUI. To do this, select the desired Kafka record and press the <kbd>o</kbd> key (open). This action will open the corresponding URL in a new browser tab based on the template configured for your cluster.

    To enable this, you must define a URL template in your configuration file using the .clusters.<name-of-your-cluster>.url_template property.

    # Example configuration structure
.clusters.my-cluster-name.url_template = "https://example.com/topic/{topic}/offset/{offset}"