DOCSEARCHbyAlgolia
Submit Docs

Factor House Local Playground

website·Indexed 30 days ago

https://fh-docs-staging-docs.factor-house.workers.dev

Documentation for deploying and configuring local data stacks using Docker Compose. Includes guides for Apache Kafka with Kpow, Unified Analytics with Flex, Apache Pinot for real-time OLAP, OpenMetadata for governance, and observability environments for data lineage.

Tokens
637.9K
Snippets
3K
Records
4.6K
Agent score
38%

What's inside Factor House

  2. Overview of the Factor House Playground

    The Playground provides a set of self-contained environments for hands-on exploration of modern data platform architectures. It covers real-world scenarios involving event-driven systems, real-time data processing, lakehouse architectures, and operational analytics. Key technologies available in the Playground include Kafka, Flink, Spark, Iceberg, and Pinot, with integrated capabilities for observability, orchestration, and integration. Supported workloads include stream processing, change data capture (CDC), data lake ingestion, and OLAP-style querying.

  3. Overview of Klang Language Syntax

    Klang is a scripting language used in the kREPL to process bounded sequences of structured data from Kafka topics. It is a blend of JQ and Clojure, utilizing EDN (a superset of JSON) as its data representation. Klang consists of pure Clojure functions, pure JQ pipelines, and side-effecting Klang functions (like consume).

  4. Overview of Klang Language Syntax

    Klang is a scripting language used in the kREPL to process bounded sequences of structured data from Kafka topics. It is a blend of JQ and Clojure, allowing users to use either or both syntaxes. Klang is based on S-Expressions where the outermost parentheses are optional. It supports three types of functions: pure Clojure functions, pure JQ pipelines, and side-effecting Klang functions (e.g., consume).

  5. Overview of the Factor House Local Kafka Data Stack

    This stack provides a complete Apache Kafka development and monitoring environment. It is designed for developers and operations teams to build, test, and manage Kafka-based applications locally using a modern, Zookeeper-less architecture (KRaft mode). The environment is ideal for event-driven architectures, microservices communication, and data integration pipelines.

  6. Overview of Kpow accessibility features

    Kpow provides several accessibility features to ensure an inclusive experience for all engineers, including full keyboard navigation of the user interface, support for screen readers for users who are blind or have low vision, and high-contrast theme options. The application is designed to support browser zooming to increase text size without breaking page content.

  7. Overview of Factor House Local Labs for real-time data pipelines

    Factor House Local labs provide a modular, production-inspired entry point for developers to build real-time data pipelines. The labs cover a variety of technologies and patterns, including:

    • Kafka & Schema Registry: Producing/consuming Avro-encoded messages and managing schema evolution.
    • Kafka Connect & Kpow: Deploying and managing source/sink connectors via UI and API.
    • Stream Processing: Using Kafka Streams (with Kpow Streams Agent for topology visualization) and Apache Flink (via SQL and DataStream APIs).
    • Lakehouse Integration: Writing data to Apache Iceberg and MinIO using Flink SQL, Flink Table API, Kafka Connect, and Spark Structured Streaming.
    • Real-time Analytics: Using Apache Pinot for multi-stage query engines and the Flink SQL Gateway for serving stats via REST API.
    # Access the lab source code on GitHub
https://github.com/factorhouse/examples/tree/main/fh-local-labs

  8. Introduction to kJQ pipeline model

    kJQ is a filter-based language for processing structured data, based on the jq language. The fundamental concept is the pipeline, where data flows from left to right through a series of transformations using the pipe (|) operator. Every expression in kJQ is a filter that takes an input and produces an output.
    input | transform1 | transform2 | output

  9. Overview of Kpow Multi-tenancy

    Multi-tenancy in Kpow allows administrators to restrict the set of Kafka resources accessible to specific user roles. Users operating within a tenant see a consistent 'synthetic cluster-view' containing only the resources assigned to that tenant. Users can only create resources that are valid for their assigned tenant. A user role may be assigned to multiple tenants.

    Resource Evaluation Logic:

    1. All resources are implicitly excluded by default.
    2. Explicitly included resources are added.
    3. Inferred resources (e.g., topics consumed by an included group) are added to maintain consistency.
    4. Exclusions are applied last, overriding previous inclusions.

  10. Overview of Kpow Multi-tenancy

    Multi-tenancy in Kpow restricts the set of Kafka resources accessible to a user role. When operating within a tenant, users see a consistent 'synthetic cluster-view' containing only the resources assigned to that tenant. Users can only create resources that are valid for their assigned tenant. A single user role can be assigned to multiple tenants.

    Resource Evaluation Logic:

    1. All resources are implicitly excluded by default.
    2. Explicitly included resources are added.
    3. Inferred resources (e.g., topics consumed by an included group) are automatically added to maintain consistency.
    4. Exclusions are applied last, overriding any previous inclusions.