DOCSEARCHbyAlgolia
Submit Docs

Confluent demo-scene

repository·master·Indexed 23 days ago

https://github.com/confluentinc/demo-scene

A collection of scripts, samples, and demo environments for Confluent Platform talks. Includes practical examples for building applications and data pipelines using Confluent Cloud, ksqlDB, and the Confluent Admin REST APIs. Featured demos include an Agentic RAG AI application utilizing Flink SQL, OpenAI remote models, and MongoDB Atlas source connectors.

Tokens
282.2K
Snippets
715
Records
1K
Agent score
82%

What's inside demo-scene

  1. Overview of streaming XML into Kafka

    master

    This project explores different architectural patterns for ingesting XML data into Kafka. The approaches range from simple 'dirty hacks' to robust enterprise solutions using Kafka Connect.

    Key strategies discussed include:

    • The Dirty Hack: A lightweight, potentially less structured approach for quick ingestion.
    • Kafka Connect with Single Message Transform (SMT): Using Kafka Connect to ingest data and applying transformations to handle the XML structure during the ingestion process.
    • Kafka Connect FilePulse Connector: Utilizing the specialized FilePulse connector to specifically handle file-based XML ingestion.

    For a complete end-to-end implementation example, refer to the guide on streaming XML messages from IBM MQ into Kafka and then into MongoDB.

  3. Overview of Industry Themes use cases

    master

    The industry-themes directory contains various industry-specific use cases demonstrating stream processing applications. Most use cases provide the necessary data, producers, and ksqlDB code to demonstrate how to solve specific business problems using stream processing.

    Available use cases include:

    • Utilities: Tracking field assets.
    • Finance: Foreign Exchange market trading client services, Banking next-best-offer engines, and Insurance next-best-offer engines.
    • Healthcare: Looking up claim statistics by patient with FHIR events and medical device alerts from noisy sensors.
    • Insurance: Using a graph model to detect insurance fraud.
    • Logistics: Auto part tracking with enriched GPS events and truck GPS/sensor correlation for alerts.
    • Retail/E-commerce: Pizza order status tracking with nested JSON, activity-based discounting for e-commerce, and spend-based segmentation for grocery.

  4. Overview of the Confluent + MongoDB Hybrid Cloud Demo

    master

    This demo demonstrates a hybrid cloud data journey that bridges on-premise data with Confluent Cloud. The workflow involves:

    1. Data Generation: Data is generated on-premise in a traditional database (MongoDB).
    2. CDC Collection: Change Data Capture (CDC) events are collected on-premise.
    3. Cloud Bridging: Data is bridged to Confluent Cloud (on GCP).
    4. Stream Processing: ksqlDB is used to manipulate and enrich the data streams.
    5. Cloud Integration: The processed data is leveraged in cloud-native services.

    Key ksqlDB capabilities demonstrated include filtering streams, joining live events with reference data, performing stateful aggregations, and converting streams between formats (e.g., JSON to AVRO).

  5. Overview of the Micronaut, Kafka, and Lambda Demo

    master

    This demo showcases event-driven applications using Micronaut, Apache Kafka, and AWS Lambda. It is composed of three primary Micronaut projects:

    1. order-service: A standard Micronaut application utilizing the Kafka feature.
    2. order-id-lambda: A Micronaut Serverless Function designed for AWS Lambda.
    3. order-fulfillment-service: A Micronaut Messaging application integrated with Kafka.

    The demo is designed to run using Confluent Cloud, a cloud-native service for Apache Kafka.

  7. Explore demo-scene contents and categories

    master

    The demo-scene repository contains scripts and samples used to support Confluent Platform talks. The content is organized into several functional categories:

    • Applications: Real-world use cases like Telegram bots, Pac-Man streaming, and event-driven microservices.
    • Data Pipelines: ETL and analytics examples involving Databases, ksqlDB, and Elasticsearch (e.g., MySQL/Debezium/ksqlDB/Elasticsearch).
    • ksqlDB (formerly KSQL): Tutorials on ksqlDB basics, troubleshooting, UDFs, and specific use cases like ATM fraud detection.
    • Kafka Connect: Guides on Single Message Transforms (SMT), cluster building, error handling, and specific connectors (S3, JDBC, Elasticsearch, MQTT, etc.).
    • Confluent Cloud: Getting started guides for Java and Go, and integration examples with AWS Lambda, Micronaut, and MongoDB Atlas.
    • Confluent Platform: Demos for advanced features like Tiered Storage, Cluster Linking, and Admin REST APIs.
    • Misc: Ecosystem overviews, kafkacat export/import hacks, and specialized data analysis (Sonos, Wi-Fi pcap).

  9. Set up a Streaming ETL pipeline from MongoDB to Snowflake

    master

    This project demonstrates how to build a streaming ETL pipeline that moves data from MongoDB Atlas through Apache Kafka (Confluent Cloud) and into Snowflake. The pipeline uses Kafka Connect to ingest data, ksqlDB for real-time stream processing (identifying promotions based on stock levels and transaction rates), and a Snowflake Sink connector to land the processed data in a data warehouse.

    High-Level Workflow

    1. Ingest: Use a MongoDB Source connector to stream data from MongoDB into Kafka topics.
    2. Process: Use ksqlDB to join transaction streams with inventory data and calculate metrics like STOCK_LEVEL and TXN_PER_HOUR using windowed joins.
    3. Sink: Use a Snowflake Sink connector to stream the resulting processed data (e.g., identified promotions) into Snowflake.

    Prerequisites

    • MongoDB Atlas Account: For the source database.
    • Confluent Cloud Account: To host the Kafka cluster and ksqlDB.
    • Snowflake Account: For the destination data warehouse.
    • Docker & Docker Compose: To run local Kafka Connect workers for data generation.

    Note: Ensure your MongoDB and Confluent Cloud clusters are in the same region (e.g., us-east-1 on AWS) to minimize latency and costs.

  10. Set up the MQTT Kafka tracker architecture

    master

    The MQTT Kafka tracker demonstrates a data pipeline where:

    1. OwnTracks (mobile app) sends JSON data to an MQTT server.
    2. Kafka Connect (MQTT Source) ingests MQTT data into a Kafka topic.
    3. ksqlDB parses and transforms the data (applying schemas and extracting fields).
    4. Kafka Connect (Elasticsearch Sink) streams the transformed data to Elasticsearch.
    5. Kibana visualizes the data.

    Prerequisites

    • MQTT Server: Requires Hostname, Port, Username, and Password (e.g., CloudMQTT).
    • OwnTracks App: Installed on a phone and configured with your MQTT settings.
    • Docker/Docker Compose: Required for running the local stack or the Kafka Connect worker.

  11. What is the Hotset in Tiered Storage?

    master
    In Confluent Platform Tiered Storage, the Hotset refers to the recent log segments that remain on local disk. When the hotset time interval expires for a log segment, that segment is deleted from the local disk but continues to exist in object storage (e.g., S3). This allows for efficient local access to recent data while offloading older data to cheaper storage.

  12. Overview of Kafka Connect Single Message Transforms (SMT)

    master

    Kafka Connect's Single Message Transform (SMT) functionality allows you to modify data as it passes through a data pipeline. SMTs are ideal for simple, stateless transformations applied to individual messages.

    When to use SMTs vs. other tools:

    • Use SMTs for simple modifications like adding fields, renaming fields, masking data, or changing topic routing.
    • Use Kafka Streams or ksqlDB for complex transformations that require state, such as joins or aggregations.