repository·master·Indexed 42 days ago
https://github.com/ertis-research/kafka-mlA framework for managing TensorFlow/Keras and PyTorch machine learning pipelines on Kubernetes, designed to connect ML models to real-time data streams via Apache Kafka. It includes a Django 3.0.7 back-end for model management and various data source clients supporting Avro and RAW formats for training and inference workflows.
app.py, which uses Flask Views to handle various HTTP methods (such as GET and POST).app.py, which implements the API through Flask Views. These Views handle various HTTP methods such as GET and POST to facilitate model execution and interaction.Kafka-ML is a framework designed to manage machine learning (ML) pipelines on Kubernetes, specifically for TensorFlow/Keras and PyTorch (Ignite) models. It enables the design, training, and inference of ML models by connecting them to data streams via Apache Kafka. This allows for direct integration with real-time data sources like IoT streams.
Key features include:
Kafka-ML provides several client implementations in the datasources/ directory for streaming data to Apache Kafka. These clients support both training and inference workflows using Avro or RAW formats.
avro_sink.py: Sends training data using the Avro format. Requirement: You must provide definitions for two Avro schemas: one for the label and one for the data.raw_sink.py: Sends training data using the RAW format.federated_raw_sink.py: Sends training data using the RAW format specifically to topics prepared for federated learning.avro_inference.py: Sends inference data using the Avro format. Requirement: You must provide an Avro schema definition for the data.sink.py: The base file that provides the underlying logic used by both the Avro and RAW sink clients.The Federated Backend is built with Django 3.2.13 and Python 3.5-3.8. Key files include:
automl/models.py: Defines database tables and attributes.automl/views.py: Implements the RESTful API via Views (handling HTTP methods like GET, POST).automl/serializers.py: Handles encoding/decoding models to/from JSON.automl/urls.py: Maps RESTful Views to URL paths.autoweb/settings.py: The main configuration file.The Back-end is implemented using Django 3.0.7. Key files include:
automl/models.py: Defines database tables and attributes.automl/views.py: Implements the RESTful API via Views (handling HTTP methods like GET, POST).automl/serializers.py: Handles encoding/decoding models to and from JSON.automl/urls.py: Maps RESTful Views to URL paths.autoweb/settings.py: The main configuration file for the project.The TensorFlow inference module is designed to run as a Kubernetes Job within Kafka-ML. Once deployed, the module performs a continuous loop: it waits for data from a configured Kafka INPUT_TOPIC, processes it using a model downloaded via MODEL_URL, and sends the resulting predictions to a configured OUTPUT_TOPIC.
Key components of the module include:
inference.py: The main entry point executed during the inference Job.decoders.py: Handles data stream decoding for formats including RAW, Avro, JSON, and TELEGRAF_STR_JSON.config.py: Manages debug configurations.utils.py: Contains shared utility functions.The PyTorch training module is a task executed within a Kubernetes Job. It operates using a control-message-driven lifecycle:
CONTROL_TOPIC that matches the local DEPLOYMENT_ID.RESULT_URL.BATCH, KWARGS_FIT, etc.).RESULT_URL.Key files in this module:
training.py: The main entry point executed by the training Job.config.py: Configuration for debugging.utils.py: Common utility functions.The TensorFlow training module is a task designed to run within a Kubernetes-orchestrated TensorFlow training Job in Kafka-ML.
Lifecycle:
CONTROL_TOPIC that matches the DEPLOYMENT_ID.RESULT_URL.RESULT_URL.Core Files:
training.py: The main entry point executed during the training Job.decoders.py: Contains decoders for RAW, Avro, JSON, and TELEGRAF_STR_JSON data streams.config.py: Configuration for debugging.utils.py: Shared utility functions.The PyTorch model inference module is designed to run as a Kubernetes Job. Once deployed, it performs a continuous loop: it waits for data arriving on a configured Kafka input topic, processes it using a PyTorch model, and sends the resulting predictions to a configured Kafka output topic.
Key components of the module include:
inference.py: The main entry point executed during the inference Job.config.py: Used for debug configuration.utils.py: Contains common utility functions.The Kafka-ML Federated Module implements federated learning, a privacy-preserving approach where model training occurs on decentralized devices. Instead of transferring raw sensitive data to a central server, local devices perform training and only share model updates or gradients. These updates are aggregated by a central server to improve a global model, which is then redistributed to devices.
Current Support: Currently, federated learning support is limited to TensorFlow/Keras models. The usage pattern is similar to single-model deployment, but the configuration deployment form includes additional fields specific to federated orchestration.
The federated training module is designed to run as a TensorFlow training Job within a Kubernetes environment.
Lifecycle:
Core Components:
federated_training.py: The main entry point executed during the training Job.KafkaModelEngine.py: Handles reading and loading TensorFlow models from Apache Kafka in the cloud.FederatedKafkaMLAggregationSink.py: Handles sending trained models and metrics to Apache Kafka in the cloud.decoders.py: Provides decoders for various data formats: RAW, Avro, JSON, and TELEGRAF_STR_JSON.