DOCSEARCHbyAlgolia
Submit Docs

Michelangelo AI Documentation

website·Indexed 30 days ago

https://michelangelo-ai.org/docs

A platform for machine learning operations (MLOps) providing tools for ML pipelines, distributed training, model registry, and inference serving. Includes operator guides for Kubernetes deployment via Helm charts, network configuration with Envoy proxy, and observability using Prometheus. Supports integration with Comet ML and MLflow, and provides a React library for UI components.

Tokens
137.5K
Snippets
656
Records
934
Agent score
57%

What's inside Michelangelo AI

  1. Overview of core Go backend services

    The Go backend implements three primary server-side services:

    1. API Server (go/cmd/apiserver/): A central gRPC server acting as the control plane. It validates resources, stores them as Kubernetes CRDs, and invokes API hooks on mutating operations.
    2. Controller Manager (go/cmd/controllermgr/): Manages Kubernetes controllers for ML resources (e.g., RayCluster, SparkJob, InferenceServer). It uses a reconcile loop to align the actual cluster state with the desired state in etcd and MySQL.
    3. Worker (go/cmd/worker/, go/worker/plugins/): Hosts Cadence workflow and activity workers. It uses plugins in go/worker/plugins/ to extend the worker with domain-specific Starlark builtins.

  2. Overview of EvaluationReportService

    The EvaluationReportService provides CRUD and listing methods for EvaluationReport objects. An EvaluationReport stores charts, filters, and data point sources describing the evaluation results of a trained model. These reports are referenced by ModelSpec fields such as performance_evaluation_report, feature_evaluation_report, feature_quality_report, and explainability_report. Reports can be sealed to prevent further modification.

  3. Overview of the Michelangelo AI Ingester

    The Ingester is a Kubernetes controller that synchronizes all Michelangelo AI CRDs into MySQL. This decouples metadata storage from etcd, ensuring that every CRD object created via the API Server is mirrored in MySQL. The Ingester manages the lifecycle of these objects, including adding finalizers to Kubernetes objects and performing soft-deletes in MySQL (setting a delete_time instead of removing the row) before the object is fully removed from Kubernetes.

  4. Overview of Michelangelo AI ML Pipelines and Uniflow

    ML Pipelines on Michelangelo AI enable the creation and management of end-to-end machine learning workflows (data preparation, training, and evaluation). These are built using Uniflow, a Python-first framework where workflows are defined using standard Python functions and can be executed locally or at production scale.

  5. Overview of Michelangelo ML Platform components

    Michelangelo is an open-source machine learning platform originally developed by Uber for building, training, and deploying ML models at scale. The open-source release consists of several core components: the Uniflow pipeline framework, the API server, the controller, a local sandbox for development, and a CLI.

  6. Overview of Michelangelo AI Notification Workflow

    Michelangelo AI provides a notification workflow that triggers when a PipelineRun changes state. The controller manager detects the transition and starts an io.michelangelo.notification.PipelineRunFanout workflow on the notification_worker task queue (Temporal or Cadence). A worker then fans out the notification to configured Sinks (e.g., EmailSink, SlackSink). By default, these Sinks are no-ops that log warnings; platform operators must replace them with real delivery implementations.

  7. Overview of Michelangelo AI Metadata Storage

    Michelangelo AI uses a MySQL-backed metadata store to sync Kubernetes custom resources (CRDs) via an 'ingester' controller. This allows API and operations workflows to query metadata without relying solely on etcd. The SQL store is used for platform metadata only, not for training datasets or feature values.

  8. Overview of Michelangelo ML Platform capabilities

    Michelangelo is an open-source machine learning platform designed for production scale. It provides a comprehensive suite of tools for the ML lifecycle, including:

    • Model Training: Scalable distributed training with experiment tracking, hyperparameter tuning, and automatic resource management.
    • Deployment: One-click production deployment featuring canary releases, traffic splitting, and automatic rollbacks.
    • Feature Management (Coming Soon): Centralized feature store with versioning, lineage tracking, and real-time serving.
    • Evaluation (Coming Soon): Automated metrics, A/B testing, and performance benchmarking.
    • Monitoring (Coming Soon): Real-time drift detection, alerting, and observability.

  9. Overview of Michelangelo Native Transforms

    michelangelo.lib.native_transform provides a library of PyTorch feature-transform layers (such as StandardScaler, Bucketization, Cast, and Concatenate) that run identically at training time and serving time. This prevents training-serving skew by using the exact same code for scaling, bucketizing, or tokenizing columns in both environments. The system uses a declarative TransformSpec (a DAG of layer specs) which can be parsed from a dictionary or YAML file.

  10. Overview of the Uniflow I/O plugin system

    The I/O plugin system allows for reading and writing Python objects to storage backends, including local file systems and remote object stores (GCS, S3, HDFS, etc.) using fsspec integration.

    Key components include:

    • IO: The abstract base class for serialization/deserialization interfaces.
    • IORegistry: A registry that maps Python types to their corresponding I/O handlers.
    • BytesIOIO: A built-in handler for BytesIO objects.
    • default_io: A module-level IORegistry singleton pre-populated with a BytesIO handler. This is the default registry used by @task for argument and result serialization unless a different registry is specified via the io parameter.

  11. Overview of Michelangelo AI Serving

    Michelangelo AI Serving is a control plane for managing ML model serving infrastructure on Kubernetes. It automates the complete lifecycle of inference servers and model deployments, including provisioning infrastructure (Deployments, Services), managing model configurations, routing traffic, health monitoring, and automatic cleanup.

  12. Overview of Michelangelo AI and Kueue Integration

    Michelangelo AI can delegate batch-job admission to Kueue to provide quota-aware queueing and gang admission. This backend is opt-in per compute cluster; clusters not opted-in maintain default immediate-admission behavior.

    When enabled, the control plane validates that a job has a corresponding LocalQueue before dispatch. Upon dispatch, the k8s engine labels the created RayCluster with kueue.x-k8s.io/queue-name (resolved from the job's project). Kueue then suspends the RayCluster and admits it once ClusterQueue quota is available.