DOCSEARCHbyAlgolia
Submit Docs

Apache DolphinScheduler

repository·dev·Indexed 10 days ago

https://github.com/apache/dolphinscheduler

A modern, cloud-native data orchestration platform for agile, low-code development of high-performance workflows. It supports complex task dependencies, various built-in job types, and provides comprehensive Kubernetes deployment options via Helm, including configurations for API-Server, Alert-Server, Master, and resource storage (S3, HDFS, OSS, GCS, ABS).

Tokens
176.4K
Snippets
421
Records
816
Agent score
96%

What's inside Apache DolphinScheduler

  1. Overview of DolphinScheduler system modules

    dev

    The DolphinScheduler architecture is composed of several specialized modules:

    • dolphinscheduler-master: Handles workflow management and orchestration.
    • dolphinscheduler-worker: Manages task execution.
    • dolphinscheduler-alert: Provides the AlertServer service for notifications.
    • dolphinscheduler-api: Provides the ApiServer web application service.
    • dolphinscheduler-common: Contains constants, utilities, and base classes.
    • dolphinscheduler-dao: Handles database access operations.
    • dolphinscheduler-extract: Provides SDKs for master, worker, and alert modules.
    • dolphinscheduler-service: Includes services for Quartz, Zookeeper, log client access, and API modules.
    • dolphinscheduler-ui: The front-end user interface.

  2. Overview of Apache DolphinScheduler

    dev

    Apache DolphinScheduler is a distributed, extensible, and visual DAG (Directed Acyclic Graph) workflow task scheduling system. It is designed to manage complex dependencies in data processing pipelines and is intended to be 'out-of-the-box' ready.

    Key Features:

    • Flexible Deployment: Supports Standalone, Cluster, Docker, and Kubernetes deployment modes.
    • Multiple Management Interfaces: Create and manage workflows via Web UI, Python SDK, or Open API.
    • High Availability: Uses a decentralized multi-master/multi-slave architecture that supports native horizontal scaling.
    • High Performance: Capable of supporting tens of millions of tasks per day.
    • Cloud Native: Supports orchestrating workflows across multiple clouds/data centers and allows for custom task types.
    • Workflow Control: Provides version control for workflows and instances, and supports pausing, stopping, or resuming tasks/workflows.
    • Multi-tenancy & Security: Supports multi-tenant environments and permission control for projects and data sources.

  3. What is Apache DolphinScheduler

    dev

    Apache DolphinScheduler is a distributed, visual workflow task scheduling platform designed for enterprise-level scenarios. It provides a solution for visualizing operations, workflows, and end-to-end data processing procedures.

    Key capabilities include:

    • Complex Dependency Management: Solves intricate big data task dependencies and trigger relationships in DataOps orchestration.
    • DAG-based Execution: Assembles tasks using a Directed Acyclic Graph (DAG) streaming mode.
    • Real-time Monitoring: Monitors task execution status in real-time.
    • Task Lifecycle Management: Supports operations such as retry, recovery (restarting from specific failed nodes), pause, resume, and killing tasks.

  4. Monitor system status via the Home Page

    dev

    The Apache DolphinScheduler Home Page provides a high-level overview of your system's health and activity. Use it to observe global statistics and drill down into specific details:

    • Task State Statistics: View the distribution of task statuses across the system.
    • Workflow State Statistics: Monitor the status of all workflows.
    • Project Statistics: Track activity and status for all user projects.
    • Drill-down Capabilities: From the home page, you can navigate into individual processes to inspect the specific status of tasks and access task logs.

  5. Ways to contribute to DolphinScheduler

    dev

    Participation in the DolphinScheduler community is not limited to code. You can contribute by:

    • Issue Management: Reporting problems via GitHub issues or answering questions asked by other users.
    • Documentation: Helping to improve or expand the project documentation.
    • Testing: Adding new test cases to ensure stability.
    • Code Quality: Adding comments to existing code.
    • Development: Submitting Pull Requests (PRs) to fix bugs or implement new features.
    • Community & Content: Publishing application case studies, scheduling process analyses, technical articles, or promoting DolphinScheduler at technical conferences and meetups.

  6. Understand the DolphinScheduler directory structure

    dev

    DolphinScheduler is organized into several component directories, each containing its own binaries, configurations, and libraries. Understanding this structure is essential for locating configuration files and startup scripts for specific services.

    Key components include:

    • bin/: Contains global application commands and the dolphinscheduler-daemon.sh script.
    • api-server/: Contains the API server components, including conf/application.yaml and conf/common.properties.
    • master-server/: Contains the master server components.
    • worker-server/: Contains the worker server components.
    • alert-server/: Contains the alert server components.
    • standalone-server/: A complete package for a single-node deployment.
    • tools/: Contains metadata management tools like upgrade-schema.sh.
    • ui/: Contains front-end web resources.

    Each server component (api, master, worker, alert, standalone) typically follows a consistent internal structure:

    • bin/: Startup scripts (e.g., start.sh) and JVM argument configurations (jvm_args_env.sh).
    • conf/: Configuration files including application.yaml (service-specific), common.properties (shared services like storage/credentials), and dolphinscheduler_env.sh (environment variables).
    • libs/: Component libraries.
    • log/ or logback-spring.xml: Logging configurations.
    ├── bin                                         directory of DolphinScheduler application commands, configurations scripts
│   ├── dolphinscheduler-daemon.sh              script to start or shut down DolphinScheduler application
│   ├── env                                     directory of scripts to load environment variables
│   │   ├── dolphinscheduler_env.sh             script to export environment variables [eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
├── alert-server                                directory of DolphinScheduler alert-server commands, configurations scripts and libs
│   ├── bin
│   │   └── start.sh                            script to start DolphinScheduler alert-server
│   ├── conf
│   │   ├── application.yaml                    configurations of alert-server
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc.
│   ├── libs                                    directory of alert-server libs
├── api-server                                  directory of DolphinScheduler api-server commands, configurations scripts and libs
│   ├── bin
│   │   └── start.sh                            script to start DolphinScheduler api-server
│   ├── conf
│   │   ├── application.yaml                    configurations of api-server
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc.
│   ├── libs                                    directory of api-server libs
│   └── ui                                      directory of api-server related front-end web resources
├── master-server                               directory of DolphinScheduler master-server commands, configurations scripts and libs
│   ├── bin
│   │   └── start.sh                            script to start DolphinScheduler master-server
│   ├── conf
│   │   ├── application.yaml                    configurations of master-server
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc.
│   ├── libs                                    directory of master-server libs
├── standalone-server                           directory of DolphinScheduler standalone-server commands, configurations scripts and libs
│   ├── bin
│   │   └── start.sh                            script to start DolphinScheduler standalone-server
│   ├── conf
│   │   ├── application.yaml                    configurations of standalone-server
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc.
│   ├── libs                                    directory of standalone-server libs
│   └── ui                                      directory of standalone-server related front-end web resources
├── tools                                       directory of DolphinScheduler metadata tools commands, configurations scripts and libs
│   ├── bin
│   │   └── upgrade-schema.sh                   script to initialize or upgrade DolphinScheduler metadata
│   ├── conf
│   │   ├── application.yaml                    configurations of tools
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc.
│   ├── libs                                     directory of tool libs
│   └── sql                                     .sql files to create or upgrade DolphinScheduler metadata
├── worker-server                               directory of DolphinScheduler worker-server commands, configurations scripts and libs
│   ├── bin
│   │   └── start.sh                            script to start DolphinScheduler worker-server
│   ├── conf
│   │   ├── application.yaml                    configurations of worker-server
│   │   ├── common.properties                   configurations of common-service like storage, credentials, etc.
│   ├── libs                                    directory of worker-server libs
└── ui                                          directory of front-end web resources

  7. Use the dolphinscheduler-extract RPC interfaces to communicate with servers

    dev

    The dolphinscheduler-extract module provides the Remote Procedure Call (RPC) interfaces required to communicate with different Apache DolphinScheduler server components. Depending on which server component you need to interact with, you should use the corresponding module:

    • Master Server: Use dolphinscheduler-extract-master to communicate with the master server.
    • Worker Server: Use dolphinscheduler-extract-worker to communicate with the worker server.
    • Alert Server: Use dolphinscheduler-extract-alert to communicate with the alert server.

    For developers building custom RPC clients or servers, the following foundational modules are available:

    • dolphinscheduler-extract-base: Contains the core interfaces for defining RPC clients and servers.
    • dolphinscheduler-extract-common: Contains shared interfaces used by both the master and worker components.

  8. View project overview and statistics

    dev

    Clicking a project name in the project management list opens the Project Home Page, which provides high-level metrics and statistics for the project within a specified time range:

    • Task status statistics: Counts task instances by status (e.g., success, failure, running, stop, kill, waiting threads, etc.).
    • Process status statistics: Counts workflow instances by status (e.g., submission success, running, failure, success, etc.).
    • Workflow definition statistics: Displays the count of workflow definitions created by the current user or granted by an administrator.

  9. Core features of Apache DolphinScheduler

    dev

    Apache DolphinScheduler is a distributed visual workflow scheduler designed for high reliability and scalability. Key capabilities include:

    • Visual DAG Definition: Use a drag-and-drop interface to define Directed Acyclic Graphs (DAGs) for workflows and manage run-time controls.
    • Extensible Task Types: Supports over 10 task types including Shell, MR (MapReduce), Spark, and SQL. The system is cross-language and modular, allowing for easy extension of new task types.
    • Workflow Lifecycle Management: Workflows can be managed through various states including timed execution, pausing, resuming, and stopping. This allows for fine-grained control over global and local parameters.
    • High Availability and Scalability: Features a decentralized design with a self-supporting HA (High Availability) task queue to prevent overload. It supports multi-tenancy and online resource management, capable of handling 100,000 data tasks per day.

  10. What is a DolphinScheduler Improvement Proposal (DSIP)

    dev

    A DolphinScheduler Improvement Proposal (DSIP) is a formal process used to introduce major improvements to the Apache DolphinScheduler codebase. DSIPs are intended for significant changes rather than small incremental updates.

    A change is considered a DSIP if it involves:

    • Any major new feature or major improvement.
    • Introducing or removing components.
    • Any major change to public interfaces, such as API endpoints or significant Web UI changes.

    If there is doubt about whether a change qualifies, committers may treat it as a DSIP. DSIPs are tracked using GitHub Issues (labeled with DSIP) and Apache mailing list threads.

  11. Overview of the MLflow Task Plugin

    dev

    The MLflow task plugin in Apache DolphinScheduler allows you to execute MLflow tasks within your workflows. It is designed to manage the machine learning lifecycle, including experimentation and deployment.

    Currently, the plugin supports:

    MLflow Projects

    Used to package data science code for reproducible runs on any platform. Supported types include:

    • BasicAlgorithm: Includes LogisticRegression, svm, lightgbm, and xgboost.
    • AutoML: Includes tools like autosklean and flaml.
    • Custom projects: Allows running your own custom MLflow projects.

    MLflow Models

    Used to deploy machine learning models in various serving environments. Supported types include:

    • MLFLOW: Uses MLflow models serve to deploy a model service.
    • Docker: Runs a container after packaging the Docker image.

    Note: Support for the MLflow Model Registry is planned for future releases.

  12. Manage Data Sources, Resources, and Alarms

    dev

    Beyond core workflow execution, DolphinScheduler provides management modules for resources and notifications. Permissions for these modules are typically managed by an Administrator.

    Data Source Management

    Users can create and manage data source connections. The permissions of the connection (including custom parameters) directly dictate the type and scope of SQL statements that can be executed by tasks using that connection.

    Resource Center

    Used for storing files required by workflows. Supported storage methods include:

    • Local file systems
    • Distributed file systems (e.g., HDFS)
    • Cloud object storage

    Alarm Management

    Users can configure authorized alarm channels to receive notifications for workflow timeouts or execution results. Note that alarm configurations may contain sensitive credentials that require careful management.