DOCSEARCHbyAlgolia
Submit Docs

Apache DolphinScheduler Documentation

repository·master·Indexed 40 days ago

https://github.com/apache/dolphinscheduler-website

Documentation for Apache DolphinScheduler, covering the official website source code, MasterServer startup processes, Workflow execution lifecycles, and the WorkflowAsCode Python API introduced in version 2.0.2. Includes guides on DAG construction, Zookeeper registration, and alert plugin configurations for WeCom and DingTalk.

Tokens
47.9K
Snippets
79
Records
221
Agent score
50%

What's inside Apache DolphinScheduler

  1. Overview of Apache DolphinScheduler

    master

    Apache DolphinScheduler is a distributed big data workflow task scheduler incubated by the Apache Software Foundation. It is designed to manage intricate dependencies between big data tasks by assembling them into a DAG (Directed Acyclic Graph).

    Core Capabilities:

    • DAG Visualization: Makes the entire data process intuitive and visible.
    • Task Management: Supports operations such as retry, recovery from designated nodes, suspend, and Kill tasks.
    • Rich Task Types: Provides a wide variety of task types for different big data ecosystems.
    • Monitoring: Real-time monitoring of task running status, including history, task logs, and alarm mechanisms.

  2. Case Study: Building a Machine Learning Platform with DolphinScheduler

    master

    Lizhi, a large-scale UGC audio community, successfully migrated from Azkaban to Apache DolphinScheduler to power its Machine Learning Intelligent Recommendation Platform.

    Why DolphinScheduler was chosen:

    • Visual DAG Programming: Supports drag-and-drop UI for defining complex machine learning workflows (data acquisition $\rightarrow$ preprocessing $\rightarrow$ training $\rightarrow$ evaluation $\rightarrow$ release).
    • High Availability: Distributed decentralized architecture and fault tolerance mechanisms ensure stability for massive data processing.
    • Extensibility: Easy to develop and integrate custom modules for specific AI paradigms.
    • User Accessibility: Meets the needs of recommendation engineers and data analysts who require a simple, low-code interface rather than manual script packaging.

    Real-world Scale:

    Lizhi runs over 1,600 processes and 12,000+ tasks daily on DolphinScheduler.

  3. New features in Apache DolphinScheduler 2.0.5

    master

    Apache DolphinScheduler version 2.0.5 introduced several functional optimizations and bug fixes:

    Key Optimizations

    • Worker Fault Tolerance: Improved the fault tolerance process. If a server is interrupted due to excessive pressure, tasks can be automatically transferred to other workers to prevent interruption.
    • Resource Center Re-uploading: Users can now re-upload files in the Resource Center. This allows for automatic updating of execution scripts without needing to reconfigure task parameters.
    • DingTalk Alert Plugin: Added support for Markdown information types in DingTalk alarm content.
    • UI/UX Improvements:
      • Optimized flags on pages where tasks are prohibited from running to distinguish them from normal tasks.
      • Added prompts in the task box to display long task names.
      • Modified workflow editing to jump directly to the list page upon completion.

    Bug Fixes

    • Fixed incorrect task runs when worker groups contain uppercase letters.
    • Fixed workflows that could not be stopped when a task failed and retried.
    • Fixed an issue where JDBC connection parameters could not include the @ character.
    • Fixed issues with complement tasks not ending normally and failures when entering sub-workflows from the UI.

  4. Case Study: Migrating from Airflow to Apache DolphinScheduler at Youzan

    master

    This case study details the migration of Youzan's Big Data Development Platform (DP) from Apache Airflow to Apache DolphinScheduler.

    Why Youzan switched from Airflow:

    • Maintenance Costs: Airflow's Python-based stack resulted in higher maintenance and iteration costs compared to DolphinScheduler's Java-based stack.
    • Performance Bottlenecks: Airflow (pre-v2.0) suffered from single-point DAG scanning/parsing, causing significant delays (60s-70s) as the number of DAGs grew. DolphinScheduler demonstrated twice the throughput under similar conditions.
    • Stability & HA Issues: Airflow's High Availability (HA) via the Airflow Scheduler Failover Controller used a master-slave mode that could fail to detect deadlocks, leading to scheduling failures. DolphinScheduler uses a decentralized multi-Master/multi-Worker architecture for better fault tolerance.
    • Deployment & Scalability: DolphinScheduler supports standardized deployment via Kubernetes and Docker, and its scheduling capability scales linearly with cluster size.

    Key Advantages of DolphinScheduler identified by Youzan:

    • Throughput: Significantly higher task processing capacity.
    • Deployment: Easier operations (Ops) due to the Java stack and container support.
    • Features: Flexible task-dependent configurations, refined time granularity (hour, day, week, month), and worker group isolation.
    • Community: Highly active community with frequent updates and detailed documentation.

  5. Usage scenarios for Apache DolphinScheduler in XWBank

    master

    XWBank utilizes Apache DolphinScheduler across three primary data processing scenarios:

    1. Offline Data Development and Task Scheduling: Used for banking data warehouses and data marts involving daily or monthly offline data processing.
    2. Quasi-real-time Data Development and Task Scheduling: Used for minute-by-minute batch calculations. This typically involves data fused by Flink from upstream message queues and pushed to Clickhouse.
    3. Other non-ETL user-defined data batch processing: Used for model data calculation tasks pushed from metrics management platforms and tag management systems. This often involves low-code platforms dynamically generating SQL and pushing it to the scheduler for self-service data analysis.

  6. Case Study: Tujia Big Data Platform using Apache DolphinScheduler

    master

    Tujia implemented Apache DolphinScheduler in 2019 to serve as the command center for their big data platform. The platform manages the full data lifecycle: data source (MySQL, Dubbo, HTTP), data collection (Canal, Flume, Kafka), data storage (HDFS), data management (metadata, logs, alarms, dispatching), and data service (ad hoc queries, reports).

    Key benefits realized by Tujia include:

    • Reliability: Decentralized design avoids single points of failure; tasks automatically restart on other nodes if a node fails.
    • Scalability: The cluster can scale resources as task volume increases.
    • Versatility: Supports diverse task types including Shell, MR, Spark, SQL (MySQL, PostgreSQL, Hive, SparkSQL), Python, Sub_Process, and Procedure.
    • Operational Control: Enables workflow timing, dependency scheduling, manual scheduling, pause/stop/resume, and retry/alarm on failure.

  7. Python API updates in Apache DolphinScheduler 3.1.2

    master

    Apache DolphinScheduler version 3.1.2 introduces several optimizations and features for the Python API (pydolphinscheduler). Key updates include:

    • Authentication: Added token authentication for the Python API/gateway.
    • Resource Management: Fixed issues where uploading to the resource center via Python API would fail.
    • CRUD Operations: Added support for Create, Read, Update, and Delete (CURD) operations in the project, tenant, and user sections.
    • Terminology Change: The term process definition has been changed to workflow within the Python API.
    • Execution Control: Added support to set the execute type via pydolphinscheduler.
    • Validation: The Python API now validates its version at launch.

    These changes aim to improve the stability and usability of programmatic orchestration via Python.

  8. Release Notes for Apache DolphinScheduler 3.0.1

    master

    Apache DolphinScheduler 3.0.1 focuses on optimizing the scheduling kernel and the User Interface (UI). Key areas of improvement include:

    • Scheduling Kernel: Fixes for infinite loops in workflow execution, dependent task errors, and OOM (Out of Memory) issues when workers kill tasks.
    • UI Optimizations: Improvements to workflow list name overlapping, data source modification error reporting, and SQL script editing (added highlight and auto-complete).
    • Bug Fixes: Numerous fixes across LDAP, S3, Quartz, and various task types (SQL, Spark-SQL, etc.).
    • Documentation: Corrections to workflow descriptions, worker group creation, and Python API links.

  9. Optimizations in Apache DolphinScheduler 2.0 alpha

    master

    Key areas of optimization in the 2.0 alpha release include:

    • API & Installation: Optimized RestApi and the install.sh installation script (to simplify the configuration process).
    • Workflow & Groups: Optimized workflow version management, alarm group function, and worker group management function.
    • Data Operations: Optimized import and export processes.

  11. New features in Apache DolphinScheduler 2.0 alpha

    master

    The 2.0 alpha release introduces several new functional capabilities:

    • Task Management: Task result transfer function, and new task plugin functions.
    • Task Components: Added Switch task and Pigeon task components.
    • Environment & Configuration: Added environmental management function.
    • Data Operations: Added batch import, export, and batch move functions.
    • Extensibility: New registration center plug-in function via SPI.

  12. Apache DolphinScheduler 3.0.0 Alpha Release Notes

    master

    The 3.0.0 Alpha release of Apache DolphinScheduler introduces a new UI, support for AWS, service separation, data quality verification, task groups, customizable time zones, a list of task definitions, new alert types, and new Python API features.

    Key bug fixes in this release include:

    • Resolving S3a Minio tenant creation failures.
    • Fixing duplicate authorized projects during project authorization.
    • Addressing PostgreSQL connection failures during server startup.
    • Fixing 'datasource plugin 'spark' is not found' errors.
    • Correcting MapReduce command built-in parameter locations.
    • Fixing issues where user and queue parameters failed to work in ProcessDefinition.
    • Resolving resource file deletion conditions and worker resource exhaustion issues.