DOCSEARCHbyAlgolia
Submit Docs

Apache ShardingSphere-ElasticJob Documentation

repository·master·Indexed 27 days ago

https://github.com/apache/shardingsphere-elasticjob

A lightweight, decentralized distributed scheduling solution for Internet-scale applications. It provides task sharding, high availability, and resource management, allowing developers to scale job processing capacity by adding servers. Key features include Elastic Schedule, Resource Assign, Job Governance, and a unified job API. Supports integration with Spring Boot via the elasticjob-spring-boot-starter and offers a pluggable architecture using SPI for custom error handlers, sharding strategies, and job class name providers.

Tokens
67.8K
Snippets
174
Records
349
Agent score
92%

What's inside Apache ShardingSphere-ElasticJob

  1. Overview of ElasticJob distributed scheduling

    master

    ElasticJob is a distributed scheduling solution designed for internet-scale ecosystems and massive task volumes. It provides elastic scheduling, resource management, and job governance through an open architecture.

    Key features include:

    • Elastic Scheduling: Supports task sharding and high availability in distributed scenarios, allowing horizontal scaling of throughput and execution efficiency.
    • Resource Allocation: Aggregates identical tasks to the same executor and dynamically allocates resources to new tasks.
    • Job Governance: Provides failover, missed job re-execution, and self-diagnostic repair.
    • Open Ecosystem: Offers a unified interface for extensible job types (e.g., data stream, script, HTTP, file, big data) and seamless integration with Spring dependency injection.
    • Visual Management: Supports a separate UI (shardingsphere-elasticjob-ui) for job control, execution history tracking, and registry management.

  2. Overview of ElasticJob features

    master

    ElasticJob is a lightweight, decentralized distributed task sharding solution. Key features include:

    • Elastic Schedule: Supports job sharding and high availability in distributed systems. Allows scaling out to improve throughput and efficiency.
    • Resource Assign: Executes jobs on suitable time and assigned resources, including dynamic resource appending.
    • Job Governance: Provides failover, misfired job handling, and self-diagnosis/recovery in unstable distributed environments.
    • Job Open Ecosystem: Provides a unified job API for extensions, supporting various job types (dataflow, script, HTTP, file, big data) and integration with Spring IOC.
    • Admin Console: Enables job administration, job event trace queries, and registry center management.

  4. Overview of Hugo Learn theme features

    master

    The hugo-theme-learn is a Hugo theme specifically designed for documentation. Key features include:

    • Automatic Search: Built-in search functionality.
    • Multi-language Support: Support for internationalization (i18n).
    • Hierarchical Menus: Supports infinite menu nesting levels.
    • Automatic Navigation: Automatically generated 'Next' and 'Previous' buttons for menu item navigation.
    • Tree Structure: Organizes content using a tree-like page structure where pages belong to parent pages.
    • Mermaid Diagrams: Support for flowchart, sequence, and gantt diagrams via Mermaid.
    • Shortcodes: Includes shortcodes for buttons, attachments, listing child pages, expand/collapse sections, and notice messages (Tip, Info, Note, Attention).

  5. Overview of ElasticJob

    master
    ElasticJob is a lightweight, decentralized distributed scheduled job solution designed for Internet scenarios. It provides distributed task sharding services, allowing developers to focus on business logic while the framework handles non-functional requirements like job scale-out, high availability, and resource management. It uses a unified job API that allows for easy deployment and extension.

  6. Understand ElasticJob Failover mechanism

    master

    Failover is a temporary compensation mechanism used when a job server goes down during execution. It allows unfinished tasks from the failed shard to be completed by other active job nodes.

    Key characteristics:

    • No immediate re-sharding: ElasticJob does not re-shard during the current execution cycle; it waits for the next scheduled interval to perform formal re-sharding.
    • Compensation: Other servers sense the down server and execute the remaining items to ensure the job completes.
    • Prerequisite: The job implementation must be idempotent to ensure correctness when tasks are re-executed on different nodes.

  9. Understand ElasticJob Sharding and Sharding Items

    master

    ElasticJob achieves horizontal scalability through sharding. A job is split into multiple independent sharding items, which are distributed across available job servers.

    • Sharding Items: Represented by integers starting from 0 and ending at totalShardingItems - 1.
    • Responsibility: Each server runs only the sharding items assigned to it. As servers are added or removed, ElasticJob reassigns these items in near real-time to balance the load.
    • Implementation: Developers are responsible for mapping the numeric sharding item to their specific business logic.

  10. Key features of ElasticJob

    master

    ElasticJob provides several core capabilities for distributed task management:

    • Elastic Schedule: Supports job sharding and high availability in distributed systems. You can scale out to improve throughput and efficiency by allocating more resources.
    • Resource Assign: Executes jobs at suitable times on assigned resources. It can aggregate the same job to the same executor and dynamically append resources to newly assigned jobs.
    • Job Governance: Includes mechanisms for Failover, handling Misfired jobs, and self-diagnosis/recovery during unstable distributed environments.
    • Job Open Ecosystem: Provides a unified job API for extensions and supports various job types (e.g., dataflow, script, HTTP, file, big data). It is designed to work with Spring IOC.
    • Admin Console: (Available via a separate project) Provides job administration, job event trace queries, and registry center management.

  11. Understand ElasticJob Console design and limitations

    master

    Design Concept

    The console is decoupled from ElasticJob. It manages job status by reading from or updating the job registration center data.

    Important Limitation: The console can control the lifecycle of a job (enable/disable, stop, delete) by modifying global configurations, but it cannot start the job process itself. The console and the job server are completely separate; the console cannot control the job server directly.

    Unsupported Items

    • Add assignment: Jobs are automatically added to the console the first time they run.
    • Job distribution: ElasticJob is started as a JAR and does not have a built-in job distribution function.

  12. Understand ElasticJob Job Types (Class vs Type)

    master

    ElasticJob 3.x categorizes jobs into two main types based on how the business logic is provided:

    1. Class-based Jobs: The developer implements a specific job interface to define the business logic directly in Java code.
      • Examples: Simple type, Dataflow type.
    2. Type-based Jobs: The developer provides a type name via external configuration. The logic is not implemented in the application code but is handled by an external provider.
      • Examples: Script type, HTTP type.